dweet.io + InfluxDB + Grafana

In this post I'm showing you how to ingest sensor data from dweet.io into the time series database InfluxDB and visualize the result in Grafana, a dashboard for metrics.

The concrete steps we will perform are:

  1. Provision EC2 instance
  2. Install InfluxDB
  3. Install Grafana
  4. Getting data from dweet.io things
  5. Putting it all together

Provision an EC2 instance

Before we set up InfluxDB, we first provision an EC2 instance (I picked an m3.large, RHEL 6.5 64bit). In order to install stuff, log in as so (assuming you private key is at ~/.ssh/mhausenblas and you know the public host name of the instance):

ssh -i ~/.ssh/mhausenblas ec2-user@ec2-54-75-160-154.eu-west-1.compute.amazonaws.com

For convenience you will want to add the public IP address of the EC2 instance to your local /etc/hosts, using the hostname influxdb.

Install InfluxDB

To set up InfluxDB manually, do the following:

# install:
wget http://s3.amazonaws.com/influxdb/influxdb-latest-1.x86_64.rpm
sudo rpm -ivh influxdb-latest-1.x86_64.rpm
# start:
sudo /etc/init.d/influxdb start

Then, visit InfluxDB's WebUI at influxdb:8083 to get started and learn how to create a database called dweet with a username/password, the same as well as a database called grafana, again with same username/password. So you end up with the following:

Two databases in InfluxDB

Install Grafana

I suggest you quickly skim Philip O'Toole's instructions how to set up Grafana with InfluxDB because we'll be roughly following it.

First, Grafana needs a Web server to run, because it's a client-side Web app. So let's install Nginx. First create a repo file:

sudo touch /etc/yum.repos.d/nginx.repo

and edit nginx.repo so that it says the following:

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/rhel/6/$basearch/
gpgcheck=0
enabled=1

Then install Nginx and start it thereafter:

sudo yum install nginx
sudo service nginx restart

Now let's download and install Grafana:

# install:
wget http://grafanarel.s3.amazonaws.com/grafana-1.7.0.tar.gz
tar xvfz grafana-1.7.0.tar.gz
cd grafana-1.7.0
cp config.sample.js config.js

Next configure Grafana via config.js:

datasources: {
  'dweet': {
    type: 'influxdb',
    url: 'http://localhost:8086/db/dweet',
    username: 'dweet',
    password: 'dweet'
    default: true
    },
  'grafana': {
    type: 'influxdb',
    url: 'http://localhost:8086/db/grafana',
    username: 'grafana',
    password: 'grafana',
    grafanaDB: true
    }
}

Configure Nginx to use Grafana as default by editing
/etc/nginx/conf.d/default.conf to change root to wherever you've installed Grafana (/home/ec2-user/grafana-1.7.0 in my case):

server {
   ...
   root /home/ec2-user/grafana-1.7.0;
   ...
}

To make it work, you'll have to have port 80 enabled in the security group and also patch iptables of the EC2 instance. You do the latter by adding following line to /etc/sysconfig/iptables on the EC2 instance and then restart it via service iptables restart:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

Finally, make sure Nginx has necessary permissions:

chmod 755 /home/ec2-user

Getting data from dweet.io things

Using dweet.io we first discover and pick a thing, for example, manually like so:

curl https://dweet.io:443/get/latest/dweet/for/AvocadoGrove
  {
      "by": "getting",
      "the": "dweets",
      "this": "succeeded",
      "with": [
          {
              "content": {
                  "aiHotWaterTemp_degreesF": 98.463,
                  "aiOutsideTemp_degreesF": 95.935,
                  "ai_local_io_0400": 120.864,
                  "ai_local_io_0401": 0.684,
                  "ai_local_io_0402": 32.16,
                  "ai_local_io_0403": 0.003,
                  "ai_local_io_0404": 120.696,
                  "ai_local_io_0405": 1.196,
                  "ai_local_io_0406": 42.72,
                  "ai_local_io_0407": 0.005,
                  "ai_local_io_0408": 0.144,
                  "ai_local_io_0409": 0,
                  "ai_local_io_0410": 0,
                  "ai_local_io_0411": 0,
                  "ai_local_io_0412": 0.002,
                  "ai_local_io_0413": 0.002,
                  "ai_local_io_0700": -459.43,
                  "ai_local_io_0701": -459.43
              },
              "created": "2014-09-06T16:56:32.305Z",
              "thing": "AvocadoGrove"
          }
      ]
  }

In order to ingest data to InfluxDB we want to extract the field with[0].content.X, for example:

  with[0].content.aiHotWaterTemp_degreesF == 98.463

To make things easier, I've written a script that allows to automatically ingest dweet.io data into InfluxDB and which is available on GitHub.

Putting it all together

As we have now InfluxDB and Grafana installed and connected, it's time to automatically create some data points using above mentioned influxdb-dweet.io script:

$ python dwingest.py influxdb cam-05-1 lux 30
2014-09-07T06:47:17 Using InfluxDB host at influxdb:8086
2014-09-07T06:47:17 =======================================
2014-09-07T06:47:17 Querying cam-05-1:lux every 30 sec
2014-09-07T06:47:17 ---------------------------------------
2014-09-07T06:47:19 Ingested value: 25714.4
2014-09-07T06:47:49 ---------------------------------------
2014-09-07T06:47:49 Ingested value: 26313.6
...

The above command ingests a data point with the key lux from the thing cam-05-1 every 30 sec into InfluxDB running on host influxdb.

Now we're good to go! Visit Grafana's WebUI at influxdb:80, add graphs to your liking by following the instructions and eventually you should see something like the following:

Two dweet.io things as time series in Grafana

Conclusion

Today you've learned how to set up InfluxDB on a EC2 instance along with Grafana and ingest data from dweet.io things as well as how to visualize a time series in Grafana.

I hope you had fun reading the blog post, trying it out yourself and please do let me know how things are working for you or what other stuff you came up with.

comments powered by Disqus