While building up sloppy.io, monitoring the resource usage of all running docker containers is an important task. Right now one of my favourite way is using cadvisor in combination with heapster. cadvisor comes with a lot of interesting information like RAM, CPU and network usage and has an API too. heapster on the other side collects all this data from different cadvisor instances and saves it to influxdb a time series database. To have nice charts about all this information the heapster guys also created a ready to go grafana container.

grafana

grafana

heapster comes with kubernetes and coreos support but at sloppy.io we are using mesos but we are on the lucky side because heapster also comes with an option to use an hosts file, where you just define the hosts where cadvisor is running. heapster then connects to this hosts and pulls the data.

So, what we have to do is installing cadvisor on all mesos slaves, then start influxdb, grafana and heapster on a host where we want to collect the data. Sounds like a lot of work but as we want to monitor docker all this pieces are also provided as containers. So let's go.

On your mesos slaves:
docker run --name cadvisor --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro
 --publish=8080:8080 --detach=true  google/cadvisor:latest
Point your browser to http://mesosslaveip:8080/ and cadvisor should be there. cadvisor

cadvisor

cadvisor

cadvisor

Then on the host where you want to run heapster, influxdb and grafana:

docker run -d -p 8083:8083 -p 8086:8086 --name influxdb kubernetes/heapster_influxdb

The influxdb admin UI is availabe at http://hostip:8083/

Before starting heapster we have to define the hosts file. This is used by heapster to know on which hosts cadvisor is running to collect the data from.

{"items":[ {"name":"host1","ip":"192.168.9.101"},{"name":"host2","ip":"192.168.9.102"},{"name":"host3","ip":"192.168.9.103"} ] }

Save the file as hosts where we can export it to the heapster container in the next step. Here i save it to /export/heapster/hosts

Start heapster

docker run --name heapster --link influxdb:influxdb -v /export/heapster/hosts:/var/run/heapster/hosts -d kubernetes/heapster:v0.14.2 --sink="influxdb:http://influxdb:8086" --source="cadvisor:external?cadvisorPort=8080" 
cadvisorPort is an option which is not realy needed here as cadvisors default port is 8080 anyway. I just used it as an example for options you can use.

Now, start grafana:

docker run -d -p 80:80 -e INFLUXDB_HOST=host_ip_ kubernetes/heapster_grafana:v0.7

Grafana is now available at http://host_ip:80

grafana

grafana

The build in dashboard has already some graphs but if you want all information cadvisor provides also in grafana you have to create it yourself. Just check the queries for existing graphs as a start.

You mabye want to check out the latest versions for the docker images at the hub as maybe the versions i used here are already outdated.