Warning: Virtual Machines need to have at least 2 cores for InfluxDB to perform optimally.
Heapster enables monitoring of Clusters using cAdvisor.
Heapster supports Kubernetes natively and collects resource usage of all the Pods running in the cluster. It was built to showcase the power of core Kubernetes concepts like labels and pods and the awesomeness that is cAdvisor.
Heapster can be used to enable cluster wide monitoring on other Cluster management solutions by running a simple cluster specific buddy container that will help heapster with discovery of hosts. For example, take a look at this guide for setting up Cluster monitoring in CoreOS.
#####How heapster works on Kubernetes:
- Discovers all minions in a Kubernetes cluster
- Collects container statistics from the cadvisors running on the minions
- Organizes stats into Pods
- Stores Pod stats in a configurable backend
Along with each container stat entry, it's Pod ID, Container name, Pod IP, Hostname and Labels are also stored. Labels are stored as key:value pairs.
Heapster currently supports in-memory and InfluxDB backends. Patches are welcome for adding more storage backends.
#####Run Heapster in a Kubernetes cluster with an Influxdb backend and Grafana
Step 1: Setup Kube cluster
Fork the Kubernetes repository and turn up a Kubernetes cluster, if you haven't already. Make sure kubecfg.sh is exported. By default, cAdvisor runs as a Pod on all nodes using a static manifest file that is distributed via salt. Make sure that it is running on port 4194 on all nodes.
Step 2: Start a Pod with Influxdb, grafana and elasticsearch
$ kubectl.sh create -f deploy/influx-grafana-pod.jsonStep 3: Start Influxdb service
$ kubectl.sh create -f deploy/influx-grafana-service.jsonStep 4: Update firewall rules
Open up ports tcp:80,8083,8086,9200.
$ gcutil addfirewall --allowed=tcp:80,tcp:8083,tcp:8086,tcp:9200 --target_tags=kubernetes-minion heapsterStep 5: Start Heapster Pod
$ kubectl.sh create -f deploy/heapster-pod.jsonVerify that all the pods and services are up and running:
$ kubectl.sh get pods$ kubectl.sh get servicesTo start monitoring the cluster using grafana, find out the the external IP of the minion where the 'influx-grafana' Pod is running from the output of kubectl.sh get pods influx-grafana, and visit http://<minion-ip>:80.
To access the Influxdb UI visit http://<minion-ip>:8083.
Step 6: Create CPU usage continuous query
For All containers CPU usage graph, connect to Influxdb UI, open k8s database and execute:
select container_name, derivative(cpu_cumulative_usage) as cpu_usage
from stats group by time(10s), container_name, hostname into cpu_stats
#####Hints
- Grafana's default username and password is 'admin'. You can change that by modifying the grafana container here
- To enable memory and swap accounting on the minions follow the instructions here
Contributions, questions, and comments are all welcomed and encouraged! Heapster and cAdvisor developers hang out in #google-containers room on freenode.net. We also have the google-containers Google Groups mailing list.