Note: This is a customized version of https://github.com/stefanprodan/dockprom.git Credit: Stefan Prodan
A monitoring solution for Docker hosts and containers with Prometheus, Grafana, cAdvisor and InfluxDB.
Clone this repository on your Docker host, cd into dockprom directory and run compose up:
git clone https://github.com/manishgupta24/dockprom.git
cd dockprom
ADMIN_USER=admin ADMIN_PASSWORD=admin docker-compose up -dPrerequisites:
- Docker Engine >= 1.13
- Docker Compose >= 1.11
Containers:
- Prometheus (metrics database)
http://<host-ip>:9090 - Prometheus-Pushgateway (push acceptor for ephemeral and batch jobs)
http://<host-ip>:9091 - Grafana (visualize metrics)
http://<host-ip>:3000 - cAdvisor (containers metrics collector)
- Caddy (reverse proxy and basic auth provider for prometheus and alertmanager)
- InfluxDB (timeseries database for storing prometheus metrics)
Navigate to http://<host-ip>:3000 and login with user admin password admin. You can change the credentials in the compose file or by supplying the ADMIN_USER and ADMIN_PASSWORD environment variables on compose up. The config file can be added directly in grafana part like this
grafana:
image: grafana/grafana:5.2.4
env_file:
- config
and the config file format should have this content
GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=changeme
GF_USERS_ALLOW_SIGN_UP=false
If you want to change the password, you have to remove this entry, otherwise the change will not take effect
- grafana_data:/var/lib/grafana
Grafana is preconfigured with dashboards and Prometheus as the default data source:
- Name: Prometheus
- Type: Prometheus
- Url: http://prometheus:9090
- Access: proxy
The pushgateway is used to collect data from batch jobs or from services.
To push data, simply execute:
echo "some_metric 3.14" | curl --data-binary @- http://user:password@localhost:9091/metrics/job/some_job
Please replace the user:password part with your user and password set in the initial configuration (default: admin:admin).
In Grafana versions >= 5.1 the id of the grafana user has been changed. Unfortunately this means that files created prior to 5.1 won’t have the correct permissions for later versions.
| Version | User | User ID |
|---|---|---|
| < 5.1 | grafana | 104 |
| >= 5.1 | grafana | 472 |
There are two possible solutions to this problem.
- Change ownership from 104 to 472
- Start the upgraded container as user 104
To change ownership of the files run your grafana container as root and modify the permissions.
First perform a docker-compose down then modify your docker-compose.yml to include the user: root option:
grafana:
image: grafana/grafana:5.2.2
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources:/etc/grafana/datasources
- ./grafana/dashboards:/etc/grafana/dashboards
- ./grafana/setup.sh:/setup.sh
entrypoint: /setup.sh
user: root
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
Perform a docker-compose up -d and then issue the following commands:
docker exec -it --user root grafana bash
# in the container you just started:
chown -R root:root /etc/grafana && \
chmod -R a+r /etc/grafana && \
chown -R grafana:grafana /var/lib/grafana && \
chown -R grafana:grafana /usr/share/grafana
To run the grafana container as user: 104 change your docker-compose.yml like such:
grafana:
image: grafana/grafana:5.2.2
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources:/etc/grafana/datasources
- ./grafana/dashboards:/etc/grafana/dashboards
- ./grafana/setup.sh:/setup.sh
entrypoint: /setup.sh
user: "104"
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
Update the targets/target_nodes.json. In above json file add/remove the host:port to/from targets list. The update will be immediately reflected in the prometheus server.