4.4 KiB
| title | date |
|---|---|
| Linux: How to set up monitoring with alerts to Telegram | 2023-03-04 |
This article describes how to set up monitoring with alerts to Telegram using Grafana, Prometheus, Alertmanager, Node-exporter and Cadvisor.
First step: Cloning the repository
Log in to the server or go to local terminal and run the following commands:
git clone https://github.com/digitalstudium/grafana-docker-stack.git
cd grafana-docker-stack
git checkout alertmanager
Second step: setting the external address of the server
Open the docker-compose.yml file and on lines 22 and 38 change the address 127.0.0.1 to the address of the server where you want to install Prometheus.
Third step: creating a bot in Telegram
In the Telegram search, type Botfather and follow the first link:
Then press Start:
After that, create a bot using the Botfather instructions.
As a result, you should receive a message with an API token:
Copy this token and paste it on line 14 of the configs/alertmanager.yml file as the value of the bot_token parameter.
Then create a telegram group and add the created bot to this group. This group will receive notifications (alerts).
Fourth step: Get the group id
In the group you added the bot to, write some command, for example: /my_id foobar
Then in the browser follow the link
https://api.telegram.org/botINSERT_BOT_TOKEN_HERE/getUpdates replacing INSERT_BOT_TOKEN_HERE with the token created in step 3.
You should get something like this page:
If there is nothing on your page, try sending the command /my_id foobar to the group again.
You need to copy the chat id value from this page and paste it on line 15 of the configs/alertmanager.yml file as the value of the chat_id parameter. Note that the value of the chat_id parameter must be without quotes and with a hyphen at the beginning.
Fifth step: Deploying the docker stack
While in the grafana-docker-stack folder, run the following commands:
docker swarm init
docker stack deploy -c docker-compose.yml monitoring
Wait 5-10 minutes then run the command
docker ps
The command output should contain 5 containers: prometheus, grafana, alertmanager, node-exporter, grafana.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
46fba26e7234 gcr.io/cadvisor/cadvisor:v0.47.0 "/usr/bin/cadvisor -…" 5 days ago Up 5 days (healthy) 8080/tcp monitoring_cadvisor.1.q02qcn798dh0rydo1dzslylse
f212e3c66786 prom/alertmanager:v0.25.0 "/bin/alertmanager -…" 6 days ago Up 6 days 9093/tcp monitoring_alertmanager.1.oysziztrqnur7xr0hr82avunz
c16fb14929e2 prom/prometheus:v2.42.0 "/bin/prometheus --c…" 6 days ago Up 6 days 9090/tcp monitoring_prometheus.1.yslspi4fgjp7ic4f5e18gm99a
9bf01ce6b7a1 grafana/grafana:9.3.6-ubuntu "/run.sh" 6 days ago Up 6 days 3000/tcp monitoring_grafana.1.mypn85x12xw37ucprr33knuwk
58efdb46f5c3 kindest/node:v1.25.3 "/usr/local/bin/entr…" 6 days ago Up 6 days 127.0.0.1:46579->6443/tcp kind-control-plane
ae15e453e517 prom/node-exporter:v1.5.0 "/bin/node_exporter …" 7 days ago Up 7 days 9100/tcp monitoring_node-exporter.1.uecim10ow12h1qlpox5lg0c5r
If you see this output, then everything turned out successfully. If not, try repeating all previous steps.
Sixth step: Check if it works
Go to [server ip-address or domain name]:3000
You should get Grafana opened. Enter login admin and password admin. Grafana will ask you to change your password, change it to any other.
Under Dashboard -> Browse -> General you will see 2 dashboards: Cadvisor exporter and Node Exporter Full. Open them and make sure everything works.
Summary
Now you can evaluate server performance through Grafana graphs.
You will also receive alerts about problems with the server in the Telegram group. Alert rules can be configured in configs/prometheus/alert_rules.yml