78 lines
4.4 KiB
Markdown
78 lines
4.4 KiB
Markdown
|
---
|
||
|
|
title: "Linux: How to set up monitoring with alerts to Telegram"
|
||
|
|
date: "2023-03-04"
|
||
|
|
---
|
||
|
|
This article describes how to set up monitoring with alerts to Telegram using Grafana, Prometheus, Alertmanager, Node-exporter and Cadvisor.
|
||
|
|
### First step: Cloning the repository
|
||
|
|
Log in to the server or go to local terminal and run the following commands:
|
||
|
|
```bash
|
||
|
|
git clone https://github.com/digitalstudium/grafana-docker-stack.git
|
||
|
|
cd grafana-docker-stack
|
||
|
|
git checkout alertmanager
|
||
|
|
```
|
||
|
|
<!--more-->
|
||
|
|
### Second step: setting the external address of the server
|
||
|
|
Open the `docker-compose.yml` file and on lines 22 and 38 change the address `127.0.0.1` to the address of the server where you want to install Prometheus.
|
||
|
|
### Third step: creating a bot in Telegram
|
||
|
|
In the Telegram search, type `Botfather` and follow the first link:
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
Then press `Start`:
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
After that, create a bot using the `Botfather` instructions.
|
||
|
|
As a result, you should receive a message with an API token:
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
Copy this token and paste it on line 14 of the `configs/alertmanager.yml` file as the value of the `bot_token` parameter.
|
||
|
|
|
||
|
|
Then create a telegram group and add the created bot to this group. This group will receive notifications (alerts).
|
||
|
|
|
||
|
|
### Fourth step: Get the group id
|
||
|
|
In the group you added the bot to, write some command, for example: `/my_id foobar`
|
||
|
|
|
||
|
|
Then in the browser follow the link
|
||
|
|
`https://api.telegram.org/botINSERT_BOT_TOKEN_HERE/getUpdates` replacing `INSERT_BOT_TOKEN_HERE` with the token created in step 3.
|
||
|
|
You should get something like this page:
|
||
|
|
|
||
|
|
|
||
|
|
If there is nothing on your page, try sending the command `/my_id foobar` to the group again.
|
||
|
|
|
||
|
|
You need to copy the chat id value from this page and paste it on line 15 of the `configs/alertmanager.yml` file as the value of the `chat_id` parameter. Note that the value of the `chat_id` parameter must be without quotes and with a hyphen at the beginning.
|
||
|
|
|
||
|
|
### Fifth step: Deploying the docker stack
|
||
|
|
While in the `grafana-docker-stack` folder, run the following commands:
|
||
|
|
```bash
|
||
|
|
docker swarm init
|
||
|
|
docker stack deploy -c docker-compose.yml monitoring
|
||
|
|
```
|
||
|
|
Wait 5-10 minutes then run the command
|
||
|
|
```bash
|
||
|
|
docker ps
|
||
|
|
```
|
||
|
|
|
||
|
|
The command output should contain 5 containers: prometheus, grafana, alertmanager, node-exporter, grafana.
|
||
|
|
```plaintext
|
||
|
|
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||
|
|
46fba26e7234 gcr.io/cadvisor/cadvisor:v0.47.0 "/usr/bin/cadvisor -…" 5 days ago Up 5 days (healthy) 8080/tcp monitoring_cadvisor.1.q02qcn798dh0rydo1dzslylse
|
||
|
|
f212e3c66786 prom/alertmanager:v0.25.0 "/bin/alertmanager -…" 6 days ago Up 6 days 9093/tcp monitoring_alertmanager.1.oysziztrqnur7xr0hr82avunz
|
||
|
|
c16fb14929e2 prom/prometheus:v2.42.0 "/bin/prometheus --c…" 6 days ago Up 6 days 9090/tcp monitoring_prometheus.1.yslspi4fgjp7ic4f5e18gm99a
|
||
|
|
9bf01ce6b7a1 grafana/grafana:9.3.6-ubuntu "/run.sh" 6 days ago Up 6 days 3000/tcp monitoring_grafana.1.mypn85x12xw37ucprr33knuwk
|
||
|
|
58efdb46f5c3 kindest/node:v1.25.3 "/usr/local/bin/entr…" 6 days ago Up 6 days 127.0.0.1:46579->6443/tcp kind-control-plane
|
||
|
|
ae15e453e517 prom/node-exporter:v1.5.0 "/bin/node_exporter …" 7 days ago Up 7 days 9100/tcp monitoring_node-exporter.1.uecim10ow12h1qlpox5lg0c5r
|
||
|
|
```
|
||
|
|
If you see this output, then everything turned out successfully. If not, try repeating all previous steps.
|
||
|
|
|
||
|
|
### Sixth step: Check if it works
|
||
|
|
|
||
|
|
Go to [server ip-address or domain name]:3000
|
||
|
|
You should get Grafana opened. Enter login `admin` and password `admin`. Grafana will ask you to change your password, change it to any other.
|
||
|
|
Under Dashboard -> Browse -> General you will see 2 dashboards: Cadvisor exporter and Node Exporter Full. Open them and make sure everything works.
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
Now you can evaluate server performance through Grafana graphs.
|
||
|
|
You will also receive alerts about problems with the server in the Telegram group. Alert rules can be configured in `configs/prometheus/alert_rules.yml`
|