--- title: "Linux: How to set up monitoring with alerts to Telegram" date: "2023-03-04" --- This article describes how to set up monitoring with alerts to Telegram using Grafana, Prometheus, Alertmanager, Node-exporter and Cadvisor. ### First step: Cloning the repository Log in to the server or go to local terminal and run the following commands: ```bash git clone https://github.com/digitalstudium/grafana-docker-stack.git cd grafana-docker-stack git checkout alertmanager ``` ### Second step: setting the external address of the server Open the `docker-compose.yml` file and on lines 22 and 38 change the address `127.0.0.1` to the address of the server where you want to install Prometheus. ### Third step: creating a bot in Telegram In the Telegram search, type `Botfather` and follow the first link: ![Botfather search](/images/botfather1.png "Botfather search") Then press `Start`: ![Botfather start](/images/botfather2.png "Botfather start") After that, create a bot using the `Botfather` instructions. As a result, you should receive a message with an API token: ![Botfather API token](/images/botfather3.png "Botfather API token") Copy this token and paste it on line 14 of the `configs/alertmanager.yml` file as the value of the `bot_token` parameter. Then create a telegram group and add the created bot to this group. This group will receive notifications (alerts). ### Fourth step: Get the group id In the group you added the bot to, write some command, for example: `/my_id foobar` Then in the browser follow the link `https://api.telegram.org/botINSERT_BOT_TOKEN_HERE/getUpdates` replacing `INSERT_BOT_TOKEN_HERE` with the token created in step 3. You should get something like this page: ![Telegram getting chat id](/images/chat_id.png "Telegram getting chat id") If there is nothing on your page, try sending the command `/my_id foobar` to the group again. You need to copy the chat id value from this page and paste it on line 15 of the `configs/alertmanager.yml` file as the value of the `chat_id` parameter. Note that the value of the `chat_id` parameter must be without quotes and with a hyphen at the beginning. ### Fifth step: Deploying the docker stack While in the `grafana-docker-stack` folder, run the following commands: ```bash docker swarm init docker stack deploy -c docker-compose.yml monitoring ``` Wait 5-10 minutes then run the command ```bash docker ps ``` The command output should contain 5 containers: prometheus, grafana, alertmanager, node-exporter, grafana. ```plaintext CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 46fba26e7234 gcr.io/cadvisor/cadvisor:v0.47.0 "/usr/bin/cadvisor -…" 5 days ago Up 5 days (healthy) 8080/tcp monitoring_cadvisor.1.q02qcn798dh0rydo1dzslylse f212e3c66786 prom/alertmanager:v0.25.0 "/bin/alertmanager -…" 6 days ago Up 6 days 9093/tcp monitoring_alertmanager.1.oysziztrqnur7xr0hr82avunz c16fb14929e2 prom/prometheus:v2.42.0 "/bin/prometheus --c…" 6 days ago Up 6 days 9090/tcp monitoring_prometheus.1.yslspi4fgjp7ic4f5e18gm99a 9bf01ce6b7a1 grafana/grafana:9.3.6-ubuntu "/run.sh" 6 days ago Up 6 days 3000/tcp monitoring_grafana.1.mypn85x12xw37ucprr33knuwk 58efdb46f5c3 kindest/node:v1.25.3 "/usr/local/bin/entr…" 6 days ago Up 6 days 127.0.0.1:46579->6443/tcp kind-control-plane ae15e453e517 prom/node-exporter:v1.5.0 "/bin/node_exporter …" 7 days ago Up 7 days 9100/tcp monitoring_node-exporter.1.uecim10ow12h1qlpox5lg0c5r ``` If you see this output, then everything turned out successfully. If not, try repeating all previous steps. ### Sixth step: Check if it works Go to [server ip-address or domain name]:3000 You should get Grafana opened. Enter login `admin` and password `admin`. Grafana will ask you to change your password, change it to any other. Under Dashboard -> Browse -> General you will see 2 dashboards: Cadvisor exporter and Node Exporter Full. Open them and make sure everything works. ## Summary Now you can evaluate server performance through Grafana graphs. You will also receive alerts about problems with the server in the Telegram group. Alert rules can be configured in `configs/prometheus/alert_rules.yml`