自动化监控运维(五) Alertmanager 基于 email 告警配置

Posted on Edited on In Word count in article: 2.4k Reading time ≈ 2 mins.

一、 安装 Alertmanager

1.1 基于二进制安装

  • 下载
    https://github.com/prometheus/alertmanager/releases

  • 解压
    tar -xzvf alertmanager-0.26.0.linux-amd64.tar.gz

  • 运行
    ./alertmanager --config.file="alertmanager.yml"

1.2 基于 docker 安装

docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager

二、 配置 Alertmanager

2.1 配置 Alertmanager 告警方式

  • 编辑配置文件: alertmanager.yml
  • 字段需要注明端口

    ``` yml
    global: 
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.qq.com:25'
      smtp_from: '[email protected]'
      smtp_auth_username: '[email protected]' #
      smtp_auth_password: 'yourpasswd/authenticCode'
    route:
      group_by: ['alertname']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: 'email'
    receivers:
      - name: 'email'
        email_configs:
          - to: '[email protected]'
    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'dev', 'instance']

2.2 配置 Prometheus

  • 编辑配置文件 prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["192.168.1.201:9093"]

rule_files:
   - "/usr/local/prometheus/rules/*"
  • ``` 用于存放告警规则



    ### 2.3 告警示例

    ``` vim /usr/local/prometheus/rules/first.rules.yml ```

    ``` yml
    groups:
    - name: cpuAlertGroup
      rules:
      - alert: hostCPUUsageTooHigh
        expr: (1 - sum(increase(node_cpu_seconds_total{mode="idle"}[1m])) by (instance) / sum(increase(node_cpu_seconds_total[1m])) by (instance) ) * 100 > 50
        for: 30s
        labels:
          biz_type: cpu_usage
        annotations:
          summary: "Instance {{ $labels.instance }} CPU usgae high"
          description: "{{ $labels.instance }} CPU usage above 50% (current : {{ $value }})"

三、 测试

  • 测试脚本
#!/usr/bin/python
# -*- coding: UTF-8 -*-

### 用法: python cpu_mem.py 4 1024
### 解释: 
###     占满 4个核心
###     占用1024MB内存

import sys
import time
from multiprocessing import Process

def exec_func(bt):

    while True: 
        for i in range(0, 9600000):
            pass
        time.sleep(bt)

if __name__ == "__main__":

    if len(sys.argv) != 3:
        print('Need one argument! ')
        sys.exit()
    cpu_logical_count = int(sys.argv[1])
    cpu_sleep_time = 0.01
    memory_used_mb = int(sys.argv[2])

    try:
        s = ' ' * (memory_used_mb * 1024 * 1024)
    except MemoryError:
        print("剩余内存不足，内存有溢出......")

    try:
        p = Process(target=exec_func, args=("bt",))
        ps_list = []
        for i in range(0, cpu_logical_count):
            ps_list.append(Process(target=exec_func, args=(cpu_sleep_time,)))
        for p in ps_list:
            p.start()
        for p in ps_list:
            p.join()
    except KeyboardInterrupt:
        print("资源浪费结束!")

图
图