Prometheus 应用监控

监控Nginx

nginx：1.16.1、nginx-vts-exporter：0.10.3、nginx-module-vts：1.6.3

监控Nginx主要用到以下三个模块：

• nginx-module-vts：Nginx的监控模块，能够提供JSON格式的数据产出。
• nginx-vts-exporter：主要用于收集Nginx的监控数据，并给Prometheus提供监控接口，默认端口号9913。
• Prometheus：监控Nginx-vts-exporter提供的Nginx数据，并存储在时序数据库中，可以使用PromQL对时序数据进行查询和聚合

nginx-module-vts模块编译
nginx_vts_exporter依赖nginx-module-vts模块，安装此模块无需任何其他依赖。模块与Nginx的版本兼容性如下：

1.11.x (last tested: 1.11.10)
1.10.x (last tested: 1.10.3)
1.8.x (last tested: 1.8.0)
1.6.x (last tested: 1.6.3)
1.4.x (last tested: 1.4.7)

1.安装依赖包

$ yum -y install gcc gcc-c++ autoconf automake make      
$ yum -y install zlib zlib-devel openssl openssl-devel pcre pcre-devel

2.添加www用户

$ groupadd -f www
$ useradd -g www www

3.下载nginx（版本1.16.1）

$ wget http://nginx.org/download/nginx-1.16.1.tar.gz
$ tar xf nginx-1.16.1.tar.gz

3.克隆vts模块仓库

https://github.com/vozlt/nginx-module-vts/releases

$ git clone git://github.com/vozlt/nginx-module-vts.git

4.编译nginx配置，添加vts模块
在nginx编译时添加vts模块：--add-module=/path/to/nginx-module-vts

$ cd nginx-1.16.1/

# 配置nginx
# --prefix 指定安装的目录
# /usr/local/nginx 是安装目录，不能和自己下载的文件目录重了
#./configure --prefix=/usr/local/nginx

$ ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-stream --with-http_gzip_static_module --with-http_sub_module --add-module=/root/nginx-module-vts

$ make && make install

5.Nginx config配置
更改Nginx Conf的配置，添加监控接口/status

$ vim /usr/local/nginx/conf/nginx.conf
user www;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  ‘$remote_addr - $remote_user [$time_local] "$request" ‘
                      ‘$status $body_bytes_sent "$http_referer" ‘
                      ‘"$http_user_agent" "$http_x_forwarded_for"‘;

    access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  65;

    # enabled nginx status
    vhost_traffic_status_zone;
    gzip  on;

    server {
        listen       80;
        server_name  localhost;
        location /status {
             vhost_traffic_status_display;
             vhost_traffic_status_display_format html;
             # Only allow local access the url
             allow 127.0.0.1;
        }
      }
}

检查nginx语法并启动

$ /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

$ /usr/local/nginx/sbin/nginx

6.curl查看捕获nginx数据

$ curl http://127.0.0.1/status/format/json

7.安装nginx-vts-exporter
下载地址：https://github.com/hnlq715/nginx-vts-exporter/releases

$ tar xf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
$ cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter  /usr/local/bin/

//启动命令
nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1/status/format/json

6.将nginx-vts-exporter添加system系统服务

$ vim /usr/lib/systemd/system/nginx-vts-exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1/status/format/json

[Install]
WantedBy=multi-user.target

7.启动添加的system服务

$ systemctl daemon-reload
$ systemctl start nginx-vts-exporter.service
$ systemctl enable nginx-vts-exporter.service

// nginx-vts-exporter 默认端口 - 9913
$ netstat -lntup | grep "9913"
tcp6       0      0 :::9913                 :::*                    LISTEN      9195/nginx-vts-expo

8.使用prometheus监控修改监控端配置文件

$ vim /usr/local/prometheus/prometheus.yml
  - job_name: ‘nginx‘
    static_configs:
      - targets: [‘10.0.0.171:9913‘]

9.启动prometheus服务

$ systemctl restart prometheus.service

10.查看监控端是否介入

11.Granfana导入Nginx监控图表

https://grafana.com/grafana/dashboards/2949

监控MySQL
文中主要监控MySQL/MySQL主从信息
版本：mysql-5.7，mysql_exporter-0.12.1

mysql_exporter下载地址：https://github.com/prometheus/mysqld_exporter/releases

1、下载mysql_exporter并解压

$ tar xf mysqld_exporter-0.12.1.linux-amd64.tar.gz

// 将mysql_exporter二进制文件拷贝至/usr/local/bin
$ cp mysqld_exporter-0.12.1.linux-amd64/mysqld_exporter /usr/local/bin/

2.需要授权用户给exporter使用

> CREATE USER ‘exporter‘@‘localhost‘ IDENTIFIED BY ‘abc12345‘ WITH MAX_USER_CONNECTIONS 3;

// 可查看主从运行情况查看线程，及所有数据库。
> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO ‘exporter‘@‘localhost‘;

为该用户设置最大连接数为了避免监控数据过大导致服务器超载

3.修改mysql配置文件，添加刚才创建的exporter用户和密码

$ vim /etc/my.cnf
[client]
user=exporter
password=abc12345

4.启动exporter客户端，需指定mysql配置文件，读取exporter用户和密码

$ mysqld_exporter --config.my-cnf=/etc/my.cnf

常用参数：
// 选择采集innodb
--collect.info_schema.innodb_cmp
// innodb存储引擎状态
--collect.engine_innodb_status
// 指定配置文件
--config.my-cnf="/etc/my.cnf"

5.添加system系统服务

$ vim /usr/lib/systemd/system/mysql_exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/mysqld_exporter --config.my-cnf=/etc/my.cnf

[Install]
WantedBy=multi-user.target

6.启动添加的system服务

$ systemctl daemon-reload
$ systemctl start mysql_exporter.service
$ systemctl enable mysql_exporter.service

// mysql_export默认端口 - 9104
$ netstat -lntup | grep "9104"
tcp6       0      0 :::9104                 :::*                    LISTEN      34137/mysqld_export

7.curl查看捕获mysql数据

curl http://localhost:9104/metrics

8.使用prometheus监控修改监控端配置文件

$ vim /usr/local/prometheus/prometheus.yml
  - job_name: ‘mysql‘
    scrape_interval: 5s
    # 静态添加node
    static_configs:
      - targets: [‘10.0.0.171:9104‘]

9.启动prometheus服务

$ systemctl restart prometheus.service

10.查看监控端是否介入

11.Granfana导入MySQL监控图表

https://grafana.com/grafana/dashboards/8564/revisions

监控Redis
本文主要监控redis信息
版本：redis-3.2.2，redis_exporter-1.3.4

安装部署redis：https://blog.51cto.com/12643266/2349586
redis_exporter下载地址：https://github.com/oliver006/redis_exporter/releases/

1.下载redis_exporter插件

代理插件不一定非要安装在redis端

$ wget https://github.com/oliver006/redis_exporter/releases/download/v1.3.4/redis_exporter-v1.3.4.linux-amd64.tar.gz
$ tar xf redis_exporter-v1.3.4.linux-amd64.tar.gz
$ cp redis_exporter-v1.3.4.linux-amd64/redis_exporter  /usr/local/bin/

2.启动redis_exporter登陆redis

// 无密码
redis_exporter -redis.addr 127.0.0.1:6379 
// 有密码
redis_exporter -redis.addr 172.0.0.1:6379  -redis.password 123456

3.添加system系统服务

$ vim /usr/lib/systemd/system/redis_exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/redis_exporter -redis.addr 127.0.0.1:6379 

[Install]
WantedBy=multi-user.target

4.启动添加的system服务

$ systemctl daemon-reload
$ systemctl start redis_exporter.service
$ systemctl enable redis_exporter.service

// redis_exporter默认端口 - 9121
$ netstat -lntup | grep 9121""
tcp6       0      0 :::9121                 :::*                    LISTEN      4465/redis_exporter

5.本地curl访问

curl http://127.0.0.1:9121/metrics

6.prometheus.yml加入redis节点，然后重启prometheus：

- job_name: ‘redis‘
    static_configs:
      - targets: [‘10.0.0.171:9121‘]

7.重启prometheus服务端

systemctl restart prometheus.service

8.查看监控端是否介入

9.Granfana导入Redis监控图表
redis节点监控：https://grafana.com/grafana/dashboards/2751
redis哨兵监控：https://grafana.com/grafana/dashboards/9570