promethus监控nginx

promethues监控nginx可选两个exporter，通过nginx_exporter主要是获取nginx-status中的内建的指标，nginx自身提供status信息，较为简单，promethues中对应的metrics也较少，想要监控更多的指标可以通过nginx-vts-exporter采集信息，依赖在编译nginx的时候添加nginx-module-vts模块来实现。nginx virtual host traffic status模块是nginx第三方模块之一，vts提供了访问虚拟主机状态的信息，包含server，upstream以及cache的当前状态，类似于NGINX Plus 提供的在线活动监控功能。

Nginx_exporter安装

修改nginx.conf中监听80端口的server段添加如下location配置

server {
    listen 80;
    ..............................
    location /nginx_status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
    }
    ................................
    }

重新加载配置文件，访问测试是否可以获取监控信息

# nginx -s reload
# curl http://localhost/nginx_status
Active connections: 1     #活跃的连接数量
server accepts handled requests  #总共处理了1959个连接 , 成功创建1959次握手, 总共处理了2198个请求
 1959 1959 2198 
Reading: 0 Writing: 1 Waiting: 0 
读取客户端的连接数0
响应数据到客户端的数量1
Nginx已经处理完正在等候下一次请求指令的驻留连接0,
开启keep-alive的情况下,Waiting=active–(reading+writing)

下载nginx_exporter，需要使用go语言环境将源码编译成二进制可执行文件，完成后运行命令查看使用帮助

# ./nginx_exporter --help
Usage of ./nginx_exporter:
  -insecure  #若使用了https默认忽略服务证书
        Ignore server certificate if using https (default true)
  -nginx.scrape_uri string  #访问nginx_status的地址，默认为localhost,可根据实际情况修改
        URI to nginx stub status page (default "http://localhost/nginx_status")
  -telemetry.address string #默认监听9113端口
        Address on which to expose metrics. (default ":9113")
  -telemetry.endpoint string
        Path under which to expose metrics. (default "/metrics")

若不需要特殊设置，可直接启动exporter，成功运行后在promethues server端添加target

 # nohup /opt/nginx_exporter &

Nginx-vts模块安装

克隆git仓库

# git clone git://github.com/vozlt/nginx-module-vts.git

添加nginx-module-vts模块，构建编译配置，完成后make&&make install

./configure --prefix=/usr/local/nginx \
            --with-http_gzip_static_module \
            .....................................
            --add-module=/usr/local/src/nginx-module-vts

修改nginx配置文件

http {
......................................................
    vhost_traffic_status_zone;
......................................................
    server {
        listen       80;
        server_name  localhost;
.......................................................
        location /status {
               vhost_traffic_status_display;
               vhost_traffic_status_display_format html; 
        }
}
}

启动nginx，查看输出

# curl http://localhost/status/format/json
{
    "hostName": "nginx-vts",
    "nginxVersion": "1.12.2",
    "loadMsec": 1523696640973,
    "nowMsec": 1523706344593,
    "connections": {
        "active": 3,
        "reading": 0,
        "writing": 1,
        "waiting": 2,
        "accepted": 108,
        "handled": 108,
        "requests": 9728
    },
    "sharedZones": {
        "name": "ngx_http_vhost_traffic_status",
        "maxSize": 1048575,
        "usedSize": 2446,
        "usedNode": 1
    },
    "serverZones": {
        "localhost": {
            "requestCounter": 9727,
            "inBytes": 3086128,
            "outBytes": 33977540,
            "responses": {
                "1xx": 0,
                "2xx": 9727,
                "3xx": 0,
                "4xx": 0,
                "5xx": 0,
                "miss": 0,
                "bypass": 0,
                "expired": 0,
                "stale": 0,
                "updating": 0,
                "revalidated": 0,
                "hit": 0,
                "scarce": 0
            },
            "requestMsec": 0,
            "requestMsecs": {
                "times": [1523706282909, 1523706283914, 1523706284910, 1523706285913, 1523706333910, 1523706334914, 1523706335918, 1523706336914, 1523706337911, 1523706338911, 1523706339914, 1523706340911, 1523706341910, 1523706342911, 1523706343911],
                "msecs": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
            },
            "overCounts": {
                "maxIntegerSize": 18446744073709551615,
                "requestCounter": 0,
                "inBytes": 0,
                "outBytes": 0,
                "1xx": 0,
                "2xx": 0,
                "3xx": 0,
                "4xx": 0,
                "5xx": 0,
                "miss": 0,
                "bypass": 0,
                "expired": 0,
                "stale": 0,
                "updating": 0,
                "revalidated": 0,
                "hit": 0,
                "scarce": 0
            }
        },
        "*": {
            "requestCounter": 9727,
            "inBytes": 3086128,
            "outBytes": 33977540,
            "responses": {
                "1xx": 0,
                "2xx": 9727,
                "3xx": 0,
                "4xx": 0,
                "5xx": 0,
                "miss": 0,
                "bypass": 0,
                "expired": 0,
                "stale": 0,
                "updating": 0,
                "revalidated": 0,
                "hit": 0,
                "scarce": 0
            },
            "requestMsec": 0,
            "requestMsecs": {
                "times": [1523706283914, 1523706284910, 1523706285913, 1523706286317, 1523706334914, 1523706335918, 1523706336914, 1523706337911, 1523706338911, 1523706339914, 1523706340911, 1523706341910, 1523706342911, 1523706343911, 1523706281909],
                "msecs": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
            },
            "overCounts": {
                "maxIntegerSize": 18446744073709551615,
                "requestCounter": 0,
                "inBytes": 0,
                "outBytes": 0,
                "1xx": 0,
                "2xx": 0,
                "3xx": 0,
                "4xx": 0,
                "5xx": 0,
                "miss": 0,
                "bypass": 0,
                "expired": 0,
                "stale": 0,
                "updating": 0,
                "revalidated": 0,
                "hit": 0,
                "scarce": 0
            }
        }
    }
}

TIPS:

若当前线上已运行的nginx在当初编译时未加载vts模块，想要通过vts获取监控指标需要重新编译，在第一次编译安装的configure命令后添加参数--add-module=/usr/local/src/nginx-module-vts，再替换可执行文件。重新编译后会在nginx源码根目录下新生成一个objs目录，此目录下的nginx为新生成的二进制文件

# ll objs
total 7032
drwxr-xr-x 3 root root    4096 Apr 14 14:31 addon
-rw-r--r-- 1 root root   17998 Apr 14 14:31 autoconf.err
-rw-r--r-- 1 root root   48900 Apr 14 14:31 Makefile
-rwxr-xr-x 1 root root 7000009 Apr 14 14:33 nginx
-rw-r--r-- 1 root root    5433 Apr 14 14:33 nginx.8
-rw-r--r-- 1 root root    7482 Apr 14 14:31 ngx_auto_config.h
-rw-r--r-- 1 root root     657 Apr 14 14:31 ngx_auto_headers.h
-rw-r--r-- 1 root root    6183 Apr 14 14:31 ngx_modules.c
-rw-r--r-- 1 root root   93680 Apr 14 14:33 ngx_modules.o
drwxr-xr-x 9 root root    4096 Apr 14 09:48 src

将第一次编译的二进制文件备份，然后替换新的

# cp /apollo/env/nginx/thirdparty.nginx-1.12/sbin/nginx /apollo/env/nginx/thirdparty.nginx-1.12/sbin/nginx.bak20180414
# cp /usr/local/src/nginx-1.12.2/objs/nginx /apollo/env/nginx/thirdparty.nginx-1.12/sbin/nginx
cp: overwrite `/apollo/env/nginx/thirdparty.nginx-1.12/sbin/nginx'? y

替换前需要先停止nginx服务，否则会报错

cp: cannot create regular file `/apollo/env/nginx/thirdparty.nginx-1.12/sbin/nginx': Text file busy

Nginx-vts-exporter安装

在需要加入监控的客户端主机下载nginx-vts-exporter解压

[root@nginx-vts ~]# tar xvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz -C /opt/
nginx-vts-exporter-0.10.3.linux-amd64/
nginx-vts-exporter-0.10.3.linux-amd64/LICENSE
nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter
# cd /opt/nginx-vts-exporter-0.10.3.linux-amd64
# ls
LICENSE  nginx-vts-exporter
# ./nginx-vts-exporter  -h
Usage of ./nginx-vts-exporter:
  -insecure
        Ignore server certificate if using https (default true)
  -metrics.namespace string
        Prometheus metrics namespace. (default "nginx")
  -nginx.scrape_timeout int
        The number of seconds to wait for an HTTP response from the nginx.scrape_uri (default 2)
  -nginx.scrape_uri string
        URI to nginx stub status page (default "http://localhost/status")
  -telemetry.address string
        Address on which to expose metrics. (default ":9913")
  -telemetry.endpoint string
        Path under which to expose metrics. (default "/metrics")
  -version
        Print version information.
# nohup /opt/nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter -nginx.scrape_uri=http://localhost/status/format/json &

成功运行后在promethues Server端添加target，访问http://clientIP:9913/metrics 可查看到相关的监控指标，Grafana官网提供了VTS Stats模板，复制模板ID即可导入新的dashboard