神出鬼没的Enderman

收集Nginx的access和error日志

部署服务

• 参考上一篇文章安装nginx和filebeat
• 分别修改配置文件后重启服务

nginx.conf

#db01,db02,db03
vi /etc/nginx/nginx.conf
 
user  nginx;
worker_processes  1;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
 
events {
    worker_connections  1024;
}
 
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
#添加json日志格式
    log_format json '{ "time_local": "$time_local", '
                           '"remote_addr": "$remote_addr", '
                           '"referer": "$http_referer", '
                           '"request": "$request", '
                           '"status": $status, '
                           '"bytes": $body_bytes_sent, '
                           '"agent": "$http_user_agent", '
                           '"x_forwarded": "$http_x_forwarded_for", '
                           '"up_addr": "$upstream_addr",'
                           '"up_host": "$upstream_http_host",'
                           '"upstream_time": "$upstream_response_time",'
                           '"request_time": "$request_time"'
    ' }';
 
 
#    access_log  /var/log/nginx/access.log  main;
#修改为json格式日志
    access_log  /var/log/nginx/access.log  json;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
 
    include /etc/nginx/conf.d/*.conf;
}

filebeat.yml

#db01,db02,db03
vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
#不添加以下参数会产生报错
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

启动服务测试效果

启动nginx和filebeat后,使用ab命令访问3台主机的nginx生成日志

ab -n 100 -c 100 http://10.0.0.51/
ab -n 100 -c 100 http://10.0.0.52/
ab -n 100 -c 100 http://10.0.0.53/

查看结果

修改filebeat配置文件分离日志

filebeat.yml

• 官方文档1: https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-input-log.html#filebeat-input-log-common-options
• 官方文档2: https://www.elastic.co/guide/en/beats/filebeat/6.6/elasticsearch-output.html#indices-option-es
• 修改filebeat配置文件,重启服务

#db01,db02,db03
vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
#添加tags参考官方文档1
  tags: ["access"]
 
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
#  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
#区分tags参考官方文档2
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"
#不添加以下参数会产生报错
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

清空kibana,elasticsearch数据,再生成新的数据

ab -n 100 -c 100 http://10.0.0.51/db01
ab -n 100 -c 100 http://10.0.0.52/db02
ab -n 100 -c 100 http://10.0.0.53/db03

查看es-head数据

重复之前kibana创建索引的步骤重新创建索引

• access

• error

收集tomcat日志

部署服务

• 参考上一篇文章安装filebeat
• 如果没有java则需要安装
• 启动后访问10.0.0.51:8080

#db01
#安装java环境
yum install java-1.8.0-openjdk.x86_64 -y
#查看版本
java -version
 
#安装tomcat
yum install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc -y
 
#启动检查
systemctl start tomcat
systemctl status tomcat

修改日志格式

• tomcat默认日志格式和nginx类似,可以通过修改配置文件来修改日志为json格式
• 展示配置文件为134-139行,修改内容为默认配置文件139行

#db01
vi /etc/tomcat/server.xml
 
        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" --> 
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

修改filebeat配置文件获取日志

• 修改配置文件后重启filebeat,再制造日志

#db01
vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: log
  enabled: true
  paths:
#默认的日志文件有日期信息,日期部分需要用通配符替换
    - /var/log/tomcat/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
#添加输出部分参数
    - index: "tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "tomcat"
#不添加以下参数会产生报错
setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
setup.template.enabled: false
setup.template.overwrite: true

查看es-head数据

重复之前kibana创建索引的步骤重新创建索引

收集tomcat日志(匹配多行日志)

• 参考上一篇文章安装filebeat
• 官方文档: https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html
• 参考默认配置文件66行
• 因为java日志的输出信息非常多,需要将多行拼成一个事件,所以需要多行匹配模式
• 因为elasticsearch本身就是java开发的,所以我们可以直接收集ES的日志

修改filebeat配置文件获取日志

• 修改配置文件后重启filebeat,再制造日志

filebeat.inputs:
#java
- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  tags: ["es"]
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
#java
    - index: "es-java-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "es"
#不添加以下参数会产生报错
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

查看多行日志

如果没有多行日志,可以通过故意使用格式错误的配置文件启动,生成多行错误日志

重复之前kibana创建索引的步骤重新创建索引

查看明细

收集docker日志

抓取单个容器日志

以nginx容器为例,分别抓取access和error日志

• 参考上一篇文章安装filebeat
• 参考上一篇文章安装docker

拉取镜像生成容器

#db01
docker pull nginx
docker container run -p 80:80 -d --name nginx nginx:latest

• nignx日志查看/var/lib/docker/containers/*/*-json.log

#普通日志
{"log":"/docker-entrypoint.sh: Configuration complete; ready for start up\n","stream":"stdout","time":"2020-06-02T08:32:38.455755903Z"}
#错误日志
{"log":"2020/06/02 08:46:11 [error] 27#27: *1 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 10.0.0.1, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"10.0.0.51\"\n","stream":"stderr","time":"2020-06-02T08:46:11.115066506Z"}

修改filebeat配置文件获取日志

• 修改配置文件后重启filebeat,再制造日志

vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: docker
  containers.ids:
#获取指定容器编号的日志
    - '78a347c4a0e495e73f4e43dc7ad76c897888aa853ee9727330070c9acbff08aa'
 
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
#  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
#stdout是普通日志的标志,根据这个标志建立索引
        stream: "stdout"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
#stderr是错误日志的标志,根据这个标志建立索引
        stream: "stderr"
 
#不添加以下参数会产生报错
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

查看结果

收集所有容器的日志

创建容器

docker container run -p 8080:80 -d --name nginx2 nginx:last

修改filebeat配置文件获取日志

• 修改配置文件后重启filebeat,再制造日志

vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: docker
  combine_partial: true
  containers:
    ids:
      - "*"
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
 
#不添加以下参数会产生报错
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

查看结果

• 可以收集所有容器的普通日志和错误日志,但是无法区分

收集所有容器的日志并区分日志

每个容器的ID都不一样,为了区分不同服务运行的不同容器,可以使用docker-compose通过给容器添加labels标签来作为区分.然后filbeat把容器日志当作普通的json格式来解析并传输到Elasticsearch

安装docker-compose

-这里使用pip安装,默认源为国外,可以使用国内加速

• 相关网站: https://mirrors.tuna.tsinghua.edu.cn/help/pypi/

yum install -y python2-pip
#pip加速操作命令
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
#继续安装docker-compose
pip install docker-compose
#检查
docker-compose version

编写docker-compose.yml

vi docker-compose.yml
 
version: '3'
services:
  tomcat:
    image: nginx:latest
    # 设置labels
    labels:
      service: tomcat
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
  nginx:
    image: nginx:latest
    # 设置labels
    labels:
      service: nginx
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"

启动容器

#在当前目录下启动,启动的容器会在json格式的日志中打上service的标签
docker-compose up

修改filebeat配置文件获取日志

• 修改配置文件后重启filebeat,再制造日志

vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: log
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-tomcat-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "tomcat"
 
#不添加以下参数会产生报错
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

查看结果

收集所有容器的日志并区分日志类型

修改filebeat配置文件获取日志

• 上一个配置文件区分了不同容器,没有区分容器的普通日志和错误日志
• 修改配置文件后重启filebeat,再制造日志

vi /etc/filebeat/filebeat.yml
 
filebeat.inputs:
- type: log
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
 
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
#通过容器标识和日志标识区分不容容器中不同的日志类型
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
        stream: "stdout"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
        stream: "stderr"
    - index: "docker-tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "tomcat"
        stream: "stdout"
    - index: "docker-tomcat-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "tomcat"
        stream: "stderr"
 
#不添加以下参数会产生报错
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

查看结果