EFK架构分析nginx，tomcat日志及kibana出图展示
|
27
|
0
|
零碎文章

1730 字
|
27 分钟

ES 的 master 选举流程

1.启动检查：启动时会检查集群是否有 master，如果有则不发起选举 master。
2.初始状态与信息发送：刚开始启动，所有节点均认为自己是 master，并向集群的其他节点发送信息（包含 ClusterStateVersion，ID 等）。
3.获取可选举节点列表：基于类似 gossip 协议获取所有可以参与 master 选举的节点列表。
4.第一次比较：先比较 “ClusterStateVersion”，谁最大，谁优先级高，会被选举出 master。
5.第二次比较：如果通过 “ClusterStateVersion” 比不出来，则比较 ID，谁的 ID 小，就优先成为 master。
6.选举完成条件：当集群半数以上节点参与选举完成后，则完成 master 选举，比如有 N 个节点，仅需要 “(N/2)+1” 个节点就可以确认 master。
7.通报选举结果：master 选举完成后，会向集群列表通报最新的 master 节点，此时才意味着选举完成。

ES 的 master 选举流程如下：节点启动时会先检查集群是否已有 master，若有则不发起选举。刚开始启动时，所有节点都认为自己是 master，并向其他节点发送包含 ClusterStateVersion、ID 等信息。基于类似 gossip 协议获取所有可参与选举的节点列表，然后先比较 “ClusterStateVersion”，数值最大者优先级高，会被选举为 master，若无法比较出则比较 ID，ID 小的优先成为 master。当集群中超过半数节点参与选举完成后，即 N 个节点中至少有 “(N/2)+1” 个节点参与就可确认 master。最后，master 选举完成后会向集群列表通报最新的 master 节点，至此选举才算真正完成。

Filebeat 基于模块分析 Nginx 日志

1. 查看 NGINX 的日志

[root@elk92 ~]# cat /var/log/nginx/access.log 
221.218.213.9 - - [11/Mar/2025:18:27:23 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
221.218.213.9 - - [11/Mar/2025:18:30:16 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
221.218.213.9 - - [11/Mar/2025:18:30:16 +0800] "GET /favicon.ico HTTP/1.1" 404 197 "http://10.0.0.92/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
21.218.213.9 - - [12/Mar/2025:10:28:08 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
1.218.213.9 - - [12/Mar/2025:10:28:08 +0800] "GET /favicon.ico HTTP/1.1" 404 134 "http://10.0.0.92/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
2.218.213.9 - - [12/Mar/2025:10:28:54 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
3.218.213.9 - - [12/Mar/2025:10:29:17 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
4.218.213.9 - - [12/Mar/2025:10:29:56 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 13; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Mobile Safari/537.36"
[root@elk92 ~]#

2. Filebeat 模块管理

2.1 查看支持的模块列表
[root@elk92 ~]# filebeat modules list
Enabled:  # 表示启用的模块

Disabled:  # 表示禁用的模块列表
activemq
apache
auditd
aws
awsfargate
azure
barracuda
...
2.2 启动多个模块
[root@elk92 ~]# filebeat modules enable mysql nginx mongodb redis
Enabled mysql
Enabled nginx
Enabled mongodb
Enabled redis
[root@elk92 ~]# 
[root@elk92 ~]# ll -1 /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 297 Feb 14 00:58 /etc/filebeat/modules.d/mongodb.yml
-rw-r--r-- 1 root root 472 Feb 14 00:58 /etc/filebeat/modules.d/mysql.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
-rw-r--r-- 1 root root 567 Feb 14 00:58 /etc/filebeat/modules.d/redis.yml
[root@elk92 ~]# 
[root@elk92 ~]# filebeat modules list
Enabled:  # 发现启用的模块成功啦~
mongodb
mysql
nginx
redis

Disabled:
activemq
apache
auditd
aws
awsfargate
...
2.3 禁用多个模块
[root@elk92 ~]# filebeat modules disable mysql  mongodb redis
Disabled mysql
Disabled mongodb
Disabled redis
[root@elk92 ~]# 
[root@elk92 ~]# ll -1 /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
[root@elk92 ~]# 
[root@elk92 ~]# filebeat modules list
Enabled:
nginx

Disabled:
activemq
apache
auditd
aws
...

3. 编写 Filebeat 的配置文件

[root@elk92 ~]# cat /etc/filebeat/config/05-modules-nginx-to-es.yaml 
# 配置模块功能
filebeat.config.modules:
  # 加载Filebeat配置文件目录的"modules.d"子目录下的所有"*.yml"文件
  path: ${path.config}/modules.d/*.yml
  # 是否支持热加载
  reload.enabled: true

output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux96-modules-nginx-%{+yyyy.MM.dd}

setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux96"
setup.template.pattern: "haoshuaicongedu-linux96-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk92 ~]#

4. 修改 nginx 模块文件

[root@elk92 ~]# yy /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: false
  ingress_controller:
    enabled: false
[root@elk92 ~]#

5. 启动 Filebeat 实例

[root@elk92 ~]# rm -rf /var/lib/filebeat/
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-modules-nginx-to-es.yaml

6. Kibana 出图展示

nginx数据分析过滤
kibana分析PV并出图展示
kibana分析IP数量统计
kibana分析带宽并出图展示
kibana制作Dashboard实战
kibana分析设备占比统计
kibana分析操作系统占比统计
kibana分析全球用户分布
Dashboard集合展示分析数据
image-20250312210051895

Filebeat 采集 tomcat 日志

  1. 二进制部署 tomcat 环境
  • 1.1 下载 tomcat 软件包
wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.5/bin/apache-tomcat-11.0.5.tar.gz
svip:
[root@elk93 ~]# wget http://192.168.16.253/Resources/ElasticStack/softwares/tomcat/apache-tomcat-11.0.5.tar.gz
  • 1.2 解压软件包
[root@elk93 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
  • 1.3 配置环境变量

因为es是java编写的，所以软件自带jdk，这里共用es的jdk环境，配置变量即可

[root@elk93 ~]# cat /etc/profile.d/tomcat.sh
#!/bin/bash

export JAVA_HOME=/usr/share/elasticsearch/jdk
export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.5
export PATH=$PATH:$JAVA_HOME/bin:$TOMCAT_HOME/bin

[root@elk93 ~]# 
[root@elk93 ~]# source  /etc/profile.d/tomcat.sh
[root@elk93 ~]# 
[root@elk93 ~]# java --version
openjdk 22.0.2 2024-07-16
OpenJDK Runtime Environment (build 22.0.2+9-70)
OpenJDK 64-Bit Server VM (build 22.0.2+9-70, mixed mode, sharing)
[root@elk93 ~]#
  • 1.4 修改 tomcat 配置文件

这里是指定tomcat日志格式为json格式，方便监控分析日志

[root@elk93 ~]# vim /usr/local/apache-tomcat-11.0.5/conf/server.xml 
....
<Host name="localhost"  appBase="webapps"
      unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
            prefix="tomcat.haoshuaicongedu.com_access_log" suffix=".json"
pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;request&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;http_user_agent&quot;:&quot;%{User-Agent}i&quot;}"/>
</Host>
....

[root@elk93 ~]#
  • 1.5 启动 tomcat
[root@elk93 ~]# catalina.sh start 
Using CATALINA_BASE:   /usr/local/apache-tomcat-11.0.5
Using CATALINA_HOME:   /usr/local/apache-tomcat-11.0.5
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.5/temp
Using JRE_HOME:        /usr/share/elasticsearch/jdk
Using CLASSPATH:       /usr/local/apache-tomcat-11.0.5/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.5/bin/tomcat-juli.jar
Using CATALINA_OPTS:   
Tomcat started.
[root@elk93 ~]# 
[root@elk93 ~]# 
[root@elk93 ~]# ss -ntl | grep 8080
LISTEN 0      100                     *:8080            *:*          
[root@elk93 ~]#
  • 1.6 访问测试
http://tomcat.haoshuaicongedu.com:8080/
  1. 配置 Filebeat 采集日志
  • 2.1 主配置加载模块文件

开启tomcat模块

[root@elk93 ~]# cat /tmp/modules-tomcat-to-es.yaml 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux96-modules-tomcat-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux96"
setup.template.pattern: "haoshuaicongedu-linux96-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk93 ~]# 
[root@elk93 ~]#
  • 2.2 启动 tomcat 模块
[root@elk93 ~]# filebeat  modules enable tomcat
Enabled tomcat
[root@elk93 ~]# filebeat  modules list
Enabled:
tomcat

Disabled:
...
  • 2.3 修改 tomcat 的模块文件

默认是使用udp采集数据，udp采集不到数据，所以要改成file，让他去文件去采集数据 var.input: file

指定数据文件的路径 var.paths:

指定时区 var.tz_offset: +08:00 (可选配置)

[root@elk93 ~]# yy /etc/filebeat/modules.d/tomcat.yml 
- module: tomcat
  log:
    enabled: true
    var.input: file   
    var.paths:
      - /usr/local/apache-tomcat-11.0.5/logs/*.json
    var.tz_offset: +08:00
[root@elk93 ~]#
  • 2.4 启动 tomcat 实例
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-to-es.yaml

filebeat 的 processors 处理器实战案例

1. 编写 Filebeat 的配置文件

jion格式，一坨，采集到fibana中后是一行一大串，所有字段都在一行，无法进行指定字段分析

所以要借助processors 处理器来分析，提出每个字段，可以根据想要结果看相应的字段

[root@elk93 ~]# cat /tmp/modules-tomcat-to-es.yaml 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

# 定义处理器
processors:
    # 对json字段进行解码
  - decode_json_fields:
      # 指定要解码的字段
      fields: ["event.original"]          # 这个是一个数组，里面是所有字段
      # 将解码后的json数据放在根字段下
      target: ""
      # 覆盖已经有的字段，顶级字段，默认值为false
      overwrite_keys: true
      # 保留解码错误信息，会建一个错误字段可以查看
      add_error_key: true
    # 删除字段
  - drop_fields:
      # 当条件符合时才会执行当前的处理逻辑
      when:
        equals:
          status: "404"
      # 要删除的特定字段，删除不顺眼的字段
      fields: ["log.file.path"]

output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux95-modules-tomcat-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux95"
setup.template.pattern: "haoshuaicongedu-linux95-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk93 ~]#

2. 启动 Filebeat 实例

[root@elk93 ~]# rm -rf /var/lib/filebeat/
[root@elk93 ~]# 
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-to-es.yaml

3. 测试验证

略，见视频。 参考字段 “log.file.path”。

image-20250312222228933

文档参考链接

https://www.elastic.co/guide/en/beats/filebeat/7.17/filtering-and-enhancing-data.html

EFK 分析 web 集群架构

EFK分析web集群架构图解

1. web 集群环境准备

1.1 91 节点准备 tomcat 环境
[root@elk91 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
[root@elk91 ~]# echo 10.0.0.91 > /usr/local/apache-tomcat-11.0.5/webapps/ROOT/index.html
[root@elk91 ~]# catalina.sh start
1.2 93 节点准备测试环境
[root@elk93 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
[root@elk93 ~]# echo 10.0.0.93 > /usr/local/apache-tomcat-11.0.5/webapps/ROOT/index.html
[root@elk93 ~]# catalina.sh start
1.3 92 节点 nginx 代理 tomcat 服务
[root@elk92 ~]# yy /etc/nginx/nginx.conf 
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
    worker_connections 768;
}
http {
   upstream haoshuaicongedu-linux96 {
       server 10.0.0.91:8080;
       server 10.0.0.93:8080;
   }

   server {
       server_name tomcat.haoshuaicongedu.com;
       location / {
           proxy_pass http://haoshuaicongedu-linux96;
       }
   }
    sendfile on;
    tcp_nopush on;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    ssl_prefer_server_ciphers on;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    gzip on;
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}
[root@elk92 ~]#
1.4 热加载 nginx 服务
[root@elk92 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@elk92 ~]# 
[root@elk92 ~]# nginx -s reload
[root@elk92 ~]#
1.5 访问页面测试
[root@elk92 ~]# for i in `seq 10`;do curl 10.0.0.92 -H 'host:  tomcat.haoshuaicongedu.com'; sleep 0.1;done
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
[root@elk92 ~]#

注意：如果出现问题，请将 “server.xml” 文件还原，并重启 tomcat 服务即可。

查看 91 节点 tomcat 日志：

[root@elk91 ~]# tail -100f /usr/local/apache-tomcat-11.0.5/logs/tomcat.haoshuaicongedu.com_access_log.2025-03-12.json 
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:05 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:07 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:29 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
...

查看 93 节点 tomcat 日志：

[root@elk93 ~]# tail -100f /usr/local/apache-tomcat-11.0.5/logs/tomcat.haoshuaicongedu.com_access_log.2025-03-12.json 
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:06 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:08 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:29 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:29 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
...

2. 采集 web 集群日志

2.1 采集 nginx 日志
[root@elk92 ~]# cat /etc/filebeat/config/05-modules-nginx-to-es.yaml 
# 配置模块功能
filebeat.config.modules:
  # 加载Filebeat配置文件目录的"modules.d"子目录下的所有"*.yml"文件
  path: ${path.config}/modules.d/*.yml
  # 是否支持热加载
  reload.enabled: true

output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux95-modules-nginx-%{+yyyy.MM.dd}

setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux95"
setup.template.pattern: "haoshuaicongedu-linux95-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk92 ~]# 
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-modules-nginx-to-es.yaml
2.2 采集 93 节点的 tomcat 日志
2.2.1 启动 Filebeat 实例
[root@elk93 ~]# cat /tmp/modules-tomcat-project-to-es.yaml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

processors:
  - decode_json_fields:
      fields: ["event.original"]
      target: ""
      overwrite_keys: true
      add_error_key: true
    # 转换数据类型
  - convert:
      # 定义字段的转换方式，其中from表示源字段，type表示要转换的类型。
      # 如果想要将转换的数据存储到一个新字段中，则可以使用"to"关键字，不写则就地更新。
      fields:
        - {from: "SendBytes", type: "long"}

output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux95-modules-tomcat-project-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux95"
setup.template.pattern: "haoshuaicongedu-linux95-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk93 ~]# 
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-project-to-es.yaml
2.2.2 启动 tomcat 模块
[root@elk93 ~]# yy /etc/filebeat/modules.d/tomcat.yml 
- module: tomcat
  log:
    enabled: true
    var.input: file
    var.paths:
      - /usr/local/apache-tomcat-11.0.5/logs/*.json 
    var.tz_offset: +08:00
[root@elk93 ~]#
2.3 采集 91 节点的 tomcat 日志
2.3.1 启动 Filebeat 实例
[root@elk91 ~]# cat /tmp/modules-tomcat-project-to-es.yaml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

processors:
  - decode_json_fields:
      fields: ["event.original"]
      target: ""
      overwrite_keys: true
      add_error_key: true
    # 转换数据类型
  - convert:
      # 定义字段的转换方式，其中from表示源字段，type表示要转换的类型。
      # 如果想要将转换的数据存储到一个新字段中，则可以使用"to"关键字，不写则就地更新。
      fields:
        - {from: "SendBytes", type: "long"}

output.elasticsearch:
  hosts:
  - 10.0.0.91:9200
  - 10.0.0.92:9200
  - 10.0.0.93:9200
  index: haoshuaicongedu-linux95-modules-tomcat-project-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "haoshuaicongedu-linux95"
setup.template.pattern: "haoshuaicongedu-linux95-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 0
[root@elk91 ~]# 
[root@elk91 ~]# filebeat -e -c /tmp/modules-tomcat-project-to-es.yaml
2.3.2 启动 tomcat 模块
[root@elk91 ~]# filebeat modules enable tomcat
Enabled tomcat
[root@elk91 ~]# 
[root@elk91 ~]# yy /etc/filebeat/modules.d/tomcat.yml 
- module: tomcat
  log:
    enabled: true
    var.input: file
    var.paths:
      - /usr/local/apache-tomcat-11.0.5/logs/*.json 
    var.tz_offset: +08:00
[root@elk91 ~]#
image-20250312225353889
暂无评论

发送评论 编辑评论 


				

			

						

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
							
													

					

				

			

			
			

				

					
				

			

				

											

							
							
						

																				
					
											
						

	

		
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
（╯‵□′）╯︵┴─┴
￣﹃￣
(/ω＼)
∠( ᐛ 」∠)＿
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ｀)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ(￣∇￣o)
ヾ(´･ ･｀｡)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò｡)
Σ(っ °Д °;)っ
( ,,´･ω･)ﾉ"(´っω･｀｡)
╮(╯▽╰)╭
o(*////▽////*)q
＞﹏＜
( ๑´•ω•) "(ㆆᴗㆆ)
😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣
Source: github.com/k4yt3x/flowerhd
	

	

		
颜文字
Emoji
小恐龙
花!
	


									

			

			
			
		

	






上一篇
下一篇