filebeat

日志采集客户端。

安装

二进制安装

shell

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.29-darwin-aarch64.tar.gz
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.29-darwin-aarch64.tar.gz.sha512

shasum -a 512 -c filebeat-7.17.29-darwin-aarch64.tar.gz.sha512
tar -xzf filebeat-7.17.29-darwin-aarch64.tar.gz

xattr -d -r com.apple.quarantine filebeat-7.17.29-darwin-aarch64
cd filebeat-7.17.29-darwin-aarch64

系统和架构包

OS	Arch
windows	x86_64
darwin	aarch64

测试配置语法 ./filebeat test config -c filebeat.yml

hello world

shell

cat << EOF > dev-filebeat.yml

filebeat.inputs:
  - type: stdin
    encoding: utf-8

output.console:
  pretty: true

logging.level: warning

EOF

从 stdin 读入样本数据解释后输出到 stdout: echo hello | ./filebeat -c dev-filebeat.yml -e

容器安装

shell

docker pull docker.elastic.co/beats/filebeat:7.17.29

export PATH_DATA=/d/data/filebeat
export PATH_LOG=/d/log/filebeat
export PATH_ETC=/d/etc/filebeat

# 按需修改
export PATH_LOGS=/d/log

mkdir -p $PATH_DATA
mkdir -p $PATH_LOG
mkdir -p $PATH_ETC

mkdir -p $PATH_LOGS

# 按需修改样本配置
curl -o $PATH_ETC/filebeat.yml -L -O https://raw.githubusercontent.com/elastic/beats/8.17/deploy/docker/filebeat.docker.yml

# 按需修改样本配置
cat << EOF >> $PATH_ETC/filebeat.yml
...
EOF

chmod 744 $PATH_ETC/filebeat.yml

docker run \
--restart unless-stopped \
-d \
--name=filebeat \
-m 128MB \
-v $PATH_DATA:/usr/share/filebeat/data \
-v $PATH_LOG:/usr/share/filebeat/logs \
-v $PATH_ETC/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
-v $PATH_LOGS:/d/log:ro \
-e --strict.perms=false \
docker.elastic.co/beats/filebeat:7.17.29 filebeat

参考配置文件 filebeat.config.yml

核心概念

component
input
harvester

采集 spring 日志

自定义解释 spring 日志，参考 logback 配置 /path/to/<spring_project_root>/src/main/resources/logback-spring.xml 属性值 LOG_PATTERN

xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <property
            name="LOG_PATH"
            value="/v/log/spring" />

    <property
            name="LOG_PATTERN"
            value="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level %class{20}:%method:%line - [%thread] traceId:%X{traceId} - %msg%n" />

    <appender
            name="CONSOLE"
            class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>


    <!-- File appender -->
    <appender
            name="FILE"
            class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_PATH}/app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_PATH}/app.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="FILE" />
    </root>

    <logger
            name="org.springframework"
            level="WARN" />
</configuration>

修改或添加 filebeat.yml ，要点：

支持解释 spring 堆栈多行日志，每行日志按时间戳 RFC3339 作为行开始切割 parsers.multiline
切割后按 logback pattern 拆分字段 processors.dissect
从日志完整路径解释出 appname processors.dissect
dissect tokenizer 在线校验 https://dissect-tester.jorgelbg.me

yaml

logging.level: warning

filebeat.inputs:
  - type: filestream
    id: applog
    paths:
      - /v/log/**/*.log
    parsers:
      - multiline:
          type: pattern
          # match logs: spring/logback, Go, ElasticSearch etc.
          pattern: '2\[*\d{3}-[012]\d-[0123]\d(T| )'
          negate: true
          match: after

setup.template.settings:
  index.number_of_shards: 1
processors:
  - dissect:
      tokenizer: "%{timestamp} %{level} %{class}:%{method}:%{line|integer} - [%{thread}] - %{content}"
      field: "message"
      target_prefix: ""
      overwrite_keys: true

  - dissect:
      tokenizer: "/v/log/%{appname}/%{log_filename}"
      field: "log.file.path"
      target_prefix: ""
      overwrite_keys: true

  - timestamp:
      field: timestamp
      layouts:
        - "2006-01-02T15:04:05.999-07:00"
      test:
        - "2024-12-27T22:46:06.684+08:00"

同时支持解释 Go 和 Java/Spring 日志

yaml

processors:
  - add_fields:
      target: ""
      fields:
        thread: ""
        class: ""

  - dissect:
      description: "extract the appname field from full log path"
      tokenizer: "/v/log/%{appname}/%{log_filename}"
      field: "log.file.path"
      target_prefix: ""
      overwrite_keys: true
      ignore_missing: true
      ignore_failure: true

  - dissect:
      description: "extract fields for Java/Spring"
      tokenizer: "%{timestamp} %{level} %{class}:%{method}:%{line|integer} - [%{thread}] traceId:%{traceId} - %{content}"
      field: "message"
      target_prefix: ""
      overwrite_keys: true
      ignore_missing: true
      ignore_failure: true
      when:
        regexp:
          message: " traceId:"

  - dissect:
      description: "extract fields for Go"
      when:
        equals:
          thread: ""
          class: ""
      tokenizer: "%{timestamp}\t%{level}\t%{sourcefile}:%{line}\t%{content}"
      field: "message"
      target_prefix: ""
      overwrite_keys: true
      ignore_missing: true
      ignore_failure: true

常改配置

忽略旧日志数据

yaml

filebeat.inputs:
  - type: filestream
    id: biz_app
    paths:
      - /d/log/foo/*.log
    # parse multiple lines
    parsers:
      - multiline:
          type: pattern
          pattern: '2(\[|-)*\d{3}-[012]\d-[0123]\d(T| )'
          negate: true
          match: after
          timeout: 3s

    # https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-log.html
    # scan_frequency: 10s

    # close_inactive: 10m

    # https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-filestream.html
    # ignore_older: 60m

限制使用系统资源上限

CPU/内存：以容器实例方式部署，通过限制实例间接限制。

带宽

yaml

processors:
  - rate_limit:
      limit: "10000/m"

注：超过阈值会丢弃数据。

https://www.elastic.co/guide/en/beats/filebeat/7.17/rate-limit.html

yaml

output.elasticsearch:
  compression_level: 0 # 1-9, 1 for best speed, 9 for best compression
  bulk_max_size: 50 # default is 50 on v7.x, 1600 on v8.x
  backoff.init: 1s
  backoff.max: 60s
  timeout: 90

https://www.elastic.co/guide/en/beats/filebeat/7.17/elasticsearch-output.html#_bulk_max_size

修改系统网络堆 https://www.elastic.co/guide/en/beats/filebeat/7.17/bandwidth-throttling.html

修改内部发送队列 https://www.elastic.co/guide/en/beats/filebeat/7.17/configuring-internal-queue.html

yaml

queue.mem:
  events: 4096 # default is 4096 on v7.x, 3200 on v8.x
  flush.min_events: 2048 # 2048 on v7.x, 1600 on v8.x
  flush.timeout: 1s # 1s on v7.x, 10s on v8.x

docker 环境下修正解释 hostname

yaml

# fix hostname
- add_kubernetes_metadata:
    - drop_fields:
        fields: ["host.name"]
        ignore_missing: true
    - copy_fields:
        fields:
          - from: kubernetes.node.name
            to: host.name
        fail_on_error: false
        ignore_missing: true

https://github.com/elastic/beats/issues/13589#issuecomment-688741290

添加 host IP address

yaml

# add host ip address
- add_host_metadata:
    ip_fields: ["source.ip", "host.ip"]
    host_type: "source"
    netinfo.enabled: true

使用多个家 es 实例作为 output

yaml

output.elasticsearch:
  hosts: ["localhost:9201", "localhost:9202", "localhost:9203"]

使用内置模块采集常见开源组件日志

filebeat 7.x 内置多个模块，默认支持解释常见开源组件日志，如 NGINX, Redis, MySQL, PostgreSQL 等等。

采集 Redis 日志为例

启用 redis 模块

shell

filebeat modules list
filebeat modules enable redis

# or docker
ansible your-host -a 'docker exec -it filebeat sh  -c "filebeat modules enable redis"'

上一步实际修改 filebeat 根目录对应配置文件名，来启用指定模块： modules.d/redis.yml.disabled => modules.d/redis.yml

配合 ELK 的 Kibana 组件使用，创建相关监控仪表面板和索引模板:

shell

# Load Dashboards and Index Templates
filebeat setup --dashboards
filebeat setup --index-management

修改 filebeat.yml , 增加或修改以下:

yaml

filebeat.modules:
  - module: redis
    log:
      enabled: true
      var.paths: ["c:/d/log/redis/redis-server.log"]
    slowlog:
      enabled: true
      var.hosts: ["localhost:6379"]
      var.password: "secret"

重启 filebeat ，打开 Kibana - Analytics - Dashboard 搜 [Filebeat Redis] ECS，即可看到预生成仪表面板。

另见

https://www.elastic.co/guide/en/beats/filebeat/8.17/filebeat-overview.html

Filebeat vs fluentbit

Features	filebeat (elastic/beats)	fluentbit
Language	Go	C
ensure at-least-once delivery/no data loss	yes
Hot reload	yes	yes
Accept file does not exists	yes
Parse JSON in regular expression	yes
highlights	- easy-to-use	- Use system memory (heap) for high performance - MessagePack - Supports lots of inputs, filters and oupouts
Sizes	Version 8.16.1 Docker image 354 MB	macOS binary 151 MB

filebeat ​

安装 ​

二进制安装 ​

容器安装 ​

核心概念 ​

采集 spring 日志 ​

同时支持解释 Go 和 Java/Spring 日志 ​

常改配置 ​

忽略旧日志数据 ​

限制使用系统资源上限 ​

docker 环境下修正解释 hostname ​

添加 host IP address ​

使用多个家 es 实例作为 output ​

使用内置模块采集常见开源组件日志 ​

采集 Redis 日志为例 ​

另见 ​

Filebeat vs fluentbit ​

filebeat

安装

二进制安装

容器安装

核心概念

采集 spring 日志

同时支持解释 Go 和 Java/Spring 日志

常改配置

忽略旧日志数据

限制使用系统资源上限

docker 环境下修正解释 hostname

添加 host IP address

使用多个家 es 实例作为 output

使用内置模块采集常见开源组件日志

采集 Redis 日志为例

另见

Filebeat vs fluentbit