FileBeat部署及基础使用

运维点滴记录 2022-08-27 企业信息化 84 0 百度已收录

ECS 云服务器 SWAS 轻量应用服务器 RDS 云数据库 Redis 云数据库 CDN 内容分发 OSS 对象存储 SLB 负载均衡 NAT 网关 DNS 云解析 MAIL 企业邮箱 WAF 应用防火墙 DDoS 高防 SMS 短信包 MK 云市场 XIN 心选 IM 商标注册 JZ 自营建站

一、FileBeat部署及基础使用

  • 软件包下载地址：
    ​​​https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.6-x86_64.rpm​​

1.部署FileBeat环境

yum -y localinstall filebeat-7.17.6-x86_64.rpm

2.修改FileBeat的配置⽂件

(1)编写测试的配置⽂件
mkdir /etc/filebeat/config
cat > /etc/filebeat/config/01-stdin-to-console.yml <<'EOF'
# 指定输⼊的类型
filebeat.inputs:
# 指定输⼊的类型为"stdin",表示标准输⼊
- type: stdin
# 指定输出的类型
output.console:
  # 打印漂亮的格式
  pretty: true
EOF

(2)运⾏filebeat实例
filebeat -e -c /etc/filebeat/config/01-stdin-to-console.yml 

(3)测试
⻅视频。

3.FileBeat的Log类型

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log

output.console:
  pretty: true

4.input的通配符案例

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log
    - /tmp/*.txt

output.console:
  pretty: true

5.input的通用字段案例

filebeat.inputs:
- type: log
  # 是否启动当前的输⼊类型，默认值为true
  enabled: true
  # 指定数据路径
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  # 给当前的输⼊类型搭上标签
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
  # ⾃定义字段
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  # 将⾃定义字段的key-value放到顶级字段.
  # 默认值为false，会将数据放在⼀个叫"fields"字段的下⾯.
  fields_under_root: true

output.console:
  pretty: true

6.日志过滤案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test/*.log
  # 注意，⿊⽩名单均⽀持通配符,⽣产环节中不建议同时使⽤
  # 指定⽩名单，包含指定的内容才会采集，且区分⼤⼩写!
  include_lines: ['^ERR', '^WARN','oldboyedu']
  # 指定⿊名单，排除指定的内容
  exclude_lines: ['^DBG',"linux","oldboyedu"]

output.console:
  pretty: true

7.将数据写入ES案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]

8.自定义es索引名称

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

9.多个索引写入案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

10.自定义分片和副本案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"
# 覆盖已有的索引模板
setup.template.overwrite: false
# 配置索引模板
setup.template.settings:
  # 设置分⽚数量
  index.number_of_shards: 3
  # 设置副本数量，要求⼩于集群的数量
  index.number_of_replicas: 2

11.FileBeat实现日志聚合到本地

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"

output.file:
  path: "/tmp/filebeat"
  filename: oldboyedu-linux80
  # 指定⽂件的滚动⼤⼩，默认值为20MB
  rotate_every_kb: 102400
  # 指定保存的⽂件个数，默认是7个，有效值为2-1024个
  number_of_files: 300
  # 指定⽂件的权限，默认权限是0600

12.FileBeat实现日志聚合到ES集群

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"
  tags: ["aaa"]

- type: tcp
  host: "0.0.0.0:8000"
  tags: ["bbb"]

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
  indices:
    - index: "oldboyedu-linux80-elk-aaa-%{+yyyy.MM.dd}"
      when.contains:
        tags: "aaa"

    - index: "oldboyedu-linux80-elk-bbb-%{+yyyy.MM.dd}"
      when.contains:
        tags: "bbb"

setup.ilm.enabled: false
setup.template.name: "oldboyedu-linux80-elk"
setup.template.pattern: "oldboyedu-linux80-elk*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0