FileBeat部署及基础使用

一、FileBeat部署及基础使用

  • 软件包下载地址:
    ​​​https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.6-x86_64.rpm​​

1.部署FileBeat环境

yum -y localinstall filebeat-7.17.6-x86_64.rpm

2.修改FileBeat的配置⽂件

(1)编写测试的配置⽂件
mkdir /etc/filebeat/config
cat > /etc/filebeat/config/01-stdin-to-console.yml <<'EOF'
# 指定输⼊的类型
filebeat.inputs:
# 指定输⼊的类型为"stdin",表示标准输⼊
- type: stdin
# 指定输出的类型
output.console:
# 打印漂亮的格式
pretty: true
EOF

(2)运⾏filebeat实例
filebeat -e -c /etc/filebeat/config/01-stdin-to-console.yml

(3)测试
⻅视频。

3.FileBeat的Log类型

filebeat.inputs:
- type: log
paths:
- /tmp/test.log

output.console:
pretty: true

4.input的通配符案例

filebeat.inputs:
- type: log
paths:
- /tmp/test.log
- /tmp/*.txt

output.console:
pretty: true

5.input的通用字段案例

filebeat.inputs:
- type: log
# 是否启动当前的输⼊类型,默认值为true
enabled: true
# 指定数据路径
paths:
- /tmp/test.log
- /tmp/*.txt
# 给当前的输⼊类型搭上标签
tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
# ⾃定义字段
fields:
school: "北京昌平区沙河镇"
class: "linux80"

- type: log
enabled: true
paths:
- /tmp/test/*/*.log
tags: ["oldboyedu-python","云原⽣开发"]
fields:
name: "oldboy"
hobby: "linux,抖⾳"
# 将⾃定义字段的key-value放到顶级字段.
# 默认值为false,会将数据放在⼀个叫"fields"字段的下⾯.
fields_under_root: true

output.console:
pretty: true

6.日志过滤案例

filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/test/*.log
# 注意,⿊⽩名单均⽀持通配符,⽣产环节中不建议同时使⽤
# 指定⽩名单,包含指定的内容才会采集,且区分⼤⼩写!
include_lines: ['^ERR', '^WARN','oldboyedu']
# 指定⿊名单,排除指定的内容
exclude_lines: ['^DBG',"linux","oldboyedu"]

output.console:
pretty: true

7.将数据写入ES案例

filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/test.log
- /tmp/*.txt
tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
fields:
school: "北京昌平区沙河镇"
class: "linux80"

- type: log
enabled: true
paths:
- /tmp/test/*/*.log
tags: ["oldboyedu-python","云原⽣开发"]
fields:
name: "oldboy"
hobby: "linux,抖⾳"
fields_under_root: true

output.elasticsearch:
hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]

8.自定义es索引名称

filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/test.log
- /tmp/*.txt
tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
fields:
school: "北京昌平区沙河镇"
class: "linux80"

- type: log
enabled: true
paths:
- /tmp/test/*/*.log
tags: ["oldboyedu-python","云原⽣开发"]
fields:
name: "oldboy"
hobby: "linux,抖⾳"
fields_under_root: true

output.elasticsearch:
enabled: true
hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式
setup.template.pattern: "oldboyedu-linux*"

9.多个索引写入案例

filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/test.log
- /tmp/*.txt
tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
fields:
school: "北京昌平区沙河镇"
class: "linux80"

- type: log
enabled: true
paths:
- /tmp/test/*/*.log
tags: ["oldboyedu-python","云原⽣开发"]
fields:
name: "oldboy"
hobby: "linux,抖⾳"
fields_under_root: true

output.elasticsearch:
enabled: true
hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
# index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
indices:
- index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 匹配指定字段包含的内容
when.contains:
tags: "oldboyedu-linux80"
- index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
when.contains:
tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式
setup.template.pattern: "oldboyedu-linux*"

10.自定义分片和副本案例

filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/test.log
- /tmp/*.txt
tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
fields:
school: "北京昌平区沙河镇"
class: "linux80"

- type: log
enabled: true
paths:
- /tmp/test/*/*.log
tags: ["oldboyedu-python","云原⽣开发"]
fields:
name: "oldboy"
hobby: "linux,抖⾳"
fields_under_root: true

output.elasticsearch:
enabled: true
hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
# index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
indices:
- index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 匹配指定字段包含的内容
when.contains:
tags: "oldboyedu-linux80"
- index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
when.contains:
tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式
setup.template.pattern: "oldboyedu-linux*"
# 覆盖已有的索引模板
setup.template.overwrite: false
# 配置索引模板
setup.template.settings:
# 设置分⽚数量
index.number_of_shards: 3
# 设置副本数量,要求⼩于集群的数量
index.number_of_replicas: 2

11.FileBeat实现日志聚合到本地

filebeat.inputs:
- type: tcp
host: "0.0.0.0:9000"

output.file:
path: "/tmp/filebeat"
filename: oldboyedu-linux80
# 指定⽂件的滚动⼤⼩,默认值为20MB
rotate_every_kb: 102400
# 指定保存的⽂件个数,默认是7个,有效值为2-1024个
number_of_files: 300
# 指定⽂件的权限,默认权限是0600

12.FileBeat实现日志聚合到ES集群

filebeat.inputs:
- type: tcp
host: "0.0.0.0:9000"
tags: ["aaa"]

- type: tcp
host: "0.0.0.0:8000"
tags: ["bbb"]

output.elasticsearch:
enabled: true
hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
indices:
- index: "oldboyedu-linux80-elk-aaa-%{+yyyy.MM.dd}"
when.contains:
tags: "aaa"

- index: "oldboyedu-linux80-elk-bbb-%{+yyyy.MM.dd}"
when.contains:
tags: "bbb"

setup.ilm.enabled: false
setup.template.name: "oldboyedu-linux80-elk"
setup.template.pattern: "oldboyedu-linux80-elk*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0