filter 决定字段干不干净。流水线按这个顺序:能 JSON 就 JSON → 否则 Grok → Date 校正 @timestamp → Mutate 清理 → 需要再 GeoIP/UA。
Grok
非结构化日志基本靠它:
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}"
}
tag_on_failure => ["_grokparsefailure"]
}
}
Nginx access 一类:
filter {
grok {
match => {
"message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status:int} %{NUMBER:bytes:int}'
}
}
}
多条 pattern 时把最常见的放前面,并 break_on_match => true:
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}",
"%{IP:client_ip} %{GREEDYDATA:msg}"
]
}
break_on_match => true
}
自定义 pattern 丢到目录里:
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{APP_TIME:timestamp} %{GREEDYDATA:msg}" }
}
# /etc/logstash/patterns/app
APP_TIME \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
调试用 Kibana 的 Grok Debugger,或网页 grokdebug。写复杂正则前先确认有没有现成 pattern。
JSON / Date
filter {
json {
source => "message"
skip_on_invalid_json => true
}
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "ISO8601", "UNIX"]
timezone => "Asia/Shanghai"
target => "@timestamp"
}
}
@timestamp 不对,Kibana 时间轴会错位——国内服务器尤其注意 timezone。JSON 解析失败会有 _jsonparsefailure。
Mutate
日常改字段几乎都是它:
filter {
mutate {
add_field => { "env" => "prod" }
remove_field => ["host", "agent", "ecs", "timestamp"]
rename => { "client_ip" => "ip" }
convert => {
"status" => "integer"
"bytes" => "integer"
"duration" => "float"
}
lowercase => ["level"]
gsub => [
"path", "\\\\", "/"
]
split => { "tags_raw" => "," }
}
}
类型不对后面聚合会很疼,数字字段尽早 convert。
GeoIP / UserAgent
filter {
if [ip] {
geoip {
source => "ip"
target => "geoip"
fields => ["city_name", "country_name", "location"]
}
}
if [user_agent] {
useragent {
source => "user_agent"
target => "ua"
}
}
}
不是所有管道都需要,按查询需求加,省 CPU。
条件与丢弃
filter {
if [status] =~ /^5/ {
mutate { add_tag => ["server_error"] }
}
if [path] == "/health" or [path] == "/metrics" {
drop {}
}
if "_grokparsefailure" in [tags] {
mutate { add_field => { "parse_error" => "true" } }
}
}
健康检查日志量很大,能 drop 就 drop。也可用 prune 批量删字段。
一条比较完整的 filter
filter {
if [fields][format] == "json" {
json { source => "message" }
} else {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:level}\] %{GREEDYDATA:msg}"
}
}
}
date {
match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
timezone => "Asia/Shanghai"
}
mutate {
lowercase => ["level"]
remove_field => ["timestamp"]
}
if [ip] {
geoip { source => "ip"; target => "geoip" }
}
}
原则:先保证 @timestamp 和关键维度字段(level、status、service)正确,再谈富化。Grok 写到能用就行,别炫技。
转载请注明:周志洋的博客 » Logstash过滤器插件详解


