ELK日誌收集記錄

logstash在需要收集日誌的伺服器里運行，將日誌數據發送給es

在kibana頁面查看es的數據

es和kibana安裝：

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

cat << EOF >/etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

yum install -y --enablerepo=elasticsearch elasticsearch

# 安裝完成後，在終端里可以找到es的密碼
# 修改密碼：'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'
# config file: /etc/elasticsearch/elasticsearch.yml
# network.host: 0.0.0.0 允許其他伺服器訪問
# http.port 修改成可以外部訪問的埠

# 啟動es
systemctl start elasticsearch.service

# 測試是否可以訪問：curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:es_host
# 如果要在其他伺服器里訪問的話，需要先把證書移過去：/etc/elasticsearch/certs/http_ca.crt，直接複製證書的內容，在客戶端保存成一個證書文件即可
# 在客戶端里測試是否可以訪問：curl --cacert path_to_ca.crt -u elastic https://localhost:es_host

# install kibana
cat << EOF >/etc/yum.repos.d/kibana.repo
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

# kibana和es可以安裝到同一臺伺服器
yum install -y kibana
# /etc/kibana/kibana.yml 修改server.port為外部可以訪問的埠，server.host修改為0.0.0.0允許其他伺服器訪問，elasticsearch部分的可以先不用設置，
# root用戶使用：/usr/share/kibana/bin/kibana --allow-root
systemctl start kibana.service
# 首次打開kibana頁面需要添加elastic的token，使用如下命令生成token
# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
# 登錄的時候也需要es的用戶名和密碼
# 登錄成功之後，/etc/kibana/kibana.yml的底部會自動添加elasticsearch的連接信息

需要收集日誌的伺服器里安裝logstash：

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

cat <<EOF > /etc/yum.repos.d/logstash.repo
[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install -y logstash
ln -s /usr/share/logstash/bin/logstash /usr/bin/logstash

# install filebeat
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat <<EOF > /etc/yum.repos.d/filebeat.repo
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum install -y filebeat
ln -s /usr/share/filebeat/bin/filebeat /usr/bin/logstash

#filebeat->logstash->ES
#filebeat從具體目錄里拿文件的內容發送給logstash，logstash將數據發送給es

midr -m 777 -p /data/logstash

cat <<EOF >/data/logstash/filebeat.conf
filebeat.inputs:
- type: log
  paths:
    - /your_log_path/*.log
output.logstash:
  hosts: ["127.0.0.1:5044"]
EOF


cat <<EOF >/data/logstash/logstash.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
    client_inactivity_timeout => 600
  }
}

filter{
  mutate{
    remove_field => ["agent"]
    remove_field => ["ecs"]
    remove_field => ["event"]
    remove_field => ["tags"]
    remove_field => ["@version"]
    remove_field => ["input"]
    remove_field => ["log"]
  }
}

output {
  elasticsearch {
    hosts => ["https://es_ip_address:es_port"]
    index => "log-from-logstash"
    user => "es_user_name"
    password => "es_password"
    ssl_certificate_authorities => "path_to_es_http_ca.crt"
  }
}
EOF

#es_http_ca.crt的內容和es伺服器里的/etc/elasticsearch/certs/http_ca.crt內容相同
#filter里移除一些不必要的欄位

#啟動
logstash -f /data/logstash/logstash.conf >/dev/null 2>&1 &
filebeat -e -c /data/logstash/filebeat.conf >/dev/null 2>&1 &

啟動之後，filebeat.conf里配置的日誌路徑里可以copy一些文件做測試，或者已經有一些日誌文件的話，都可以在kabana里看到配置的index被自動創建：

 創建一個DataView就可以查看index里的文檔內容：

 在Discover里選擇配置的dataview查看數據：