学习 ELK .001

 

官网:https://www.elastic.co/cn/elasticsearch/

Elastic Stack

1, 下载 :https://www.elastic.co/cn/downloads/elasticsearch

sudo mkdir /works
sudo chown -R chengchao:chengchao /works
mkdir -p /works/Downloads/elasticsearch
mkdir -p /works/apps
cd /works/Downloads/elasticsarch
wget -c https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.1.1-linux-x86_64.tar.gz
tar -zxvf elasticsearch-8.1.1-linux-x86_64.tar.gz  -C /works/apps/
cd /works/apps

3, 本地集群启动

bin/elasticsearhc
bin/elasticsearch -Ehttp.port=8200 -Epath.data=node2
bin/elasticsearch -Ehttp.port=7200 -Epath.data=node3


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.

ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  qWNrR05afzqrxrTJB+y5

ℹ️  HTTP CA certificate SHA-256 fingerprint:
  c037a75233e050300d50635626dd2ce6eff495559889b326b944d368591ef31a

ℹ️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjEuMSIsImFkciI6WyIxOTIuMTY4LjMuMjAzOjkyMDAiXSwiZmdyIjoiYzAzN2E3NTIzM2UwNTAzMDBkNTA2MzU2MjZkZDJjZTZlZmY0OTU1NTk4ODliMzI2Yjk0NGQzNjg1OTFlZjMxYSIsImtleSI6Ik5pLUV6bjhCOFB0YWxMYng2VWFPOnFxTjBVWEwyVDRDWEEzN1ZxRFRaM1EifQ==

ℹ️  Configure other nodes to join this cluster:
• On this node:
  ⁃ Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
  ⁃ Uncomment the transport.host setting at the end of config/elasticsearch.yml.
  ⁃ Restart Elasticsearch.
• On other nodes:
  ⁃ Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


4, 验证

https://192.168.3.203:9200/ https://192.168.3.203:9200/_cat/nodes

http://ip:9200/_cluster/status

Elasticsearch 常用术语

  • Document 文档数据
  • Index 索引
  • Type 索引中的数据类型
  • Field 字段,文档的属性
  • Query DSL 查询语法

CURD

ES create

POST /account/person/1

{
    "name" : "John",
    "lastname" : "Doe",
    "job_description" : "Systems administrator and Linux specialit"
}

{
    "_index" : "accounts",
    "_type" : "person",
    "_id" : "1",
    "_version" : 1,
    "result" : "created",
    "_shards" : {
        "total" : 2,
        "successful" : 1,
        "failed" : 0
    }
}

GET /accounts/person/1

POST /accounts/person/1/_update
{
    "doc" : {
        "job_description" : " Linux specialist"
    }
}

DELETE /accounts/person/1

Query String 

GET /accounts/person/_search?q=john

GET /accounts/person/_search

{
    "query" : {
        "term" : {
            "name" : {
                "value" : "john"
            }
        }
    }
}

Kibana

官网: https://www.elastic.co/cn/kibana/

下载: https://www.elastic.co/cn/downloads/kibana

sudo mkdir /works
sudo chown -R chengchao:chengchao /works
mkdir -p /works/Downloads/kibana
mkdir /works/apps
cd /works/Downloads/kibana
wget -c https://artifacts.elastic.co/downloads/kibana/kibana-8.1.1-linux-x86_64.tar.gz
tar -zxvf kibana-8.1.1-linux-x86_64.tar.gz -C /works/apps/
cd /works/apps/kibana-8.1.1/config

配置 kibana.yml


i Kibana has not been configured.

Go to http://localhost:5601/?code=512496 to get started.

bin/elasticsearch-reset-password --username kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [kibana_system] user successfully reset.
New value: fM9h_VNU76D+4WOo0u4X

验证:http://192.168.3.202:5601

使用 elastic 登录。

Beats

Filebeat Input 配置简介

yaml 语法:

filebeat.prospectors:
  - input_type: log
    paths:
      - /var/log/apache/httpd-*.log
  - input_type: log
    paths:
      - /var/log/messages
      - /var/log/*.log

input_type :

  • log
  • stdin

Filebeat Output 配置简介

  • Console
  • Elasticsearch
  • Logstash
  • Kafka
  • Redis
  • File

例子:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "s3cr3t"
  
output.consoel:
  pretty: true

Filebeat Filter 配置简介

  • Input 时处理
    • include_lines
    • exclude_lines
    • exclude_files
  • Output 前处理 – Processor
    • drop_event
    • drop_fields
    • decode_json_fields
    • include_fields

Packetbeat 解析 http 协议

  • 解析 elasticsearch http 请求
pakcetbeat.interfaces.device: lo0
packetbeat.protocols.http:
  ports: [9200]
  send_request: true
  include_body_for: ['application/json', 'x-ww w-form-urlencoded']
output.console:
  pretty: true

运行

sudo ./packetbeat -e -c es.yml -strict.perms=false

logstash

output {
    stdout {
        codec => rebydebug
    }
}

运行:

head -n 2 logs/nginx.log | bin/logstash -f nginx_logstash.conf 

参考(照抄)


If you like TeXt, don’t forget to give me a star. :star2:

Star This Project