使用Logstash写数据

本节我们将配置Logstash，将http log发送到OpenSearch Serverless

Logstash的工作模式如下：

img

  • Inputs用于获取数据到Logstash，例如从日志文件，Kinesis，Kafka等数据源。
  • filters用于处理获取到的数据
  • outputs用于将数据写入到目的地，可以配置多个目标。

模拟http日志

我们将使用log generator来模拟一个http服务器产生的日志：

pip install log-generator

创建一个文件httpd-log-generator-schema.yml，里面内容如下：

name: Apache General Access
file: /home/ec2-user/environment/serverless-generators/httpd.log
format: "{log_ip} - - [{log_time} +0000] \"{log_method} {log_path} HTTP/1.1\" {log_status} {log_bytes}"
frequency:
  seconds: 5
offset:
  seconds: 0
jitter:
  seconds: 5
amount: 50
fields:
  log_ip:
    type: ip
  log_time:
    type: timestamp
    format: "%d/%b/%Y:%H:%M:%S"
  log_method:
    type: enum
    values: [POST, GET, PUT, PATCH, DELETE]
  log_path:
    type: enum
    values:
      - /auth
      - /alerts
      - /events
      - /playbooks
      - /lists
      - /fieldsets
      - /customers
      - /collectors
      - /parsers
      - /users
  log_status:
    type: enum
    values: [200, 201, 204, 300, 301, 400, 401, 403, 404, 500, 503]
  log_bytes:
    type: integer
    min: 2000
    max: 5000

运行log-generator:

log-generator httpd-log-generator-schema.yml

image-20230818112450631

这样/home/ec2-user/environment/serverless-generators/httpd.log下将会持续有日志写入。

运行Logstash

# 下载Logstash
wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz

tar -zxvf logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz
cd logstash-8.4.0/

# 更新logstash-output-opensearch插件到最新版本

./bin/logstash-plugin update logstash-output-opensearch

创建一个文件logstash-generator.conf，里面内容如下：

input {
    file {
        path => "/home/ec2-user/environment/serverless-generators/httpd.log"
        start_position => "beginning"
    }
}
filter {
    grok {
      match => { "message" => "%{HTTPD_COMMONLOG}"}
    }
}
output {
    opensearch {
        ecs_compatibility => disabled
        index => "logstash-ingest-%{+YYYY.MM.dd}"
        hosts => "https://<<YOUR_AMAZON_OPENSEARCH_SERVERLESS_HOST_ENDPOINT>>:443"
        auth_type => {
            type => 'aws_iam'
            region => 'us-east-1'
            service_name => 'aoss'
        }
        legacy_template => false
        default_server_major_version => 2
        timeout => 300
    }
}

将YOUR_AMAZON_OPENSEARCH_SERVERLESS_HOST_ENDPOINT替换为实例的OpenSearch URL。

运行logstash：

./bin/logstash -f logstash-generator.conf

进入到Dashboard，查询数据：

GET logstash-ingest-*/_search
{
    "query": {
        "match_all": {}
    }
}

结果类似于：

image-20230818113314131