fluent-bitをDockerで動かす

プロジェクト構成

以下のように適当なディレクトリを作成する。

$ tree .
.
├── Dockerfile
├── application.conf
└── docker-compose.yml

0 directories, 3 files

Dockerfile:

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:2.21.3

COPY application.conf /fluent-bit/etc/application.conf

docker-compose.yml:

version: "3.8"
services:
  fluentbit:
    build: .
    command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/etc/application.conf"]
    volumes:
      - ./application.conf:/fluent-bit/etc/application.conf

application.conf:

[SERVICE]
    Flush 5
    Log_Level info

[INPUT]
    Name dummy
    Tag  *-firelens-*
    Dummy {"date":"2022-01-23T03:10:33.317817Z","source":"stdout","log":"time:2022-01-23T03:10:33+00:00\tprotocol:HTTP/1.1\tstatus:200\tsize:1450\treqsize:150\treferer:-\tvhost:10.10.18.102\treqtime:0.176\tcache:-\truntime:-\t"}

[OUTPUT]
    Name stdout
    Match *

実行方法

docker compose up を立ち上げると上記の [INPUT] に記述したdummyのinputが出力される。

$ docker compose up

fluent-bit-sandbox-fluentbit-1  | [0] *-firelens-*: [1668664184.902488203, {"date"=>"2022-01-23T03:10:33.317817Z", "source"=>"stdout", "log"=>"time:2022-01-23T03:10:33+00:00   protocol:HTTP/1.1       status:200      size:1450       reqsize:150     referer:-     vhost:10.10.18.102      reqtime:0.176   cache:- runtime:-       "}]
fluent-bit-sandbox-fluentbit-1  | [1] *-firelens-*: [1668664185.900364801, {"date"=>"2022-01-23T03:10:33.317817Z", "source"=>"stdout", "log"=>"time:2022-01-23T03:10:33+00:00   protocol:HTTP/1.1       status:200      size:1450       reqsize:150     referer:-     vhost:10.10.18.102      reqtime:0.176   cache:- runtime:-       "}]
fluent-bit-sandbox-fluentbit-1  | [2] *-firelens-*: [1668664186.900453923, {"date"=>"2022-01-23T03:10:33.317817Z", "source"=>"stdout", "log"=>"time:2022-01-23T03:10:33+00:00   protocol:HTTP/1.1       status:200      size:1450       reqsize:150     referer:-     vhost:10.10.18.102      reqtime:0.176   cache:- runtime:-       "}]
fluent-bit-sandbox-fluentbit-1  | [3] *-firelens-*: [1668664187.900391678, {"date"=>"2022-01-23T03:10:33.317817Z", "source"=>"stdout", "log"=>"time:2022-01-23T03:10:33+00:00   protocol:HTTP/1.1       status:200      size:1450       reqsize:150     referer:-     vhost:10.10.18.102      reqtime:0.176   cache:- runtime:-       "}]

解説

特に何も指定せずに docker compose up をすると /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf で起動をしている。

$ docker compose exec fluentbit /bin/bash
bash-4.2# ps aux | grep fluent-bit
root         1  0.2  0.5 2442384 46640 ?       Ssl  05:50   0:00 /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /fluent-bit/etc/fluent-bit.conf
root        41  0.0  0.0   8996   932 pts/1    S+   05:51   0:00 grep fluent-bit

デフォルトのconfigである /fluent-bit/etc/fluent-bit.conf はserverが立っており、tcpでやりとりができるようになっている。

$ docker compose exec fluentbit /bin/bash
bash-4.2# cat /fluent-bit/etc/fluent-bit.conf
[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[OUTPUT]
    Name cloudwatch
    Match   **
    region us-east-1
    log_group_name fluent-bit-cloudwatch
    log_stream_prefix from-fluent-bit-
    auto_create_group true