fluentbit with azure blob storage

installation

fluentbit install using helm chart

$ helm repo add fluent https://fluent.github.io/helm-charts
$ kubectl create ns logging
$ helm upgrade --install fluent-bit fluent/fluent-bit -n logging

create storage account and blob container

$ az storage account create -n fluentbitteststor -g fluentbittest-rg -l koreacentral --sku Standard_LRS

config for azure blob

$ kubectl edit configmap fluent-bit -n logging
...
    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

https://docs.fluentbit.io/manual/pipeline/outputs/azure_blob

full configmap

  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

    [INPUT]
        Name tail
        Path /var/log/*.log
        Tag system.*

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

restart daemonset

After modifying the fluent-bit configmap, run rollout like below.

$ kubectl rollout restart ds fluent-bit -n logging
daemonset.apps/fluent-bit restarted

storage browser