log transfer into Loki which is in external cluster

다른 cluster에 존재하는 loki에 loki와 다른 cluster에서 동작되는 container에 sidecar 형태의 promtail을 동작시켜 log를 전달
및 간단히 분석할수 있는 label 설정등에 대하여 알아보도록 하자.

먼저 환경은 다음과 같다.

• 가칭 loki cluster
  • grafana
  • loki
• 가칭 nginx-ingress
  • nginx-ingress

먼저 grafana는 접근이 되고 있고 loki도 외부로 연결이 가능하도록 ingress로 객체를 만들어서 연결을 시도해본다.
(꼭 ingress로 만들 필요는 없고 kubernetes loadbalancer 타입과 같은 외부로 노출시킬수 있는 환경이면 된다.)

loki access on external cluster (가칭 loki cluster)

다음과 같은 외부 주소로 접근을 해보면 접근이 되는지 확인할 수 있다

jacob@dubaek:~$ curl -sk https://loki.k8s.jacobbaek.com/loki/api/v1/status/buildinfo | jq
{
  "version": "2.4.2",
  "revision": "525040a32",
  "branch": "HEAD",
  "buildUser": "root@485c8dba2f87",
  "buildDate": "2022-01-12T16:57:10Z",
  "goVersion": ""
}

• https://grafana.com/docs/loki/latest/api/

가능하다면 직접 아래링크를 참고하여 data를 push 해보자.

• https://stackoverflow.com/questions/67316535/send-logs-directly-to-loki-without-use-of-agents

grafana@dev-grafana-659cf4cf59-cv79v:/usr/share/grafana$ date +%s
1647421583
grafana@dev-grafana-659cf4cf59-cv79v:/usr/share/grafana$ curl -v -H "Content-Type: application/json" -XPOST -s "http://loki.loki.svc.jacobcluster.local:3100/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "job": "test/test" }, "values": [ [ "1647421583000000000", "testval" ] ] }]}'
*   Trying 10.233.58.214:3100...
* TCP_NODELAY set
* Connected to loki.loki.svc.jacobcluster.local (10.233.58.214) port 3100 (#0)
> POST /loki/api/v1/push HTTP/1.1
> Host: loki.loki.svc.jacobcluster.local:3100
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 103
>
* upload completely sent off: 103 out of 103 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Wed, 16 Mar 2022 09:06:34 GMT
<
* Connection #0 to host loki.loki.svc.jacobcluster.local left intact

이후 아래와 같은 grafana browser로 query를 해보면 결과가 보여진다.

참고로 아래와 같이 ingress로 loki를 외부로 expose 한후에도 동일한 value가 push 된것을 확인할 수 있었다.

jacob@dubaek:~$ curl -vk -H "Content-Type: application/json" -XPOST -s "https://loki.k8s.jacobbaek.com/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "job": "test/test" }, "values": [ [ "1647421593000000000", "testval" ] ] }]}'
*   Trying 211.211.211.211:443...
* TCP_NODELAY set
* Connected to loki.k8s.jacobbaek.com (211.211.211.211) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Mar 14 08:41:06 2022 GMT
*  expire date: Mar 14 08:41:06 2023 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffd01dfb40)
> POST /loki/api/v1/push HTTP/2
> Host: loki.k8s.jacobbaek.com:443
> user-agent: curl/7.68.0
> accept: */*
> content-type: application/json
> content-length: 103
>
* We are completely uploaded and fine
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 204
< date: Wed, 16 Mar 2022 09:08:32 GMT
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host loki.k8s.jacobbaek.com left intact

setup promtail

sidecar 형태로 promtail을 특정 application에 붙일 예정이다.

• https://grafana.com/docs/loki/latest/clients/promtail/installation/#sidecar

실제 sidecar 형태는 기존 deployment 혹은 다른 resource 배포 yaml에 다음과 같은 container를 추가로 정의해야 한다.

spec:
  ...
  template:
    spec:
      serviceAccount: SERVICE_ACCOUNT
      serviceAccountName: SERVICE_ACCOUNT
      volumes:
      - name: logs
        hostPath: HOST_PATH
      - name: promtail-config
        configMap
          name: promtail-configmap
      containers:
      - name: promtail-container
         args:
         - -config.file=/etc/promtail/promtail.yaml
         volumeMounts:
         - name: logs
            mountPath: MOUNT_PATH
         - name: promtail-config
            mountPath: /etc/promtail
...

• 참고 : https://github.com/jafernandez73/grafana-loki/blob/master/docs/promtail-setup.md#example-1

nginx-ingress에 access log들을 loki로 push

sidecar 형태로 nginx-ingress deployment에 promtail을 동작시키고
/var/log/nginx-cache/*.log 파일들의 access log들을 loki로 push 후 grafana로 검색이 되도록 해보자.

먼저 sidecar로 동작될 promtail의 설정을 추가한다.
promtail-configmap을 추가하고 sidecar 형태로 동작될 container를 deployment에 추가해준다.
또한 log파일에 access가 중요하기에 initContainer로 mount 할 directory의 권한을 맞추고 volume mount가 되도록 한다.

테스트 당시 사용한 nginx-ingress 는 다음과 같다.

jacob@dubaek:~/workspace/yamls$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml

• https://kubernetes.github.io/ingress-nginx/deploy/

해당 파일을 다음과 같이 수정하여 ingress-nginx를 설치한다.

jacob@dubaek:~/workspace/yamls$ diff -u ingress-nginx-modified.yaml ingress-nginx.yaml
--- ingress-nginx-modified.yaml 2022-03-17 17:08:19.319134100 +0900
+++ ingress-nginx.yaml  2022-03-17 14:06:48.340590700 +0900
@@ -1,33 +1,4 @@
-# promtail configmap
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: promtail-configmap
-  namespace: ingress-nginx
-data:
-  promtail.yaml: |
-    client:
-    - url: "https://loki.k8s.jacobbaek.com/loki/api/v1/push"
-      tls_config:
-        insecure_skip_verify: true
-    positions:
-      filename: /tmp/positions.yaml
-    server:
-      http_listen_port: 9080
-      grpc_listen_port: 0
-    scrape_configs:
-    - job_name: nginx-ingress
-      static_configs:
-      - targets:
-          - localhost
-        labels:
-          job: nginx-ingress-access
-          env: cache-prd
-          host: nginx
-          __path__: /var/log/nginx-cache/*.log
-#
-# ################### Original
-#
----
+
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -352,13 +323,6 @@
         app.kubernetes.io/component: controller
     spec:
       dnsPolicy: ClusterFirst
-      initContainers:
-        - name: create-init-volume
-          image: busybox
-          command: ["/bin/chown", "-R", "101:82", "/var/log/nginx-cache"]
-          volumeMounts:
-          - name: cache-logs
-            mountPath: /var/log/nginx-cache
       containers:
         - name: controller
           image: k8s.gcr.io/ingress-nginx/controller:v1.1.1@sha256:0bc88eb15f9e7f84e8e56c14fa5735aaa488b840983f87bd79b1054190e660de
@@ -430,21 +394,10 @@
             - name: webhook-cert
               mountPath: /usr/local/certificates/
               readOnly: true
-            - name: cache-logs
-              mountPath: /var/log/nginx-cache
           resources:
             requests:
               cpu: 100m
               memory: 90Mi
-        - name: promtail-container
-          image: grafana/promtail
-          args:
-          - -config.file=/etc/promtail/promtail.yaml
-          volumeMounts:
-          - name: cache-logs
-            mountPath: /var/log/nginx-cache
-          - name: promtail-config
-            mountPath: /etc/promtail
       nodeSelector:
         kubernetes.io/os: linux
       serviceAccountName: ingress-nginx
@@ -453,12 +406,6 @@
         - name: webhook-cert
           secret:
             secretName: ingress-nginx-admission
-        - name: cache-logs
-          hostPath:
-            path: /var/log/nginx-cache
-        - name: promtail-config
-          configMap:
-            name: promtail-configmap
 ---
 # Source: ingress-nginx/templates/controller-ingressclass.yaml
 # We don't support namespaced ingressClass yet

아래와 같이 설치를 진행한다.

jacob@dubaek:~/workspace/yamls$ kubectl apply -f ingress-nginx-modified.yaml

이후 access log를 남기게 되면 다음과 같이 grafana에서 검색이 이루어진다.

참고
동일pod상에서 container간 동일한 파일을 read/write 할수 있는 과정이다.
이에 대한 이해가 사전에 이루어져야 한다.

• https://www.stratoscale.com/blog/kubernetes/kubernetes-how-to-share-disk-storage-between-containers-in-a-pod/