Using Elastic Stack, Filebeat (for log aggregation)

0

In my last article I described how I used ElasticSearch, Fluentd and Kibana (EFK). Besides log aggregation (getting log information available at a centralized location), I also described how I created some visualizations within a dashboard.
[https://technology.amis.nl/2019/05/06/using-elasticsearch-fluentd-and-kibana-for-log-aggregation/]

In a new series of articles, I will dive into using Filebeat and Logstash (from the Elastic Stack) to do the same.

In this article I will talk about the installation and use of Filebeat (without Logstash).

EFK

One popular centralized logging solution is the Elasticsearch, Fluentd, and Kibana (EFK) stack.

Fluentd
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
[https://www.fluentd.org/]

ELK Stack

“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana.
[https://www.elastic.co/what-is/elk-stack]

In my previous article I already spoke about Elasticsearch (a search and analytics engine) <Store, Search, and Analyze> and Kibana (which lets users visualize data with charts and graphs in Elasticsearch) <Explore, Visualize, and Share>.

Elastic Stack

The Elastic Stack is the next evolution of the ELK Stack.
[https://www.elastic.co/what-is/elk-stack]

Logstash
Logstash <Collect, Enrich, and Transport> is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch.
[https://www.elastic.co/what-is/elk-stack]

Beats
In 2015, a family of lightweight, single-purpose data shippers were introduced into the ELK Stack equation. They are called Beats <Collect, Parse, and Ship>.
[https://www.elastic.co/what-is/elk-stack]

Filebeat
Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them to either to Elasticsearch or Logstash for indexing.
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html]

This time I won’t be using Fluentd for log aggregation. I leave it up to you to decide which product is most suitable for (log) data collection in your situation.

In a previous series of articles, I talked about an environment, I prepared on my Windows laptop, with a guest Operating System, Docker and Minikube available within an Oracle VirtualBox appliance , with the help of Vagrant. And now also I will be using that environment.

Log aggregation

In a containerized environment like Kubernetes, Pods and the containers within them, can be created and deleted automatically via ReplicaSet’s. So, it’s not always easy to now where in your environment, you can find the log file that you need, to analyze a problem that occurred in a particular application. Via log aggregation, the log information becomes available at a centralized location.

In the table below, you can see an overview of the booksservice Pods that are present in the demo environment, including the labels that are used:

Spring Boot applicationService endpointPodNamespaceLabel key
EnvironmentDatabaseappversionenvironment
DEVH2 in memoryhttp://localhost:9010/booksbooksservice-v1.0-*nl-amis-developmentbooksservice1.0development
http://localhost:9020/booksbooksservice-v2.0-*nl-amis-developmentbooksservice2.0development
TSTMySQLhttp://localhost:9110/booksbooksservice-v1.0-*nl-amis-testingbooksservice1.0testing

Labels are key/value pairs that are attached to objects, such as pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time. Each object can have a set of key/value labels defined. Each Key must be unique for a given object.
[https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/]

Elastic Stack installation order

Install the Elastic Stack products you want to use in the following order:

[https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html]

When installing Filebeat, installing Logstash (for parsing and enhancing the data) is optional.

I wanted to start simple, so I started with the installation of Filebeat (without Logstash).

Installing Filebeat

You can use Filebeat Docker images on Kubernetes to retrieve and ship the container logs.

You deploy Filebeat as a DaemonSet to ensure there’s a running instance on each node of the cluster.

The Docker logs host folder (/var/lib/docker/containers) is mounted on the Filebeat container. Filebeat starts an input for the files and begins harvesting them as soon as they appear in the folder.

I found an example Kubernetes manifest file in order to setup Filebeat, in the Elastic Filebeat documentation:

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.3/deploy/kubernetes/filebeat-kubernetes.yaml

[https://www.elastic.co/guide/en/beats/filebeat/7.3/running-on-kubernetes.html]

In line with how a previously set up my environment , from this example manifest file, I created the following manifest files (with my own namespace and labels):

  • configmap-filebeat.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-configmap
  namespace: nl-amis-logging
  labels:
    app: filebeat
    version: "1.0"
    environment: logging
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      host: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

Remark about using a ConfigMap:
By using a ConfigMap, you can provide configuration data to an application without storing it in the container image or hardcoding it into the pod specification.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/]

The data field (with key-value pairs) contains the configuration data. In our case the ConfigMap holds information in the form of the content of a configuration file (filebeat.yml).

Later on, I will change this and create the ConfigMap from a file, as I also did when I used Fluentd (see previous article).
[https://technology.amis.nl/2019/04/23/using-vagrant-and-shell-scripts-to-further-automate-setting-up-my-demo-environment-from-scratch-including-elasticsearch-fluentd-and-kibana-efk-within-minikube/]

  • daemonset-filebeat.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat-daemonset
  namespace: nl-amis-logging
  labels:
    app: filebeat
    version: "1.0"
    environment: logging
spec:
  template:
    metadata:
      labels:
        app: filebeat
        version: "1.0"
        environment: logging
    spec:
      serviceAccountName: filebeat-serviceaccount
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.3.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-configmap
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
  • clusterrolebinding-filebeat.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat-clusterrolebinding
  namespace: nl-amis-logging
subjects:
- kind: ServiceAccount
  name: filebeat-serviceaccount
  namespace: nl-amis-logging
roleRef:
  kind: ClusterRole
  name: filebeat-clusterrole
  apiGroup: rbac.authorization.k8s.io
  • clusterrole-filebeat.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat-clusterrole
  namespace: nl-amis-logging
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
  • serviceaccount-filebeat.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat-serviceaccount
  namespace: nl-amis-logging

Vagrantfile

I changed the content of Vagrantfile to:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"
  
  config.vm.define "ubuntu_minikube_helm_elastic" do |ubuntu_minikube_helm_elastic|
  
    config.vm.network "forwarded_port",
      guest: 8001,
      host:  8001,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5601,
      host:  5601,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9200,
      host:  9200,
      auto_correct: true  
      
    config.vm.network "forwarded_port",
      guest: 9010,
      host:  9010,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9020,
      host:  9020,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9110,
      host:  9110,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Minikube Helm Elastic Stack"
        vb.memory = "8192"
        vb.cpus = "1"
        
    args = []
    config.vm.provision "shell",
        path: "scripts/docker.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/minikube.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/kubectl.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/helm.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/namespaces.sh",
        args: args
        
        args = []
    config.vm.provision "shell",
        path: "scripts/elasticsearch.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/kibana.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/filebeat.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/mysql.sh",
        args: args
        
    args = []
    config.vm.provision "shell",
        path: "scripts/booksservices.sh",
        args: args
    end
    
  end

end

In the scripts directory I created a file filebeat.sh with the following content:

#!/bin/bash
echo "**** Begin installing Filebeat"

#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/filebeat-chart/*
helm create filebeat-chart

rm -rf /vagrant/helmcharts/filebeat-chart/templates/*
cp /vagrant/yaml/*filebeat.yaml /vagrant/helmcharts/filebeat-chart/templates

#Exiting: error loading config file: config file ("/etc/filebeat.yaml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /etc/filebeat.yaml')

# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart filebeat-chart"
helm install ./filebeat-chart --name filebeat-release

# Wait 1 minute
echo "**** Waiting 1 minute ..."
sleep 60

echo "**** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:filebeat-serviceaccount) ****"
kubectl auth can-i list pods --as="system:serviceaccount:nl-amis-logging:filebeat-serviceaccount" --namespace nl-amis-logging

#List helm releases
echo "**** List helm releases"
helm list -d

#List pods
echo "**** List pods with namespace nl-amis-logging"
kubectl get pods --namespace nl-amis-logging

#echo "**** Determine the pod name of the filebeat-* pod in namespace nl-amis-logging"
#podName=$(kubectl get pods --namespace nl-amis-logging | grep filebeat- | grep -E -o "^\S*")
#echo "---$podName---"
#echo "**** Check the log file of the $podName pod in namespace nl-amis-logging"
#log=$(kubectl logs $podName --namespace nl-amis-logging | grep "Connection opened to Elasticsearch cluster")
#echo "---$log---"

echo "**** End installing Filebeat"

From the subdirectory named env on my Windows laptop, I opened a Windows Command Prompt (cmd) and typed: vagrant up

This command creates and configures guest machines according to your Vagrantfile.
[https://www.vagrantup.com/docs/cli/up.html]

With the following output (only showing the part about Filebeat):

ubuntu_minikube_helm_elastic: **** Begin installing Filebeat
ubuntu_minikube_helm_elastic: **** Create Helm chart
ubuntu_minikube_helm_elastic: Creating filebeat-chart
ubuntu_minikube_helm_elastic: **** Install Helm chart filebeat-chart
ubuntu_minikube_helm_elastic: NAME: filebeat-release
ubuntu_minikube_helm_elastic: LAST DEPLOYED: Tue Sep 3 17:49:10 2019
ubuntu_minikube_helm_elastic: NAMESPACE: default
ubuntu_minikube_helm_elastic: STATUS: DEPLOYED
ubuntu_minikube_helm_elastic: RESOURCES:
ubuntu_minikube_helm_elastic: ==> v1/ConfigMap
ubuntu_minikube_helm_elastic: NAME
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: DATA AGE
ubuntu_minikube_helm_elastic: filebeat-configmap 1 0s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: ==> v1/Pod(related)
ubuntu_minikube_helm_elastic: NAME READY STATUS RESTARTS AGE
ubuntu_minikube_helm_elastic: filebeat-daemonset-9pcvc 0/1 ContainerCreating 0 0s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: ==> v1/ServiceAccount
ubuntu_minikube_helm_elastic: NAME SECRETS AGE
ubuntu_minikube_helm_elastic: filebeat-serviceaccount 1 0s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: ==> v1beta1/ClusterRole
ubuntu_minikube_helm_elastic: NAME AGE
ubuntu_minikube_helm_elastic: filebeat-clusterrole 0s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: ==> v1beta1/ClusterRoleBinding
ubuntu_minikube_helm_elastic: NAME AGE
ubuntu_minikube_helm_elastic: filebeat-clusterrolebinding 0s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: ==> v1beta1/DaemonSet
ubuntu_minikube_helm_elastic: NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: NODE SELECTOR AGE
ubuntu_minikube_helm_elastic: filebeat-daemonset
ubuntu_minikube_helm_elastic: 1 1 0 1 0 0s
ubuntu_minikube_helm_elastic: **** Waiting 1 minute …
ubuntu_minikube_helm_elastic: **** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:filebeat-serviceaccount) ****
ubuntu_minikube_helm_elastic: yes
ubuntu_minikube_helm_elastic: **** List helm releases
ubuntu_minikube_helm_elastic: NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
ubuntu_minikube_helm_elastic: namespace-release 1 Tue Sep 3 17:43:28 2019 DEPLOYED namespace-chart-0.1.0 1.0 default
ubuntu_minikube_helm_elastic: elasticsearch-release 1 Tue Sep 3 17:44:01 2019 DEPLOYED elasticsearch-chart-0.1.0 1.0 default
ubuntu_minikube_helm_elastic: kibana-release 1 Tue Sep 3 17:46:36 2019 DEPLOYED kibana-chart-0.1.0 1.0 default
ubuntu_minikube_helm_elastic: filebeat-release 1 Tue Sep 3 17:49:10 2019 DEPLOYED filebeat-chart-0.1.0 1.0 default
ubuntu_minikube_helm_elastic: **** List pods with namespace nl-amis-logging
ubuntu_minikube_helm_elastic: NAME
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: READY
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: STATUS
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: RESTARTS
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: AGE
ubuntu_minikube_helm_elastic: elasticsearch-6b46c44f7c-8cs6h
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: 1/1
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: Running
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: 0 6m12s
ubuntu_minikube_helm_elastic: filebeat-daemonset-9pcvc 0/1 ContainerCreating 0 62s
ubuntu_minikube_helm_elastic:
ubuntu_minikube_helm_elastic: kibana-6f96d679c4-qfps4 1/1 Running 0 3m37s
ubuntu_minikube_helm_elastic: **** End installing Filebeat

My demo environment now looks like:

Via the Kubernetes Web UI (Dashboard) I checked that the Filebeat components were created (in the nl-amis-logging namespace):

http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/#!/pod?namespace=nl-amis-logging

  • configmap-filebeat.yaml

Navigate to Config and Storage | Config Maps:

As mentioned earlier, the data field (with key-value pairs) contains the configuration data. In our case the ConfigMap holds information in the form of the content of a configuration file (filebeat.yml).

I tried another way to check the content of the configuration file.

I started a shell to the running container:

kubectl exec -it filebeat-daemonset-q5pb2 --namespace nl-amis-logging -- cat /etc/filebeat.yml

With the following output:

filebeat.inputs:
– type: container
  paths:
    – /var/log/containers/*.log
  processors:
    – add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        – logs_path:
            logs_path: “/var/log/containers/”

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    – type: kubernetes
#      host: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          – /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  – add_cloud_metadata:
  – add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}


output.elasticsearch:
  hosts: [‘${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}’]
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

Remark:
The double dash symbol “–” is used to separate the arguments you want to pass to the command from the kubectl arguments.
[https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/]

  • daemonset-filebeat.yaml

Navigate to Workloads | Daemons Sets:

  • clusterrolebinding-filebeat.yaml

Not applicable.

  • clusterrole-filebeat.yaml

Navigate to Cluster | Roles:

  • serviceaccount-filebeat.yaml

Navigate to Config and Storage | Secrets:

Docker container log files

Every Docker container has a folder “/var/lib/docker/containers/<containerID>” on the host machine, which contains the log file “<containerID>-json.log”.

So, I needed to know the containerID for the Pods, related to the booksservice, in my demo environment.

I got the information using several kubectl commands, leading to the following result:

PodNamespaceLabel keyContainer
appversionenvironmentcontainerIDname
booksservice-v1.0-68785bc6ff-cq769nl-amis-developmentbooksservice1.0developmentdocker://b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9booksservice-v1-0-container
booksservice-v1.0-68785bc6ff-x7mf8nl-amis-developmentbooksservice1.0developmentdocker://bff91eb16a15d0a2058919d7ce7b5077ea9d3f0542c7930f48b29ca1099a54aebooksservice-v1-0-container
booksservice-v2.0-869c5bb47d-bwdc5nl-amis-developmentbooksservice2.0developmentdocker://4344b9a63ac54218dac88148203b2394ac973fe5d1f201a1a870f213e417122cbooksservice-v2-0-container
booksservice-v2.0-869c5bb47d-nwfgfnl-amis-developmentbooksservice2.0developmentdocker://6b0e9a44932986cc6ae2353f28d4c9aff32e249abd0ac38ee22a27614aecb30fbooksservice-v2-0-container
booksservice-v1.0-5bcd5fddbd-cklswnl-amis-testingbooksservice1.0testingdocker://171b2526ae9d147dab7fb2180764d55c03c7ad706eca605ad5c849aafef5d38dbooksservice-v1-0-container
booksservice-v1.0-5bcd5fddbd-n9qkxnl-amis-testingbooksservice1.0testingdocker://f7a9b8a4021073f8c7daba0000482f9f9356495beec7c8d49b9b45f0055f9c20booksservice-v1-0-container
  • Command to list all Pods
kubectl get pods --all-namespaces

With the following output:

NAMESPACE             NAME                                    READY   STATUS    RESTARTS   AGE
kube-system           coredns-576cbf47c7-jvbzp                1/1     Running   0          23h
kube-system           coredns-576cbf47c7-mpl8t                1/1     Running   0          23h
kube-system           etcd-minikube                           1/1     Running   0          23h
kube-system           kube-addon-manager-minikube             1/1     Running   0          23h
kube-system           kube-apiserver-minikube                 1/1     Running   0          23h
kube-system           kube-controller-manager-minikube        1/1     Running   0          23h
kube-system           kube-proxy-6nhhs                        1/1     Running   0          23h
kube-system           kube-scheduler-minikube                 1/1     Running   0          23h
kube-system           kubernetes-dashboard-5bff5f8fb8-p2m57   1/1     Running   0          23h
kube-system           storage-provisioner                     1/1     Running   0          23h
kube-system           tiller-deploy-79c4c54bc4-vprfl          1/1     Running   0          23h
nl-amis-development   
booksservice-v1.0-68785bc6ff-cq769      1/1     Running   0          23h
nl-amis-development   booksservice-v1.0-68785bc6ff-x7mf8      1/1     Running   0          23h
nl-amis-development   booksservice-v2.0-869c5bb47d-bwdc5      1/1     Running   0          23h
nl-amis-development   booksservice-v2.0-869c5bb47d-nwfgf      1/1     Running   0          23h
nl-amis-logging       elasticsearch-6b46c44f7c-v569h          1/1     Running   0          23h
nl-amis-logging       filebeat-daemonset-q5pb2                1/1     Running   0          23h
nl-amis-logging       kibana-6f96d679c4-kb5qw                 1/1     Running   0          23h
nl-amis-testing       booksservice-v1.0-5bcd5fddbd-cklsw      1/1     Running   0          23h
nl-amis-testing       booksservice-v1.0-5bcd5fddbd-n9qkx      1/1     Running   0          23h
nl-amis-testing       mysql-64846c7974-w7mz9                  1/1     Running   0          23h

  • Command to get the YAML (with object information) for a particular Pod:
kubectl get pod -n nl-amis-development booksservice-v1.0-68785bc6ff-cq769 -o yaml

With the following output:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: “2019-09-04T19:49:37Z”
  generateName: booksservice-v1.0-68785bc6ff-
  labels:
    app: booksservice
    environment: development
    pod-template-hash: 68785bc6ff
    version: “1.0”
  name
booksservice-v1.0-68785bc6ff-cq769
  namespace: nl-amis-development
  ownerReferences:
  – apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: booksservice-v1.0-68785bc6ff
    uid: 207a1cd7-cf4d-11e9-95ef-023e591c269a
  resourceVersion: “1711”
  selfLink: /api/v1/namespaces/nl-amis-development/pods/booksservice-v1.0-68785bc6ff-cq769
  uid: 208017d3-cf4d-11e9-95ef-023e591c269a
spec:
  containers:
  – env:
    – name: spring.profiles.active
      value: development
    image: booksservice:v1.0
    imagePullPolicy: IfNotPresent
    name: booksservice-v1-0-container
    ports:
    – containerPort: 9090
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    – mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-4kz7x
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: minikube
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  – effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  – effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  – name: default-token-4kz7x
    secret:
      defaultMode: 420
      secretName: default-token-4kz7x
status:
  conditions:
  – lastProbeTime: null
    lastTransitionTime: “2019-09-04T19:49:37Z”
    status: “True”
    type: Initialized
  – lastProbeTime: null
    lastTransitionTime: “2019-09-04T19:49:42Z”
    status: “True”
    type: Ready
  – lastProbeTime: null
    lastTransitionTime: “2019-09-04T19:49:42Z”
    status: “True”
    type: ContainersReady
  – lastProbeTime: null
    lastTransitionTime: “2019-09-04T19:49:37Z”
    status: “True”
    type: PodScheduled
  containerStatuses:
  – containerID: docker://
b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9
    image: booksservice:v1.0
    imageID: docker://sha256:296bfc231d3bbbe6954ad9a18c3fdfe2d6dcb81a84ee450d433449a63dda4928
    lastState: {}
    name: booksservice-v1-0-container
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: “2019-09-04T19:49:41Z”
  hostIP: 10.0.2.15
  phase: Running
  podIP: 172.17.0.12
  qosClass: BestEffort
  startTime: “2019-09-04T19:49:37Z”

  • Command which uses JSONPath expressions to filter on specific fields in the JSON object and format the output:

[https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/#list-all-containers-in-all-namespaces]

kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}[{.metadata.name}{"\t"},{.metadata.namespace}{"\t"},{.metadata.labels.app}{"\t"},{.metadata.labels.environment}{"\t"},{.metadata.labels.version}{"\t"},{range .status.containerStatuses[*]}{.containerID}{"\t"},{.name}]{"\n"}{end}{end}' |\sort

[https://kubernetes.io/docs/reference/kubectl/jsonpath/]

With the following output:

[booksservice-v1.0-5bcd5fddbd-cklsw     ,nl-amis-testing        ,booksservice   ,testing        ,1.0    ,docker://171b2526ae9d147dab7fb2180764d55c03c7ad706eca605ad5c849aafef5d38d ,booksservice-v1-0-container]
[booksservice-v1.0-5bcd5fddbd-n9qkx     ,nl-amis-testing        ,booksservice   ,testing        ,1.0    ,docker://f7a9b8a4021073f8c7daba0000482f9f9356495beec7c8d49b9b45f0055f9c20 ,booksservice-v1-0-container]
[
booksservice-v1.0-68785bc6ff-cq769     ,nl-amis-development    ,booksservice   ,development    ,1.0    ,docker://b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9 ,booksservice-v1-0-container]
[booksservice-v1.0-68785bc6ff-x7mf8     ,nl-amis-development    ,booksservice   ,development    ,1.0    ,docker://bff91eb16a15d0a2058919d7ce7b5077ea9d3f0542c7930f48b29ca1099a54ae ,booksservice-v1-0-container]
[booksservice-v2.0-869c5bb47d-bwdc5     ,nl-amis-development    ,booksservice   ,development    ,2.0    ,docker://4344b9a63ac54218dac88148203b2394ac973fe5d1f201a1a870f213e417122c ,booksservice-v2-0-container]
[booksservice-v2.0-869c5bb47d-nwfgf     ,nl-amis-development    ,booksservice   ,development    ,2.0    ,docker://6b0e9a44932986cc6ae2353f28d4c9aff32e249abd0ac38ee22a27614aecb30f ,booksservice-v2-0-container]
[coredns-576cbf47c7-jvbzp       ,kube-system    ,       ,       ,       ,docker://8214e5aafc1d034d03abadaf606f9b1a1303757ea4ba5d4fdb689456711e5fad      ,coredns]
[coredns-576cbf47c7-mpl8t       ,kube-system    ,       ,       ,       ,docker://c2e03eb2b0a063120980b99e9b82067d2c72c82c2d86ad7092cd1cc6edbb54a7      ,coredns]
[elasticsearch-6b46c44f7c-v569h ,nl-amis-logging        ,elasticsearch  ,logging        ,7.0.0  ,docker://b8bdbfe91a6aff0f11ef6d62eb20fe186a7fc8eec501b81d31ad327f45009e20
,elasticsearch-container]
[etcd-minikube  ,kube-system    ,       ,       ,       ,docker://74c09a3c714c19fd6cebf1990836cc8e8427e89cea079f612679999874faaa60      ,etcd]
[filebeat-daemonset-q5pb2       ,nl-amis-logging        ,filebeat       ,logging        ,1.0    ,docker://2c65d0c5b03a113fb1d1c38b25feb9379ee25840dbc7210f572c52e8b6df610c
,filebeat]
[kibana-6f96d679c4-kb5qw        ,nl-amis-logging        ,kibana ,logging        ,7.0.0  ,docker://7aa9e0be446db37ad5d0ff76d5c7a8733559f3ce5b9766871c75a56719119c68      ,kibana-container]
[kube-addon-manager-minikube    ,kube-system    ,       ,       ,v8.6   ,docker://d7918eb9fdaa79b9a202f0526caaaa36e8fa531a7fc0a83a1ee513343f6743ea      ,kube-addon-manager]
[kube-apiserver-minikube        ,kube-system    ,       ,       ,       ,docker://d57c1cea8886da34b98df15806675d08196056460c8887655a270ca98261a543      ,kube-apiserver]
[kube-controller-manager-minikube       ,kube-system    ,       ,       ,       ,docker://3312a17ec93a141143a38779e156b96b99d1b1e9f52c27011a9cff5e510410c4      ,kube-controller-manager]
[kube-proxy-6nhhs       ,kube-system    ,       ,       ,       ,docker://36d6f50d14a2854208a849da8cff1ea2fe9217b0b19d46f0aead7aff012e24a1      ,kube-proxy]
[kubernetes-dashboard-5bff5f8fb8-p2m57  ,kube-system    ,kubernetes-dashboard   ,       ,v1.10.1        ,docker://09dbb37fd83b5ed1a1e23c43403fac8d258815c67a6b791e5c5714ba69ca3b02 ,kubernetes-dashboard]
[kube-scheduler-minikube        ,kube-system    ,       ,       ,       ,docker://956c43ecd2bb3154720055d7f1abf53b2d9287a265f5236179bb232bb1a0f55d      ,kube-scheduler]
[mysql-64846c7974-w7mz9 ,nl-amis-testing        ,mysql  ,testing        ,1.0    ,docker://3d6ce5328e7ebcbdf012791fc87d45374afe137439ff13c28fa75ff5fc408f1d      ,mysql]
[storage-provisioner    ,kube-system    ,       ,       ,       ,docker://9d01d6f19dc28d7104282ccec672b211842cff9f813ef586790d8882e0ed20c4      ,storage-provisioner]
[tiller-deploy-79c4c54bc4-vprfl ,kube-system    ,helm   ,       ,       ,docker://5c36d2f20b5be0ddffae6844837b3eaf7b1ad5356a5203ea262c547c2cf80394      ,tiller]

Investigating the content of the log file for a particular Pod

There are several ways in which you can have a look at the content of the log file for a particular Pod (let’s say for the first Pod in the table shown earlier). For example:

  • Via kubectl
kubectl --namespace=nl-amis-development logs booksservice-v1.0-68785bc6ff-cq769 --tail=20

With the following output:

2019-09-04 19:51:16.268  INFO 1 — [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’
2019-09-04 19:51:26.857  INFO 1 — [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService ‘applicationTaskExecutor’
2019-09-04 19:51:27.817  WARN 1 — [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2019-09-04 19:51:32.694  INFO 1 — [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9090 (http) with context path ”
2019-09-04 19:51:32.717  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Started BooksServiceApplication in 102.052 seconds (JVM running for 111.856)
2019-09-04 19:51:32.740  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        :
—-Begin logging BooksServiceApplication—-
2019-09-04 19:51:32.742  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-System Properties from VM Arguments—-
2019-09-04 19:51:32.743  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: null
2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Program Arguments—-
2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Currently active profile – development
2019-09-04 19:51:32.749  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Environment Properties—-
2019-09-04 19:51:32.787  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: 9090
2019-09-04 19:51:32.790  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : nl.amis.environment: development
2019-09-04 19:51:32.791  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.url: null
2019-09-04 19:51:32.819  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.username: null
2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.password: null
2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.database-platform: null
2019-09-04 19:51:32.835  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.hibernate.ddl-auto: null
2019-09-04 19:51:32.856  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-End logging BooksServiceApplication—-

  • Via the Kubernetes Web UI (Dashboard)

Navigate to Workloads | Pods:

Select Pod booksservice-v1.0-68785bc6ff-cq769 and click on Logs (hamburger menu)

Every Docker container has a folder “/var/lib/docker/containers/<containerID>” on the host machine, which contains the log file “<containerID>-json.log”.

So, in my case, the log file can be found in the directory:

/var/lib/docker/containers/b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9

I listed the files in the directory:

sudo ls -latr /var/lib/docker/containers/b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9

With the following output:

total 40
drwx——  2 root root  4096 Sep  4 19:49 checkpoints
drwx——  2 root root  4096 Sep  4 19:49 mounts
drwx—— 45 root root  4096 Sep  4 19:49 ..
-rw-r–r–  1 root root  2015 Sep  4 19:49 hostconfig.json
-rw——-  1 root root  6009 Sep  4 19:49 config.v2.json
drwx——  4 root root  4096 Sep  4 19:49 .
-rw-r—–  1 root root 10078 Sep  4 19:51 
b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9-json.log
I showed the last 20 lines of the content of the log file:

sudo tail -n 20 /var/lib/docker/containers/b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9/b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9-json.log

With the following output:

{“log”:”2019-09-04 19:51:16.268  INFO 1 — [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:16.268984841Z”}
{“log”:”2019-09-04 19:51:26.857  INFO 1 — [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService ‘applicationTaskExecutor’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:26.857813374Z”}
{“log”:”2019-09-04 19:51:27.817  WARN 1 — [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:27.826885651Z”}
{“log”:”2019-09-04 19:51:32.694  INFO 1 — [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9090 (http) with context path ”\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.694259145Z”}
{“log”:”2019-09-04 19:51:32.717  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Started BooksServiceApplication in 102.052 seconds (JVM running for 111.856)\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.720106873Z”}
{“log”:”2019-09-04 19:51:32.740  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : \n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745608597Z”}
{“log”:”—-Begin logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745623312Z”}
{“log”:”2019-09-04 19:51:32.742  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-System Properties from VM Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.74562659Z”}
{“log”:”2019-09-04 19:51:32.743  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776310672Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Program Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776326799Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Currently active profile – development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776330584Z”}
{“log”:”2019-09-04 19:51:32.749  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Environment Properties—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776339916Z”}
{“log”:”2019-09-04 19:51:32.787  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: 9090\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.81186176Z”}
{“log”:”2019-09-04 19:51:32.790  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : nl.amis.environment: development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.8118772Z”}
{“log”:”2019-09-04 19:51:32.791  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.url: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.821143308Z”}
{“log”:”2019-09-04 19:51:32.819  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.username: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826531352Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.password: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826542893Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.database-platform: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.834579246Z”}
{“log”:”2019-09-04 19:51:32.835  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.hibernate.ddl-auto: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.856886746Z”}
{“log”:”2019-09-04 19:51:32.856  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-End logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.857056567Z”}

I listed the files in the directory:

sudo ls -latr /var/log/containers

With the following output:

total 88
drwxrwxr-x 9 root syslog 4096 Sep  4 19:34 ..
lrwxrwxrwx 1 root root     57 Sep  4 19:34 etcd-minikube_kube-system_etcd-74c09a3c714c19fd6cebf1990836cc8e8427e89cea079f612679999874faaa60.log -> /var/log/pods/400930335566057521570dcbaf3dbb0b/etcd/0.log
lrwxrwxrwx 1 root root     67 Sep  4 19:34 kube-scheduler-minikube_kube-system_kube-scheduler-956c43ecd2bb3154720055d7f1abf53b2d9287a265f5236179bb232bb1a0f55d.log -> /var/log/pods/e1b3e16379a55d4c355fa42bc75eb023/kube-scheduler/0.log
lrwxrwxrwx 1 root root     67 Sep  4 19:34 kube-apiserver-minikube_kube-system_kube-apiserver-d57c1cea8886da34b98df15806675d08196056460c8887655a270ca98261a543.log -> /var/log/pods/56a6d3191989178a49a5c197f01f4179/kube-apiserver/0.log
lrwxrwxrwx 1 root root     76 Sep  4 19:34 kube-controller-manager-minikube_kube-system_kube-controller-manager-3312a17ec93a141143a38779e156b96b99d1b1e9f52c27011a9cff5e510410c4.log -> /var/log/pods/8ed2b4b6a766901d30591a648262d9f9/kube-controller-manager/0.log
lrwxrwxrwx 1 root root     71 Sep  4 19:34 kube-addon-manager-minikube_kube-system_kube-addon-manager-d7918eb9fdaa79b9a202f0526caaaa36e8fa531a7fc0a83a1ee513343f6743ea.log -> /var/log/pods/d682efea6fd7d1c11b13f78e8c81af08/kube-addon-manager/0.log
lrwxrwxrwx 1 root root     64 Sep  4 19:35 coredns-576cbf47c7-jvbzp_kube-system_coredns-8214e5aafc1d034d03abadaf606f9b1a1303757ea4ba5d4fdb689456711e5fad.log -> /var/log/pods/1c73e88d-cf4b-11e9-95ef-023e591c269a/coredns/0.log
lrwxrwxrwx 1 root root     64 Sep  4 19:35 coredns-576cbf47c7-mpl8t_kube-system_coredns-c2e03eb2b0a063120980b99e9b82067d2c72c82c2d86ad7092cd1cc6edbb54a7.log -> /var/log/pods/1cbca0a0-cf4b-11e9-95ef-023e591c269a/coredns/0.log
lrwxrwxrwx 1 root root     67 Sep  4 19:35 kube-proxy-6nhhs_kube-system_kube-proxy-36d6f50d14a2854208a849da8cff1ea2fe9217b0b19d46f0aead7aff012e24a1.log -> /var/log/pods/1c68b964-cf4b-11e9-95ef-023e591c269a/kube-proxy/0.log
lrwxrwxrwx 1 root root     77 Sep  4 19:36 kubernetes-dashboard-5bff5f8fb8-p2m57_kube-system_kubernetes-dashboard-09dbb37fd83b5ed1a1e23c43403fac8d258815c67a6b791e5c5714ba69ca3b02.log -> /var/log/pods/1eb91a11-cf4b-11e9-95ef-023e591c269a/kubernetes-dashboard/0.log
lrwxrwxrwx 1 root root     76 Sep  4 19:36 storage-provisioner_kube-system_storage-provisioner-9d01d6f19dc28d7104282ccec672b211842cff9f813ef586790d8882e0ed20c4.log -> /var/log/pods/1f7a6b51-cf4b-11e9-95ef-023e591c269a/storage-provisioner/0.log
lrwxrwxrwx 1 root root     63 Sep  4 19:36 tiller-deploy-79c4c54bc4-vprfl_kube-system_tiller-5c36d2f20b5be0ddffae6844837b3eaf7b1ad5356a5203ea262c547c2cf80394.log -> /var/log/pods/3a175e09-cf4b-11e9-95ef-023e591c269a/tiller/0.log
lrwxrwxrwx 1 root root     80 Sep  4 19:40 elasticsearch-6b46c44f7c-v569h_nl-amis-logging_elasticsearch-container-b8bdbfe91a6aff0f11ef6d62eb20fe186a7fc8eec501b81d31ad327f45009e20.log -> /var/log/pods/a6d793d3-cf4b-11e9-95ef-023e591c269a/elasticsearch-container/0.log
lrwxrwxrwx 1 root root     73 Sep  4 19:43 kibana-6f96d679c4-kb5qw_nl-amis-logging_kibana-container-7aa9e0be446db37ad5d0ff76d5c7a8733559f3ce5b9766871c75a56719119c68.log -> /var/log/pods/0148ec4e-cf4c-11e9-95ef-023e591c269a/kibana-container/0.log
lrwxrwxrwx 1 root root     65 Sep  4 19:44 filebeat-daemonset-q5pb2_nl-amis-logging_filebeat-2c65d0c5b03a113fb1d1c38b25feb9379ee25840dbc7210f572c52e8b6df610c.log -> /var/log/pods/5baef596-cf4c-11e9-95ef-023e591c269a/filebeat/0.log
lrwxrwxrwx 1 root root     62 Sep  4 19:45 mysql-64846c7974-w7mz9_nl-amis-testing_mysql-3d6ce5328e7ebcbdf012791fc87d45374afe137439ff13c28fa75ff5fc408f1d.log -> /var/log/pods/85319a39-cf4c-11e9-95ef-023e591c269a/mysql/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v1.0-5bcd5fddbd-cklsw_nl-amis-testing_booksservice-v1-0-container-171b2526ae9d147dab7fb2180764d55c03c7ad706eca605ad5c849aafef5d38d.log -> /var/log/pods/20936e0c-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v1.0-5bcd5fddbd-n9qkx_nl-amis-testing_booksservice-v1-0-container-f7a9b8a4021073f8c7daba0000482f9f9356495beec7c8d49b9b45f0055f9c20.log -> /var/log/pods/209cc8b9-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v1.0-68785bc6ff-x7mf8_nl-amis-development_booksservice-v1-0-container-bff91eb16a15d0a2058919d7ce7b5077ea9d3f0542c7930f48b29ca1099a54ae.log -> /var/log/pods/20897a07-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v2.0-869c5bb47d-bwdc5_nl-amis-development_booksservice-v2-0-container-4344b9a63ac54218dac88148203b2394ac973fe5d1f201a1a870f213e417122c.log -> /var/log/pods/20800711-cf4d-11e9-95ef-023e591c269a/booksservice-v2-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v2.0-869c5bb47d-nwfgf_nl-amis-development_booksservice-v2-0-container-6b0e9a44932986cc6ae2353f28d4c9aff32e249abd0ac38ee22a27614aecb30f.log -> /var/log/pods/2088e3d9-cf4d-11e9-95ef-023e591c269a/booksservice-v2-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 booksservice-v1.0-68785bc6ff-cq769_nl-amis-development_booksservice-v1-0-container-
b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9.log -> /var/log/pods/208017d3-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
drwxr-xr-x 2 root root   4096 Sep  4 19:49 .

I showed the last 20 lines of the content of the log file:

sudo tail -n 20 /var/log/containers/booksservice-v1.0-68785bc6ff-cq769_nl-amis-development_booksservice-v1-0-container-b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9.log

With the following output:

{“log”:”2019-09-04 19:51:16.268  INFO 1 — [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:16.268984841Z”}
{“log”:”2019-09-04 19:51:26.857  INFO 1 — [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService ‘applicationTaskExecutor’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:26.857813374Z”}
{“log”:”2019-09-04 19:51:27.817  WARN 1 — [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:27.826885651Z”}
{“log”:”2019-09-04 19:51:32.694  INFO 1 — [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9090 (http) with context path ”\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.694259145Z”}
{“log”:”2019-09-04 19:51:32.717  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Started BooksServiceApplication in 102.052 seconds (JVM running for 111.856)\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.720106873Z”}
{“log”:”2019-09-04 19:51:32.740  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : \n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745608597Z”}
{“log”:”—-Begin logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745623312Z”}
{“log”:”2019-09-04 19:51:32.742  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-System Properties from VM Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.74562659Z”}
{“log”:”2019-09-04 19:51:32.743  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776310672Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Program Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776326799Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Currently active profile – development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776330584Z”}
{“log”:”2019-09-04 19:51:32.749  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Environment Properties—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776339916Z”}
{“log”:”2019-09-04 19:51:32.787  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: 9090\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.81186176Z”}
{“log”:”2019-09-04 19:51:32.790  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : nl.amis.environment: development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.8118772Z”}
{“log”:”2019-09-04 19:51:32.791  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.url: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.821143308Z”}
{“log”:”2019-09-04 19:51:32.819  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.username: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826531352Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.password: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826542893Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.database-platform: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.834579246Z”}
{“log”:”2019-09-04 19:51:32.835  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.hibernate.ddl-auto: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.856886746Z”}
{“log”:”2019-09-04 19:51:32.856  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-End logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.857056567Z”}

I listed the files in the directory:

sudo ls -latr /var/log/pods

With the following output:

total 92
drwxrwxr-x  9 root syslog 4096 Sep  4 19:34 ..
drwxr-xr-x  3 root root   4096 Sep  4 19:34 400930335566057521570dcbaf3dbb0b
drwxr-xr-x  3 root root   4096 Sep  4 19:34 56a6d3191989178a49a5c197f01f4179
drwxr-xr-x  3 root root   4096 Sep  4 19:34 e1b3e16379a55d4c355fa42bc75eb023
drwxr-xr-x  3 root root   4096 Sep  4 19:34 8ed2b4b6a766901d30591a648262d9f9
drwxr-xr-x  3 root root   4096 Sep  4 19:34 d682efea6fd7d1c11b13f78e8c81af08
drwxr-xr-x  3 root root   4096 Sep  4 19:35 1c68b964-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:35 1c73e88d-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:35 1cbca0a0-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:36 1eb91a11-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:36 1f7a6b51-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:36 3a175e09-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:40 a6d793d3-cf4b-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:43 0148ec4e-cf4c-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:44 5baef596-cf4c-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:45 85319a39-cf4c-11e9-95ef-023e591c269a
drwxr-xr-x 23 root root   4096 Sep  4 19:49 .
drwxr-xr-x  3 root root   4096 Sep  4 19:49 20936e0c-cf4d-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:49 
208017d3-cf4d-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:49 209cc8b9-cf4d-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:49 20897a07-cf4d-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:49 2088e3d9-cf4d-11e9-95ef-023e591c269a
drwxr-xr-x  3 root root   4096 Sep  4 19:49 20800711-cf4d-11e9-95ef-023e591c269a

I showed the last 20 lines of the content of the log file:

sudo tail -n 20 /var/log/pods/208017d3-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log

With the following output:

{“log”:”2019-09-04 19:51:16.268  INFO 1 — [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:16.268984841Z”}
{“log”:”2019-09-04 19:51:26.857  INFO 1 — [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService ‘applicationTaskExecutor’\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:26.857813374Z”}
{“log”:”2019-09-04 19:51:27.817  WARN 1 — [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:27.826885651Z”}
{“log”:”2019-09-04 19:51:32.694  INFO 1 — [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9090 (http) with context path ”\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.694259145Z”}
{“log”:”2019-09-04 19:51:32.717  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Started BooksServiceApplication in 102.052 seconds (JVM running for 111.856)\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.720106873Z”}
{“log”:”2019-09-04 19:51:32.740  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : \n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745608597Z”}
{“log”:”—-Begin logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.745623312Z”}
{“log”:”2019-09-04 19:51:32.742  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-System Properties from VM Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.74562659Z”}
{“log”:”2019-09-04 19:51:32.743  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776310672Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Program Arguments—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776326799Z”}
{“log”:”2019-09-04 19:51:32.748  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : Currently active profile – development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776330584Z”}
{“log”:”2019-09-04 19:51:32.749  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-Environment Properties—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.776339916Z”}
{“log”:”2019-09-04 19:51:32.787  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : server.port: 9090\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.81186176Z”}
{“log”:”2019-09-04 19:51:32.790  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : nl.amis.environment: development\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.8118772Z”}
{“log”:”2019-09-04 19:51:32.791  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.url: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.821143308Z”}
{“log”:”2019-09-04 19:51:32.819  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.username: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826531352Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.datasource.password: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.826542893Z”}
{“log”:”2019-09-04 19:51:32.824  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.database-platform: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.834579246Z”}
{“log”:”2019-09-04 19:51:32.835  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : spring.jpa.hibernate.ddl-auto: null\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.856886746Z”}
{“log”:”2019-09-04 19:51:32.856  INFO 1 — [           main] n.a.d.s.b.BooksServiceApplication        : —-End logging BooksServiceApplication—-\n”,”stream”:”stdout”,”time”:”2019-09-04T19:51:32.857056567Z”}

Creating the ConfigMap from a file

In line with what I did when I used Fluentd (see previous article), I wanted to create the ConfigMap from a file. So, I changed the default configuration file location ( /etc/filebeat.yml) to: /etc/custom-config/filebeat.yml.
[https://technology.amis.nl/2019/04/23/using-vagrant-and-shell-scripts-to-further-automate-setting-up-my-demo-environment-from-scratch-including-elasticsearch-fluentd-and-kibana-efk-within-minikube/]

There for, in the vagrant directory I created a subdirectory structure configmaps/configmap-filebeat with a file filebeat.yml with the following content:

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      host: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

Remark:
For more information about the content of this config file, I kindly refer you to the FileBeat documentation, for example:

https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html

I created a ConfigMap that holds the filebeat config file using:

kubectl create configmap filebeat-configmap --from-file=/vagrant/configmaps/configmap-filebeat --namespace nl-amis-logging

Next, I added labels to the ConfigMap using:

kubectl label configmap filebeat-configmap --namespace nl-amis-logging app=filebeat
kubectl label configmap filebeat-configmap --namespace nl-amis-logging version="1.0"
kubectl label configmap filebeat-configmap --namespace nl-amis-logging environment=logging

A ConfigMap can be created via a yaml file, but not if you want to use the from-file option, because kubernetes isn’t aware of the local file’s path.
[https://stackoverflow.com/questions/51268488/kubernetes-configmap-set-from-file-in-yaml-configuration]

You must create a ConfigMap before referencing it in a Pod specification (unless you mark the ConfigMap as “optional”). If you reference a ConfigMap that doesn’t exist, the Pod won’t start.
ConfigMaps reside in a specific namespace. A ConfigMap can only be referenced by pods residing in the same namespace.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/]

When you create a ConfigMap using –from-file, the filename becomes a key stored in the data section of the ConfigMap. The file contents become the key’s value.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-volume]

On my Windows laptop, in the yaml directory, I deleted the file configmap-filebeat.yaml.

In the scripts directory I changed the file filebeat.sh to have the following content:

#!/bin/bash
echo "**** Begin installing Filebeat"

#Create ConfigMap before creating DaemonSet
kubectl create configmap filebeat-configmap --from-file=/vagrant/configmaps/configmap-filebeat --namespace nl-amis-logging

#Label ConfigMap
kubectl label configmap filebeat-configmap --namespace nl-amis-logging app=filebeat
kubectl label configmap filebeat-configmap --namespace nl-amis-logging version="1.0"
kubectl label configmap filebeat-configmap --namespace nl-amis-logging environment=logging

#List configmaps
echo "**** List configmap filebeat-configmap with namespace nl-amis-logging"
#kubectl describe configmaps filebeat-configmap --namespace nl-amis-logging
kubectl get configmaps filebeat-configmap --namespace nl-amis-logging -o yaml

#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/filebeat-chart/*
helm create filebeat-chart

rm -rf /vagrant/helmcharts/filebeat-chart/templates/*
cp /vagrant/yaml/*filebeat.yaml /vagrant/helmcharts/filebeat-chart/templates
#Exiting: error loading config file: config file ("/etc/filebeat.yaml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /etc/filebeat.yaml')

# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart filebeat-chart"
helm install ./filebeat-chart --name filebeat-release

# Wait 1 minute
echo "**** Waiting 1 minute ..."
sleep 60

echo "**** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:filebeat-serviceaccount) ****"
kubectl auth can-i list pods --as="system:serviceaccount:nl-amis-logging:filebeat-serviceaccount" --namespace nl-amis-logging

#List helm releases
echo "**** List helm releases"
helm list -d

#List pods
echo "**** List pods with namespace nl-amis-logging"
kubectl get pods --namespace nl-amis-logging

#echo "**** Determine the pod name of the filebeat-* pod in namespace nl-amis-logging"
#podName=$(kubectl get pods --namespace nl-amis-logging | grep filebeat- | grep -E -o "^\S*")
#echo "---$podName---"
#echo "**** Check the log file of the $podName pod in namespace nl-amis-logging"
#log=$(kubectl logs $podName --namespace nl-amis-logging | grep "Connection opened to Elasticsearch cluster")
#echo "---$log---"

echo "**** End installing Filebeat"

In the yaml directory I changed the file daemonset-filebeat.yaml to have the following content:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat-daemonset
  namespace: nl-amis-logging
  labels:
    app: filebeat
    version: "1.0"
    environment: logging
spec:
  template:
    metadata:
      labels:
        app: filebeat
        version: "1.0"
        environment: logging
    spec:
      serviceAccountName: filebeat-serviceaccount
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.3.1
        args: [
          "-c", " /etc/custom-config/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: "elasticsearch-service.nl-amis-logging"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: filebeat-config-volume
          mountPath: /etc/custom-config
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: filebeat-config-volume
        configMap:
          name: filebeat-configmap
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

So, because I made some changes, I deleted and re-installed the release of Filebeat (via Helm, the package manager for Kubernetes).

helm del --purge filebeat-release

release "filebeat-release" deleted

Next, I started the shell script:

cd /vagrant/scripts
./filebeat.sh

With the following output:

**** Begin installing Filebeat
configmap/filebeat-configmap created
configmap/filebeat-configmap labeled
configmap/filebeat-configmap labeled
configmap/filebeat-configmap labeled
**** List configmap filebeat-configmap with namespace nl-amis-logging
apiVersion: v1
data:
  filebeat.yml: “filebeat.inputs:\r\n- type: container\r\n  paths:\r\n    – /var/log/containers/*.log\r\n
    \ processors:\r\n    – add_kubernetes_metadata:\r\n        in_cluster: true\r\n
    \       host: ${NODE_NAME}\r\n        matchers:\r\n        – logs_path:\r\n            logs_path:
    \”/var/log/containers/\”\r\n\r\n# To enable hints based autodiscover, remove `filebeat.inputs`
    configuration and uncomment this:\r\n#filebeat.autodiscover:\r\n#  providers:\r\n#
    \   – type: kubernetes\r\n#      host: ${NODE_NAME}\r\n#      hints.enabled: true\r\n#
    \     hints.default_config:\r\n#        type: container\r\n#        paths:\r\n#
    \         – /var/log/containers/*${data.kubernetes.container.id}.log\r\n\r\nprocessors:\r\n
    \ – add_cloud_metadata:\r\n  – add_host_metadata:\r\n\r\ncloud.id: ${ELASTIC_CLOUD_ID}\r\ncloud.auth:
    ${ELASTIC_CLOUD_AUTH}\r\n\r\noutput.elasticsearch:\r\n  hosts: [‘${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}’]\r\n
    \ username: ${ELASTICSEARCH_USERNAME}\r\n  password: ${ELASTICSEARCH_PASSWORD}”
kind: ConfigMap
metadata:
  creationTimestamp: “2019-09-08T14:34:28Z”
  labels:
    app: filebeat
    environment: logging
    version: “1.0”
  name: filebeat-configmap
  namespace: nl-amis-logging
  resourceVersion: “42432”
  selfLink: /api/v1/namespaces/nl-amis-logging/configmaps/filebeat-configmap
  uid: c381e4bd-d245-11e9-95ef-023e591c269a
**** Create Helm chart
Creating filebeat-chart
**** Install Helm chart filebeat-chart
NAME:   filebeat-release
LAST DEPLOYED: Sun Sep  8 14:34:31 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                      READY  STATUS             RESTARTS  AGE
filebeat-daemonset-bzz86  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME                     SECRETS  AGE
filebeat-serviceaccount  1        0s

==> v1beta1/ClusterRole
NAME                  AGE
filebeat-clusterrole  0s

==> v1beta1/ClusterRoleBinding
NAME                         AGE
filebeat-clusterrolebinding  0s

==> v1beta1/DaemonSet
NAME                DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
filebeat-daemonset  1        1        0      1           0                   0s

**** Waiting 1 minute …
**** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:filebeat-serviceaccount) ****
yes
**** List helm releases
NAME                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION     NAMESPACE
namespace-release       1               Wed Sep  4 19:38:32 2019        DEPLOYED        namespace-chart-0.1.0           1.0             default
elasticsearch-release   1               Wed Sep  4 19:39:03 2019        DEPLOYED        elasticsearch-chart-0.1.0       1.0             default
kibana-release          1               Wed Sep  4 19:41:35 2019        DEPLOYED        kibana-chart-0.1.0              1.0             default
mysql-release           1               Wed Sep  4 19:45:16 2019        DEPLOYED        mysql-chart-0.1.0               1.0             default
booksservice-release    1               Wed Sep  4 19:49:37 2019        DEPLOYED        booksservice-chart-0.1.0        1.0             default
filebeat-release        1               Sun Sep  8 14:34:31 2019        DEPLOYED        filebeat-chart-0.1.0            1.0             default
**** List pods with namespace nl-amis-logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-6b46c44f7c-v569h   1/1     Running   0          3d18h
filebeat-daemonset-bzz86         1/1     Running   0          62s
kibana-6f96d679c4-kb5qw          1/1     Running   0          3d18h
**** End installing Filebeat

As can be seen in daemonset-filebeat.yaml, Filebeat uses the hostPath /var/lib/filebeat-data to persist internal data.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat-daemonset
  namespace: nl-amis-logging
  labels:
    app: filebeat
    version: "1.0"
    environment: logging
spec:
…
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

On the host (my virtual machine), I listed the files in the directory:

vagrant@ubuntu-xenial:/vagrant/scripts$ cd /var/lib/filebeat-data
vagrant@ubuntu-xenial:/var/lib/filebeat-data$ ls -latr

With the following output:

total 16
drwxr-xr-x 47 root root 4096 Sep  4 19:44 ..
-rw——-  1 root root   48 Sep  4 19:44 meta.json
drwxr-x—  3 root root 4096 Sep  4 19:44 registry
drwxr-xr-x  3 root root 4096 Sep  4 19:44 .

I started a shell to the running container:

kubectl exec -it filebeat-daemonset-bzz86 --namespace nl-amis-logging -- ls -latr

With the following output:

total 87644
drwxrwx— 1 root filebeat     4096 Aug 19 19:32 modules.d
drwxr-x— 1 root filebeat     4096 Aug 19 19:32 module
drwxr-x— 1 root filebeat     4096 Aug 19 19:32 kibana
-rw-r—– 1 root filebeat      290 Aug 19 19:32 filebeat.yml
-rw-r—– 1 root filebeat    78871 Aug 19 19:32 filebeat.reference.yml
-rwxr-x— 1 root filebeat 89142619 Aug 19 19:32 filebeat
-rw-r—– 1 root filebeat   242580 Aug 19 19:32 fields.yml
-rw-r—– 1 root filebeat      802 Aug 19 19:32 README.md
-rw-r—– 1 root filebeat   216284 Aug 19 19:32 NOTICE.txt
-rw-r—– 1 root filebeat    13675 Aug 19 19:32 LICENSE.txt
-rw-r—– 1 root filebeat       41 Aug 19 19:32 .build_hash.txt
drwxr-xr-x 1 root root         4096 Aug 19 19:32 ..
drwxrwx— 2 root filebeat     4096 Aug 19 19:32 logs
drwxr-x— 1 root filebeat     4096 Aug 19 19:32 .
drwxr-xr-x 3 root root         4096 Sep  4 19:44 data

I changed the directory:

kubectl exec -it filebeat-daemonset-bzz86 --namespace nl-amis-logging -- cd data

I listed the files in the directory:

vagrant@ubuntu-xenial:/var/lib/filebeat-data$ ls -latr

With the following output:

total 16
drwxr-xr-x 47 root root 4096 Sep  4 19:44 ..
-rw——-  1 root root   48 Sep  4 19:44 meta.json
drwxr-x—  3 root root 4096 Sep  4 19:44 registry
drwxr-xr-x  3 root root 4096 Sep  4 19:44 .

Remark:
The double dash symbol “–” is used to separate the arguments you want to pass to the command from the kubectl arguments.
[https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/]

Elasticsearch index

In the Kibana Dashboard via Management | Kibana | Index Patterns you can create an index pattern.
Kibana uses index patterns to retrieve data from Elasticsearch indices for things like visualizations.
[http://localhost:5601/app/kibana#/management/kibana/index_pattern?_g=()]

In the field “Index pattern” I entered filebeat*. The index pattern matched 1 index. Next, I clicked on button “Next step”.

In the field “Time Filter field name” I entered @timestamp.
The Time Filter will use this field to filter your data by time.
You can choose not to have a time field, but you will not be able to narrow down your data by a time range.
[http://localhost:5601/app/kibana#/management/kibana/index_pattern?_g=()]

Next, I clicked on button “Create index pattern”.

The Kibana index pattern filebeat* was created, with 1019 fields.
This page lists every field in the filebeat* index and the field’s associated core type as recorded by Elasticsearch. To change a field type, use the Elasticsearch Mapping API.

Kibana Dashboard, Discover

In the Kibana Dashboard via Discover you can see the log files.

Let’s shortly focus on the first hit.

Via a click on icon “>”, the document is expanded.

Kibana Dashboard, Visualize, creating a visualization

In the Kibana Dashboard via Visualize you can create a visualization.

I clicked on button “Create a visualization” and selected “Pie” as the type for the visualization.

As a source I chose the Index pattern, I created earlier.

After I selected filebeat* the following became visible.

In tab “Data”, in the Split Slices part, in the field “Aggregation” I selected Terms and in “Field” I selected kubernetes.container.name and left the other default settings as they were.
Then I clicked on the icon “Apply changes”, with the following result:

In tab “Options”, I selected the checkbox “Show Labels” and left the other default settings as they were.
Then I clicked on the icon “Apply changes”, with the following result:

So, the container names from the log files from the last 15 minutes are shown.

Postman

Remember that on my Windows laptop, I also wanted to be able to use Postman (for sending requests), via port forwarding this was made possible.
So, I used Postman to add books to and retrieve books from the book catalog. I did this for version 1.0 and 2.0 of the BooksService application.

From Postman I invoked a request named “GetAllBooksRequest” (with method “POST” and URL “http://locahost:9010/books”).
This concerns version 1.0 in the DEV environment.
A response with “Status 200 OK” was shown (with 4 books being retrieved):

From Postman I invoked a request named “GetAllBooksRequest” (with method “POST” and URL http://locahost:9020/books).
This concerns version 2.0 in the DEV environment.
A response with “Status 200 OK” was shown (with 4 books being retrieved):

From Postman I invoked a request named “GetAllBooksRequest” (with method “POST” and URL “http://locahost:9110/books”).
This concerns version 1.0 in the TST environment.
A response with “Status 200 OK” was shown (with 4 books being retrieved):

Remember, each time the getAllBooks method is called, this becomes visible in the container log file.

Kibana Dashboard, Visualize, creating a visualization

In the previously created visualization, I clicked on button “Refresh”. Then the booksservice containers became visible also.

Then I saved this Visualization, via a click on button “Save”.

In the pop-up “Save visualization”, in the field “Title” I entered containers_visualization_1. Next, I clicked on button “Confirm Save”.
In the left top of the screen this title then becomes visible.

Remark:
All the Saved Objects can be seen in the Kibana Dashboard via Management | Kibana | Saved Objects.

Filtering and enhancing the exported data

Filebeat provides a couple of options for filtering and enhancing exported data.
You can configure each input to include or exclude specific lines or files. This allows you to specify different filtering criteria for each input.
[https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html]

Filtering the exported data

I wanted to focus on the log files from the mysql and booksservice containers, so I needed to change the filter.

Remember the output from listing the directory /var/log/containers:

total 88
drwxrwxr-x 9 root syslog 4096 Sep  4 19:34 ..

lrwxrwxrwx 1 root root     65 Sep  4 19:44 filebeat-daemonset-q5pb2_nl-amis-logging_filebeat-2c65d0c5b03a113fb1d1c38b25feb9379ee25840dbc7210f572c52e8b6df610c.log -> /var/log/pods/5baef596-cf4c-11e9-95ef-023e591c269a/filebeat/0.log
lrwxrwxrwx 1 root root     62 Sep  4 19:45 
mysql-64846c7974-w7mz9_nl-amis-testing_mysql-3d6ce5328e7ebcbdf012791fc87d45374afe137439ff13c28fa75ff5fc408f1d.log -> /var/log/pods/85319a39-cf4c-11e9-95ef-023e591c269a/mysql/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v1.0-5bcd5fddbd-cklsw_nl-amis-testing_booksservice-v1-0-container-171b2526ae9d147dab7fb2180764d55c03c7ad706eca605ad5c849aafef5d38d.log -> /var/log/pods/20936e0c-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v1.0-5bcd5fddbd-n9qkx_nl-amis-testing_booksservice-v1-0-container-f7a9b8a4021073f8c7daba0000482f9f9356495beec7c8d49b9b45f0055f9c20.log -> /var/log/pods/209cc8b9-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v1.0-68785bc6ff-x7mf8_nl-amis-development_booksservice-v1-0-container-bff91eb16a15d0a2058919d7ce7b5077ea9d3f0542c7930f48b29ca1099a54ae.log -> /var/log/pods/20897a07-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v2.0-869c5bb47d-bwdc5_nl-amis-development_booksservice-v2-0-container-4344b9a63ac54218dac88148203b2394ac973fe5d1f201a1a870f213e417122c.log -> /var/log/pods/20800711-cf4d-11e9-95ef-023e591c269a/booksservice-v2-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v2.0-869c5bb47d-nwfgf_nl-amis-development_booksservice-v2-0-container-6b0e9a44932986cc6ae2353f28d4c9aff32e249abd0ac38ee22a27614aecb30f.log -> /var/log/pods/2088e3d9-cf4d-11e9-95ef-023e591c269a/booksservice-v2-0-container/0.log
lrwxrwxrwx 1 root root     84 Sep  4 19:49 
booksservice-v1.0-68785bc6ff-cq769_nl-amis-development_booksservice-v1-0-container-b234de2b689187f94792d45ac59fda0f2f6f6969c679d6a6ca5d8323ab8fd1c9.log -> /var/log/pods/208017d3-cf4d-11e9-95ef-023e591c269a/booksservice-v1-0-container/0.log
drwxr-xr-x 2 root root   4096 Sep  4 19:49 .

Based on this output, in the subdirectory configmaps/configmap-filebeat, I changed the file filebeat.yml to the following content:

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/mysql*.log
    - /var/log/containers/booksservice*.log
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      host: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

So, because I made some changes, I deleted and re-installed the release of Filebeat (via Helm, the package manager for Kubernetes) and the ConfigMap.

helm del --purge filebeat-release

release "filebeat-release" deleted
kubectl --namespace=nl-amis-logging delete configmap filebeat-configmap

configmap "filebeat-configmap" deleted

Next, I started the shell script:

cd /vagrant/scripts
./filebeat.sh

With the following output:

**** Begin installing Filebeat
configmap/filebeat-configmap created
configmap/filebeat-configmap labeled
configmap/filebeat-configmap labeled
configmap/filebeat-configmap labeled
**** List configmap filebeat-configmap with namespace nl-amis-logging
apiVersion: v1
data:
  filebeat.yml: “filebeat.inputs:\r\n- type: container\r\n  paths:\r\n    – 
/var/log/containers/mysql*.log\r\n
    \   – 
/var/log/containers/booksservice*.log\r\n  processors:\r\n    – add_kubernetes_metadata:\r\n
    \       in_cluster: true\r\n        host: ${NODE_NAME}\r\n        matchers:\r\n
    \       – logs_path:\r\n            logs_path: \”/var/log/containers/\”\r\n\r\n#
    To enable hints based autodiscover, remove `filebeat.inputs` configuration and
    uncomment this:\r\n#filebeat.autodiscover:\r\n#  providers:\r\n#    – type: kubernetes\r\n#
    \     host: ${NODE_NAME}\r\n#      hints.enabled: true\r\n#      hints.default_config:\r\n#
    \       type: container\r\n#        paths:\r\n#          – /var/log/containers/*${data.kubernetes.container.id}.log\r\n\r\nprocessors:\r\n
    \ – add_cloud_metadata:\r\n  – add_host_metadata:\r\n\r\ncloud.id: ${ELASTIC_CLOUD_ID}\r\ncloud.auth:
    ${ELASTIC_CLOUD_AUTH}\r\n\r\noutput.elasticsearch:\r\n  hosts: [‘${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}’]\r\n
    \ username: ${ELASTICSEARCH_USERNAME}\r\n  password: ${ELASTICSEARCH_PASSWORD}”
kind: ConfigMap
metadata:
  creationTimestamp: “2019-09-12T12:28:26Z”
  labels:
    app: filebeat
    environment: logging
    version: “1.0”
  name: filebeat-configmap
  namespace: nl-amis-logging
  resourceVersion: “53891”
  selfLink: /api/v1/namespaces/nl-amis-logging/configmaps/filebeat-configmap
  uid: d1f26a17-d558-11e9-95ef-023e591c269a
**** Create Helm chart
Creating filebeat-chart
**** Install Helm chart filebeat-chart
NAME:   filebeat-release
LAST DEPLOYED: Thu Sep 12 12:28:27 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                      READY  STATUS             RESTARTS  AGE
filebeat-daemonset-hpxtl  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME                     SECRETS  AGE
filebeat-serviceaccount  1        0s

==> v1beta1/ClusterRole
NAME                  AGE
filebeat-clusterrole  0s

==> v1beta1/ClusterRoleBinding
NAME                         AGE
filebeat-clusterrolebinding  0s

==> v1beta1/DaemonSet
NAME                DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
filebeat-daemonset  1        1        0      1           0                   0s

**** Waiting 1 minute …
**** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:filebeat-serviceaccount) ****
yes
**** List helm releases
NAME                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION     NAMESPACE
namespace-release       1               Wed Sep  4 19:38:32 2019        DEPLOYED        namespace-chart-0.1.0           1.0             default
elasticsearch-release   1               Wed Sep  4 19:39:03 2019        DEPLOYED        elasticsearch-chart-0.1.0       1.0             default
kibana-release          1               Wed Sep  4 19:41:35 2019        DEPLOYED        kibana-chart-0.1.0              1.0             default
mysql-release           1               Wed Sep  4 19:45:16 2019        DEPLOYED        mysql-chart-0.1.0               1.0             default
booksservice-release    1               Wed Sep  4 19:49:37 2019        DEPLOYED        booksservice-chart-0.1.0        1.0             default
filebeat-release        1               Thu Sep 12 12:28:27 2019        DEPLOYED        filebeat-chart-0.1.0            1.0             default
**** List pods with namespace nl-amis-logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-6b46c44f7c-v569h   1/1     Running   0          7d16h
filebeat-daemonset-hpxtl         1/1     Running   0          61s
kibana-6f96d679c4-kb5qw          1/1     Running   0          7d16h
**** End installing Filebeat

Kibana Dashboard, Discover

After a while (longer than 15 minutes), when I looked in the Kibana Dashboard via Discover there were no results found.

So, the filter seemed to work. To check this out, I used Postman to add books and retrieve books from the book catalog. I did this for version 1.0 and 2.0 of the BooksService application.

Again, I looked in the Kibana Dashboard via Discover, clicked on button “Refresh” and then there were results found.

In the previously created visualization, I clicked on button “Refresh”. Then the booksservice containers became visible.

The log files from the other containers are filtered out by Filebeat as I expected. But what happened to the log file from the mysql container?

I checked the content of that log file.

sudo tail -n 20 /var/log/containers/mysql-64846c7974-w7mz9_nl-amis-testing_mysql-3d6ce5328e7ebcbdf012791fc87d45374afe137439ff13c28fa75ff5fc408f1d.log

With the following output:

{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.560587534Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Memory barrier is not used\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.560618161Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Compressed tables use zlib 1.2.11\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.560648314Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Using Linux native AIO\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.56067815Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Using CPU crc32 instructions\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.560819023Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.56157643Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Completed initialization of buffer pool\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.568549963Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Highest supported file format is Barracuda.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.572777146Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: 128 rollback segment(s) are active.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.581715487Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: Waiting for purge to start\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.582108737Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] InnoDB: 5.6.45 started; log sequence number 1625997\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.632386895Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] Server hostname (bind-address): ‘*’; port: 3306\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.632927728Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] IPv6 is available.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.633827748Z”}
{“log”:”2019-09-04 19:45:46 1 [Note]   – ‘::’ resolves to ‘::’;\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.633866785Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] Server socket created on IP: ‘::’.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.63390417Z”}
{“log”:”2019-09-04 19:45:46 1 [Warning] Insecure configuration for –pid-file: Location ‘/var/run/mysqld’ in the path is accessible to all OS users. Consider choosing a different directory.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.634014095Z”}
{“log”:”2019-09-04 19:45:46 1 [Warning] ‘proxies_priv’ entry ‘@ root@mysql-64846c7974-w7mz9’ ignored in –skip-name-resolve mode.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.634569373Z”}
{“log”:”2019-09-04 19:45:46 1 [Note] Event Scheduler: Loaded 0 events\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.638250709Z”}
{“log”:”
2019-09-04 19:45:46 1 [Note] mysqld: ready for connections.\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.638297182Z“}
{“log”:”Version: ‘5.6.45’  socket: ‘/var/run/mysqld/mysqld.sock’  port: 3306  MySQL Community Server (GPL)\n”,”stream”:”stderr”,”time”:”2019-09-04T19:45:46.638302543Z”}

So, the last time something was written to this log file, was a while ago. That’s why it didn’t turn up in the Kibana visualization for the last 15 minutes.

But, let’s check out if we can find the same information in Kibana.

In the visualization, I clicked on button “Show dates” and change the period to Sep 4, 2019 21:30 – 22:00 (taking in account the time zone) and then I clicked on button “Refresh”.

And then the mysql container became visible.

Next, I looked in the Kibana Dashboard via Discover, clicked on button “Refresh” and then there were lots of results found.

I clicked on button “Add filter” and in the field “Field” I entered kubernetes.container.name, as “Operator” I chose is and as “Value” I chose mysql. I then added another filter. In the field “Field” I entered message, as “Operator” I chose is and as “Value” I chose mysqld: ready for connections. .

I selected the first hit.

Remark:
The second hit is not visible in the listing of the content of the mysql log file, shown earlier, because there I only showed the last 20 lines!

Via a click on icon “>”, the document is expanded.

You can also choose to view the expanded document in JSON:

{
  “_index”: “filebeat-7.3.1-2019.09.08-000001”,
  “_type”: “_doc”,
  “_id”: “OOpLEW0B1Kd52Ckb7z6X”,
  “_version”: 1,
  “_score”: null,
  “_source”: {
    “@timestamp”: “2019-09-04T19:45:46.638Z”,
    “agent”: {
      “hostname”: “ubuntu-xenial”,
      “id”: “6fc91322-3f72-494f-9b52-12df04833853”,
      “version”: “7.3.1”,
      “type”: “filebeat”,
      “ephemeral_id”: “e43d286c-fa3d-4adc-9ca3-d11ddb2b052d”
    },
    “ecs”: {
      “version”: “1.0.1”
    },
    “host”: {
      “containerized”: false,
      “name”: “ubuntu-xenial”,
      “hostname”: “ubuntu-xenial”,
      “architecture”: “x86_64”,
      “os”: {
        “codename”: “Core”,
        “platform”: “centos”,
        “version”: “7 (Core)”,
        “family”: “redhat”,
        “name”: “CentOS Linux”,
        “kernel”: “4.4.0-142-generic”
      }
    },
    “stream”: “stderr”,
    “message”: “
2019-09-04 19:45:46 1 [Note] mysqld: ready for connections.“,
    “log”: {
      “offset”: 27622,
      “file”: {
        “path”: “/var/log/containers/mysql-64846c7974-w7mz9_nl-amis-testing_mysql-3d6ce5328e7ebcbdf012791fc87d45374afe137439ff13c28fa75ff5fc408f1d.log”
      }
    },
    “input”: {
      “type”: “container”
    },
    “kubernetes”: {
      “namespace”: “nl-amis-testing”,
      “replicaset”: {
        “name”: “mysql-64846c7974”
      },
      “labels”: {
        “environment”: “testing”,
        “pod-template-hash”: “64846c7974”,
        “version”: “1.0”,
        “app”: “mysql”
      },
      “pod”: {
        “uid”: “85319a39-cf4c-11e9-95ef-023e591c269a”,
        “name”: “mysql-64846c7974-w7mz9”
      },
      “node”: {
        “name”: “minikube”
      },
      “container”: {
        “name”: “mysql”
      }
    }
  },
  “fields”: {
    “@timestamp”: [
      “2019-09-04T19:45:46.638Z”
    ],
    “suricata.eve.timestamp”: [
      “2019-09-04T19:45:46.638Z”
    ]
  },
  “highlight”: {
    “kubernetes.container.name”: [
      “@kibana-highlighted-field@mysql@/kibana-highlighted-field@”
    ],
    “message”: [
      “2019-09-04 19:45:46 1 [Note] @kibana-highlighted-field@mysqld@/kibana-highlighted-field@: @kibana-highlighted-field@ready@/kibana-highlighted-field@ @kibana-highlighted-field@for@/kibana-highlighted-field@ @kibana-highlighted-field@connections@/kibana-highlighted-field@.”
    ]
  },
  “sort”: [
    1567626346638
  ]
}

In this expanded document you can see the following value for message:

2019-09-04 19:45:46 1 [Note] mysqld: ready for connections.

Enhancing the exported data

I wanted to try out some form of enhancing the exported log files. So, I decided to add some extra fields in order to add additional information to the output.

When using the log input to read lines from log files, you can, for example, use the following configuration options, which are supported by all inputs.

  • fields

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-fields]
[https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-container.html#filebeat-input-container-fields]

  • fields_under_root

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#fields-under-root-log]
[https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-container.html#fields-under-root-container]

You can define processors in your configuration to process events before they are sent to the configured output.
[https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html#using-processors]

The add_fields processor adds additional fields to the event. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify will be grouped under the fields sub-dictionary in the event. To group the fields under a different sub-dictionary, use the target setting. To store the fields as top-level fields, set target: ”.

  • target
    (Optional) Sub-dictionary to put all fields into. Defaults to fields.
  • fields
    Fields to be added.

[https://www.elastic.co/guide/en/beats/filebeat/current/add-fields.html]

Based on the information above, in the subdirectory configmaps/configmap-filebeat, I changed the file filebeat.yml to the following content:

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/mysql*.log
    - /var/log/containers/booksservice*.log
  fields:
    my_custom_field1: 'value_of_my_custom_field1'
  fields_under_root: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - add_fields:
        target: my-custom-sub-dictionary1
        fields:
          my_custom_field2: 'value_of_my_custom_field2'
          my_custom_field3: 'value_of_my_custom_field3'
    - add_fields:
        target: my-custom-sub-dictionary2
        fields:
          my_custom_field4: 'value_of_my_custom_field4'
    - add_fields:
        fields:
          my_custom_field5: 'value_of_my_custom_field5'
        
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      host: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

So, because I made some changes, I deleted and re-installed the release of Filebeat (via Helm, the package manager for Kubernetes) and the ConfigMap.

I used Postman to retrieve books from the book catalog. I did this for version 1.0 in the TST enviroment of the BooksService application.

Again, I looked in the Kibana Dashboard via Discover, clicked on button “Refresh” and then there were results found.

I selected the first hit. Via a click on icon “>”, the document is expanded.

On the discover pane, fields with missing index-pattern-fields show hazard icons and display a warning.

To get rid of these, in the Kibana Dashboard via Management | Kibana | Index Patterns, I clicked on the icon “Refresh field list”.

Again, I looked in the Kibana Dashboard via Discover, I selected the first hit and expanded the document.

Then I chose to view the expanded document in JSON:

{
  “_index”: “filebeat-7.3.1-2019.09.08-000001”,
  “_type”: “_doc”,
  “_id”: “eer6JW0B1Kd52CkbpGMs”,
  “_version”: 1,
  “_score”: null,
  “_source”: {
    “@timestamp”: “2019-09-12T14:57:18.195Z”,
    “message”: “2019-09-12 14:57:18.188  INFO 1 — [nio-9091-exec-2] n.a.d.s.b.application.BookService        : “,
    “input”: {
      “type”: “container”
    },
    “my-custom-sub-dictionary2”: {
      “my_custom_field4”: “value_of_my_custom_field4”
    },
    “fields”: {
      “my_custom_field5”: “value_of_my_custom_field5”
    },

    “agent”: {
      “hostname”: “ubuntu-xenial”,
      “id”: “6fc91322-3f72-494f-9b52-12df04833853”,
      “version”: “7.3.1”,
      “type”: “filebeat”,
      “ephemeral_id”: “870554f1-30be-4e9f-9e1f-e288c37c6971”
    },
    “ecs”: {
      “version”: “1.0.1”
    },
    “log”: {
      “offset”: 13538,
      “file”: {
        “path”: “/var/log/containers/booksservice-v1.0-5bcd5fddbd-cklsw_nl-amis-testing_booksservice-v1-0-container-171b2526ae9d147dab7fb2180764d55c03c7ad706eca605ad5c849aafef5d38d.log”
      }
    },
    “stream”: “stdout”,
    “my_custom_field1”: “value_of_my_custom_field1”,
    “kubernetes”: {
      “container”: {
        “name”: “booksservice-v1-0-container”
      },
      “namespace”: “nl-amis-testing”,
      “replicaset”: {
        “name”: “booksservice-v1.0-5bcd5fddbd”
      },
      “labels”: {
        “version”: “1.0”,
        “app”: “booksservice”,
        “environment”: “testing”,
        “pod-template-hash”: “5bcd5fddbd”
      },
      “pod”: {
        “name”: “booksservice-v1.0-5bcd5fddbd-cklsw”,
        “uid”: “20936e0c-cf4d-11e9-95ef-023e591c269a”
      },
      “node”: {
        “name”: “minikube”
      }
    },
    “my-custom-sub-dictionary1”: {
      “my_custom_field3”: “value_of_my_custom_field3”,
      “my_custom_field2”: “value_of_my_custom_field2”
    },

    “host”: {
      “architecture”: “x86_64”,
      “os”: {
        “kernel”: “4.4.0-142-generic”,
        “codename”: “Core”,
        “platform”: “centos”,
        “version”: “7 (Core)”,
        “family”: “redhat”,
        “name”: “CentOS Linux”
      },
      “name”: “ubuntu-xenial”,
      “containerized”: false,
      “hostname”: “ubuntu-xenial”
    }
  },
  “fields”: {
    “@timestamp”: [
      “2019-09-12T14:57:18.195Z”
    ],
    “suricata.eve.timestamp”: [
      “2019-09-12T14:57:18.195Z”
    ]
  },
  “sort”: [
    1568300238195
  ]
}

This meant that, enhancing the export data, by adding some extra fields worked.

So now it’s time to conclude this article. I tried out some of the functionality of Elastic Filebeat. Besides log aggregation (getting log information available at a centralized location), I also described how I used filtering and enhancing the exported log data with Filebeat.

In a next article I will also be using Logstash between Filebeat and ElasticSearch.

About Author

Marc, active in IT (and with Oracle) since 1995, is a Principal Oracle SOA Consultant with focus on Oracle Cloud, Oracle Service Bus, Oracle SOA Suite, Oracle Database (SQL & PL/SQL) and Java, Docker, Kubernetes, Minikube and Helm. He's Oracle SOA Suite 12c Certified Implementation Specialist. Over the past 20 years he has worked for several customers in the Netherlands. Marc likes to share his knowledge through publications, blog’s and presentations.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.