Adding Grafana to my VM with Minikube and Elasticsearch

Marc Lameriks

For a demo, I needed an environment with Grafana.

Lucky for me, I had the configuration for such an environment using Vagrant and Oracle VirtualBox. In the past, I already set up such a demo environment, available within an Oracle VirtualBox appliance, as I described in a series of articles.
[https://technology.amis.nl/2019/09/15/using-elastic-stack-filebeat-for-log-aggregation/]

This demo environment was built with quite some files, and included:

  • guest Operating System (Ubuntu)
  • Docker
  • Minikube
  • Kubectl
  • Helm
  • Elasticsearch
  • Filebeat
  • Logstash
  • Kibana
  • MySQL

In this article, I will share with you the steps I took, to get Grafana in combination with Elasticsearch (as the data source), working on my demo environment. I also will describe how I created a simple dashboard with two panels with different visualizations.

Grafana

Dashboard anything. Observe everything.
Query, visualize, alert on, and understand your data no matter where it’s stored. With Grafana you can create, explore and share all of your data through beautiful, flexible dashboards.
[https://grafana.com/grafana/]

I will shortly describe some of the Grafana key features, for more information please see the Grafana website.

Grafana key features:

  • Panels
    From heatmaps to histograms. Graphs to geomaps. Grafana has fast and flexible visualizations that allows you to visualize your data, any way you want.
  • Plugins
    Connect your tools and your teams with Grafana plugins. Data source plugins hook into existing data sources via APIs and render the data in real time without requiring you to migrate or ingest your data.
  • Alerts
    With Grafana alerting, you can create, manage, and silence all of your alerts within one simple UI— allowing you to easily consolidate and centralize all of your alerts.
  • Transformations
    Transformations allow you to rename, summarize, combine, and perform calculations across different queries and data sources.
  • Annotations
    Annotate graphs with rich events from different data sources. Hover over events shows you the full event metadata and tags.
  • Panel editor
    The panel editor makes it easy to configure, customize and explore all of your panels with a consistent UI for setting data options across all of your visualizations.

[https://grafana.com/grafana/]

Choose the version of Grafana that’s best for you

Get Grafana fully managed with Grafana Cloud or run on your own infrastructure with self-managed options.

  • Open source
    Centralize the analysis, visualization, and alerting for all of your data with Grafana.

    For users who prefer to set up, administer, and maintain their own installation.

  • Cloud
    Offered as a fully managed service, Grafana Cloud is the fastest way to adopt Grafana and includes a scalable, managed backend for metrics, logs, and traces.

    Managed and administered by Grafana Labs with free and paid options for individuals, teams, and large enterprises.

    Includes a robust free tier with access to 10k metrics, 50GB logs and 50GB traces for 3 users.

  • Enterprise
    Grafana’s powerful visualization and alerting, enhanced with access to Enterprise data source plugins and built-in collaboration features.

    For organizations that have specific privacy or security requirements and need a self-managed environment.

[https://grafana.com/grafana/]

For my purpose I opted for the self-managed open source version.

Installing Grafana

I wanted to setup Grafana in my demo environment, which already included Minikube (a local Kubernetes cluster). On the Grafana website, I found an example Kubernetes manifest file.
[https://grafana.com/docs/grafana/latest/installation/kubernetes/?pg=oss-graf&plcmt=resources#deploy-grafana-on-kubernetes]

In line with how a previously set up my environment , from this example manifest file, I created the following manifest files:

  • persistent-volume-claim-grafana.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  • deployment-grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:7.5.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
  • service-grafana.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None
  type: LoadBalancer

Vagrantfile

I changed the content of Vagrantfile to:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.define "ubuntu_minikube_helm_elastic" do |ubuntu_minikube_helm_elastic|
  
    config.vm.network "forwarded_port",
      guest: 3000,
      host:  3000,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 8001,
      host:  8001,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5601,
      host:  5601,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9200,
      host:  9200,
      auto_correct: true  
      
    config.vm.network "forwarded_port",
      guest: 9010,
      host:  9010,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9020,
      host:  9020,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9110,
      host:  9110,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Minikube Helm Elastic Stack"
        vb.memory = "8192"
        vb.cpus = "2"
        
    args = []
    config.vm.provision "docker shell script", type: "shell",
        path: "scripts/docker.sh",
        args: args
        
    args = []
    config.vm.provision "minikube shell script", type: "shell",
        path: "scripts/minikube.sh",
        args: args
        
    args = []
    config.vm.provision "kubectl shell script", type: "shell",
        path: "scripts/kubectl.sh",
        args: args
        
            args = []
    config.vm.provision "helm shell script", type: "shell",
        path: "scripts/helm.sh",
        args: args
        
    args = []
    config.vm.provision "namespaces shell script", type: "shell",
        path: "scripts/namespaces.sh",
        args: args
        
    args = []
    config.vm.provision "elasticsearch shell script", type: "shell",
        path: "scripts/elasticsearch.sh",
        args: args
        
    args = []
    config.vm.provision "kibana shell script", type: "shell",
        path: "scripts/kibana.sh",
        args: args
        
    args = []
    config.vm.provision "logstash shell script", type: "shell",
        path: "scripts/logstash.sh",
        args: args
        
    args = []
    config.vm.provision "filebeat shell script", type: "shell",
        path: "scripts/filebeat.sh",
        args: args
        
    args = []
    config.vm.provision "mysql shell script", type: "shell",
        path: "scripts/mysql.sh",
        args: args
        
    args = []
    config.vm.provision "booksservices shell script", type: "shell",
        path: "scripts/booksservices.sh",
        args: args
        
    args = []
    config.vm.provision "grafana shell script", type: "shell",
        path: "scripts/grafana.sh",
        args: args
    end
    
  end

end

As you can see in this file, I used shell scripts to install the software needed for my demo environment. Besides already being able to use the Kubernetes Dashboard (in a Web Browser) on my Windows laptop (via port forwarding), I also wanted to be able to the Grafana Dashboard on my Windows laptop. So again, I used the forwarded_port configuration option, to forward a port on my host (Windows) to a port on my guest (Ubuntu).

Navigate to localhost:3000 in your browser and you should see a Grafana login page.
[https://grafana.com/docs/grafana/latest/installation/kubernetes/?pg=oss-graf&plcmt=resources#send-manifest-to-kubernetes-api-server]

Vagrant forwarded ports allow you to access a port on your host machine and have all data forwarded to a port on the guest machine, over either TCP or UDP.
[https://www.vagrantup.com/docs/networking/forwarded_ports.html]

In the scripts directory I created a file grafana.sh with the following content:

#!/bin/bash
echo "**** Begin installing Grafana"

#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/grafana-chart/*
helm create grafana-chart

rm -rf /vagrant/helmcharts/grafana-chart/templates/*
cp /vagrant/yaml/*grafana.yaml /vagrant/helmcharts/grafana-chart/templates

# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart grafana-chart"
helm install grafana-release ./grafana-chart

# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150

#List helm releases
echo "**** List helm releases"
helm list -d

echo "**** Forward local port 3000 to port 3000 on the grafana service"
kubectl port-forward service/grafana 3000:3000 </dev/null &>/dev/null &

echo "**** End installing Grafana"

As you can see, I used Helm (the package manager for Kubernetes), to deploy the Kubernetes resources, as I did and described before.
[https://technology.amis.nl/continuous-delivery/containers/using-helm-the-package-manager-for-kubernetes-to-install-two-versions-of-a-restful-web-service-spring-boot-application-within-minikube/]

From the subdirectory named env on my Windows laptop, I opened a Windows Command Prompt (cmd) and typed: vagrant up

This command creates and configures guest machines according to your Vagrantfile.
[https://www.vagrantup.com/docs/cli/up.html]

With the following output (only showing the part about Grafana):

    ubuntu_minikube_helm_elastic: **** Install Helm chart grafana-chart
    ubuntu_minikube_helm_elastic: NAME: grafana-release
    ubuntu_minikube_helm_elastic: LAST DEPLOYED: Sat Mar 12 16:13:52 2022
    ubuntu_minikube_helm_elastic: NAMESPACE: default
    ubuntu_minikube_helm_elastic: STATUS: deployed
    ubuntu_minikube_helm_elastic: REVISION: 1
    ubuntu_minikube_helm_elastic: TEST SUITE: None
    ubuntu_minikube_helm_elastic: **** Waiting 2,5 minute ...
    ubuntu_minikube_helm_elastic: **** List helm releases
    ubuntu_minikube_helm_elastic: NAME                  NAMESPACE       REVISION        UPDATED                                       STATUS          CHART                           APP VERSION
    ubuntu_minikube_helm_elastic: namespace-release     default         1               2022-03-12 15:57:45.147992882 +0000 UTC       deployed        namespace-chart-0.1.0           1.16.0
    ubuntu_minikube_helm_elastic: elasticsearch-release default         1               2022-03-12 15:58:16.392846066 +0000 UTC       deployed        elasticsearch-chart-0.1.0       1.16.0
    ubuntu_minikube_helm_elastic: kibana-release        default         1               2022-03-12 16:00:47.91021763 +0000 UTC        deployed        kibana-chart-0.1.0              1.16.0
    ubuntu_minikube_helm_elastic: logstash-release      default         1               2022-03-12 16:03:20.271988972 +0000 UTC       deployed        logstash-chart-0.1.0            1.16.0
    ubuntu_minikube_helm_elastic: filebeat-release      default         1               2022-03-12 16:05:52.295700964 +0000 UTC       deployed        filebeat-chart-0.1.0            1.16.0
    ubuntu_minikube_helm_elastic: mysql-release         default         1               2022-03-12 16:07:00.323238623 +0000 UTC       deployed        mysql-chart-0.1.0               1.16.0
    ubuntu_minikube_helm_elastic: booksservice-release  default         1               2022-03-12 16:11:19.453269408 +0000 UTC       deployed        booksservice-chart-0.1.0        1.16.0
    ubuntu_minikube_helm_elastic: grafana-release       default         1               2022-03-12 16:13:52.662794581 +0000 UTC       deployed        grafana-chart-0.1.0             1.16.0
    ubuntu_minikube_helm_elastic: **** Forward local port 3000 to port 3000 on the grafana service
    ubuntu_minikube_helm_elastic: **** End installing Grafana

Grafana Dashboard

On my Windows laptop, after my demo environment was set up, in a Web Browser I tried to start the Grafana Dashboard (via: localhost:3000), with the following response:

This site can’t be reached

The webpage at http://localhost:3000/ might be temporarily down or it may have moved permanently to a new web address.

ERR_SOCKET_NOT_CONNECTED

In order to investigate the problem, I wanted to use the Kubernetes Dashboard.

Kubernetes Dashboard

In the Web Browser on my Windows laptop, I started the Kubernetes Dashboard:

http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/service?namespace=_all

With the following output:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "services \"kubernetes-dashboard\" not found",
  "reason": "NotFound",
  "details": {
    "name": "kubernetes-dashboard",
    "kind": "services"
  },
  "code": 404
}

I remembered, that in my demo environment in order to use the Kubernetes Web UI (Dashboard), since the last VM upgrade, I had to do some manual steps.
[https://technology.amis.nl/continuous-delivery/containers/trouble-shooting-while-upgrading-a-vm-with-minikube-elasticsearch-and-kibana-using-the-vm-drivernone-option-on-my-windows-laptop-using-vagrant-and-oracle-virtualbox/]

I used vagrant ssh to open a Linux Command Prompt where I used the following command:

kubectl get pods --all-namespaces

With the following output:

NAMESPACE             NAME                                 READY   STATUS    RESTARTS   AGE
default               grafana-756fb84d84-w5rcq             1/1     Running   0          12m
kube-system           coredns-78fcd69978-qlhl4             1/1     Running   0          31m
kube-system           etcd-minikube                        1/1     Running   0          31m
kube-system           kube-apiserver-minikube              1/1     Running   0          31m
kube-system           kube-controller-manager-minikube     1/1     Running   0          31m
kube-system           kube-proxy-bhvxv                     1/1     Running   0          31m
kube-system           kube-scheduler-minikube              1/1     Running   0          31m
kube-system           storage-provisioner                  1/1     Running   0          31m
nl-amis-development   booksservice-v1.0-7998bd45ff-6jttk   1/1     Running   0          15m
nl-amis-development   booksservice-v1.0-7998bd45ff-p7whh   1/1     Running   0          15m
nl-amis-development   booksservice-v2.0-6649c5d4d8-dprgr   1/1     Running   0          15m
nl-amis-development   booksservice-v2.0-6649c5d4d8-n8h24   1/1     Running   0          15m
nl-amis-logging       elasticsearch-bc5f6d549-wdwq6        1/1     Running   0          28m
nl-amis-logging       filebeat-daemonset-972s9             1/1     Running   0          20m
nl-amis-logging       kibana-f4665d556-vhsm2               1/1     Running   0          25m
nl-amis-logging       logstash-7d85759479-76bzk            1/1     Running   0          23m
nl-amis-testing       booksservice-v1.0-b7d5d44f4-djjsh    1/1     Running   0          15m
nl-amis-testing       booksservice-v1.0-b7d5d44f4-t699h    1/1     Running   0          15m
nl-amis-testing       mysql-787c788fd6-nzx86               1/1     Running   0          19m

As I expected the Kubernetes Dashboard was not yet deployed.

I used vagrant ssh to open a Linux Command Prompt where I used the following command:

minikube dashboard --url

With the following output:

    Enabling dashboard ...
    ▪ Using image kubernetesui/dashboard:v2.3.1
    ▪ Using image kubernetesui/metrics-scraper:v1.0.7
    Verifying dashboard health ...
    Launching proxy ...
    Verifying proxy health ...
http://127.0.0.1:40061/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

Then, to stop this command, I typed Ctrl-C to return to my Linux Command Prompt.

Remark about getting just the dashboard URL:
If you don’t want to open a web browser, the dashboard command can also simply emit a URL:

minikube dashboard --url

[https://minikube.sigs.k8s.io/docs/handbook/dashboard/]

Next, I used vagrant ssh to open a Linux Command Prompt where I used the following command:

kubectl get pods --all-namespaces

With the following output:

NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                grafana-756fb84d84-w5rcq                     1/1     Running   0          18m
kube-system            coredns-78fcd69978-qlhl4                     1/1     Running   0          36m
kube-system            etcd-minikube                                1/1     Running   0          37m
kube-system            kube-apiserver-minikube                      1/1     Running   0          37m
kube-system            kube-controller-manager-minikube             1/1     Running   0          37m
kube-system            kube-proxy-bhvxv                             1/1     Running   0          36m
kube-system            kube-scheduler-minikube                      1/1     Running   0          37m
kube-system            storage-provisioner                          1/1     Running   0          37m
kubernetes-dashboard   dashboard-metrics-scraper-5594458c94-fjssq   1/1     Running   0          3m8s
kubernetes-dashboard   kubernetes-dashboard-654cf69797-g49k2        1/1     Running   0          3m8s
nl-amis-development    booksservice-v1.0-7998bd45ff-6jttk           1/1     Running   0          20m
nl-amis-development    booksservice-v1.0-7998bd45ff-p7whh           1/1     Running   0          20m
nl-amis-development    booksservice-v2.0-6649c5d4d8-dprgr           1/1     Running   0          20m
nl-amis-development    booksservice-v2.0-6649c5d4d8-n8h24           1/1     Running   0          20m
nl-amis-logging        elasticsearch-bc5f6d549-wdwq6                1/1     Running   0          33m
nl-amis-logging        filebeat-daemonset-972s9                     1/1     Running   0          26m
nl-amis-logging        kibana-f4665d556-vhsm2                       1/1     Running   0          31m
nl-amis-logging        logstash-7d85759479-76bzk                    1/1     Running   0          28m
nl-amis-testing        booksservice-v1.0-b7d5d44f4-djjsh            1/1     Running   0          20m
nl-amis-testing        booksservice-v1.0-b7d5d44f4-t699h            1/1     Running   0          20m
nl-amis-testing        mysql-787c788fd6-nzx86                       1/1     Running   0          25m

I could see that the Pods for the dashboard were running.

minikube.sh

In order to avoid these manual steps, I changed the content of my minikube.sh script to:
[in bold, I highlighted the changes]

#!/bin/bash
echo "**** Begin downloading minikube"

#Kubernetes 1.22.3 requires conntrack to be installed in root's path
sudo apt install -y conntrack

#Download a static binary
curl -o minikube https://storage.googleapis.com/minikube/releases/v1.24.0/minikube-linux-amd64
chmod +x minikube

#Add the Minikube executable to your path
sudo cp minikube /usr/local/bin/ 
rm minikube

echo "**** End downloading minikube"

echo "**** Begin starting a Cluster"

sudo sysctl fs.protected_regular=0

#Start a Cluster
minikube start \
  --vm-driver=none \
  --extra-config=kubeadm.node-name=minikube \
  --extra-config=kubelet.hostname-override=minikube

#To use kubectl or minikube commands as your own user, you may need to relocate them.
sudo cp -R /root/.kube /root/.minikube /home/vagrant
sudo chown -R vagrant /home/vagrant/.kube /home/vagrant/.minikube

sed -i 's/root/home\/vagrant/g' /home/vagrant/.kube/config

echo "**** End starting a Cluster"

echo "**** Begin preparing dashboard"
minikube dashboard --url </dev/null &>/dev/null &
echo "**** End preparing dashboard"

minikube kubectl -- get pods -A

Remark about the background process and redirecting the standard IO streams:
To run a command in the background, add the ampersand symbol (&) at the end of the command.
The shell job ID (surrounded with brackets) and process ID will be printed on the terminal.

command &

Before a command is executed, its standard input (file descriptor 0), standard output (file descriptor 1) and standard error output (file descriptor 2) may be redirected using a special notation interpreted by the shell.

Redirection of input causes the file whose name results from the expansion of word to be opened for reading on file descriptor n, or the standard input (file descriptor 0) if n is not specified.

[n]<word

The following construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be redirected to the file whose name is the expansion of word.

&>word

[https://www.gnu.org/software/bash/manual/html_node/Redirections.html]

The null device /dev/null is a device file that discards all data written to it but reports that the write operation succeeded. The null device is typically used for disposing of unwanted output streams of a process, or as a convenient empty file for input streams. This is usually done by redirection.
[https://en.wikipedia.org/wiki/Null_device]

Remark:
In order to stop the running machine and destroy its resources, I used the following command on the Windows Command Prompt: vagrant destroy

I won’t mention this step for readability, but after each problem fix, I used it.

Again, on my Windows laptop, I used vagrant up and looked at the output and then started the Kubernetes Web UI (Dashboard):

I could see that the service grafana was pending (see orange dot).
By clicking on this service, I could also see that the Pod was running.

I used vagrant ssh to open a Linux Command Prompt where I used the following command:

kubectl get services

With the following output:

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
grafana      LoadBalancer   10.100.85.101   <pending>     3000:30066/TCP   7h23m
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP          7h42m

As it turned out, if you are using a custom Kubernetes Cluster (using minikube, kubeadm or the like), there is no LoadBalancer integrated with the default setup. LoadBalancer is a valid service type but it is being used in a non-compatible platform (at least by default). In order to use LoadBalancer you must have a platform that can provide external IPs to the pods.

If you are using minikube the run “minikube tunnel”. Now when you check your services you will get the public ip.
[https://stackoverflow.com/questions/44110876/kubernetes-service-external-ip-pending]

There are two major categories of services in Kubernetes:

  • NodePort
  • LoadBalancer

minikube supports either.

NodePort access
A NodePort service is the most basic way to get external traffic directly to your service. NodePort, as the name implies, opens a specific port, and any traffic that is sent to this port is forwarded to the service.

LoadBalancer access
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets its own IP address.

Using minikube tunnel
Services of type LoadBalancer can be exposed via the minikube tunnel command. It must be run in a separate terminal window to keep the LoadBalancer running. Ctrl-C in the terminal can be used to terminate the process at which time the network routes will be cleaned up.
[https://minikube.sigs.k8s.io/docs/handbook/accessing/]

In my demo environment I already used NodePort access for services. So, I wanted to try out the steps described for LoadBalancer access.

Again, I used vagrant ssh to open an extra Linux Command Prompt where I used the following command:

minikube tunnel

With the following output:

Status:
        machine: minikube
        pid: 57801
        route: 10.96.0.0/12 -> 10.0.2.15
        minikube: Running
        services: [grafana]
    errors:
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors

Then, to stop this command, I typed Ctrl-C to return to my Linux Command Prompt.

I wanted to use port forwarding. So, in VirtualBox, for my appliance I changed the Network settings.

For Adapter 1, by choosing “Advanced” and clicking on button “Port Forwarding”, I added a Port Forwarding Rule, this time for Host Port 3001, which I forwarded to Guest Port 3001.

Remark:
Because I already set everything up for port 3000 (see file grafana.sh) to be forwarded, I knew that port was already in use, so now I temporarily used port 3001.

In my other Linux Command Prompt, again I used the following command:

kubectl get services

With the following output:

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
grafana      LoadBalancer   10.100.85.101   10.100.85.101   3000:30066/TCP   7h31m
kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP          7h50m

Then I used the following command:

socat tcp-listen:3001,fork tcp:10.100.85.101:3000 &

With the following output:

[1] 59758

Then, on my Windows laptop, I refreshed the Kubernetes Web UI (Dashboard):

I could see that the service grafana was running (see green dot).
By clicking on this service, I could also see that the Pod was running.

Grafana Dashboard

Next, on my Windows laptop, in a Web Browser I started the Grafana Dashboard (via: localhost:3000), with the expected result:

So, using the LoadBalancer access for service Grafana was now working.

However, for my demo environment, I wanted to keep things simple, so I decided to use NodePort access. I noticed that the namespace default was used for the Grafana resources and I wanted to change that also. And, lucky for me, version 8.4.4 was just released (March 16, 2022), so I could use that version.
[https://grafana.com/grafana/download?edition=oss&pg=oss-graf&plcmt=resources&platform=docker]

Adding another namespace

I did need another namespace, so I added to the yaml directory a file namespace-visualization.yaml with the following content:

apiVersion: v1
kind: Namespace
metadata:
  name: "nl-amis-visualization"
  labels:
    name: "nl-amis-visualization"

Changing Kubernetes manifest files

In line with how a previously set up my environment, I changed the content of the following manifest files (with my own namespace and labels):
[in bold, I highlighted the changes]

  • persistent-volume-claim-grafana.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: nl-amis-visualization
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  • deployment-grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: nl-amis-visualization
spec:
  selector:
    matchLabels:
      app: grafana
      version: "8.4.4"
      environment: visualization
  template:
    metadata:
      labels:
        app: grafana
        version: "8.4.4"
        environment: visualization
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana-oss:8.4.4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
  • service-grafana.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  namespace: nl-amis-visualization
  labels:
    app: grafana
    version: "8.4.4"
    environment: visualization
spec:
  ports:
    - nodePort: 30300
      port: 3000
      targetPort: 3000
  selector:
    app: grafana
    version: "8.4.4"
    environment: visualization
  sessionAffinity: None
  type: NodePort

Again, on my Windows laptop, I used vagrant up and looked at the output and then started the
Grafana Dashboard (via: localhost:3000), with the following response:

This site can’t be reached

The webpage at http://localhost:3000/ might be temporarily down or it may have moved permanently to a new web address.

ERR_SOCKET_NOT_CONNECTED

It looked like the port forwarding went wrong, probably due to the namespace change.

Remember I used the following:

echo "**** Forward local port 3000 to port 3000 on the grafana service"
kubectl port-forward service/grafana 3000:3000 </dev/null &>/dev/null &

But the service/grafana is now no longer in the namespace default and I also changed the name of the service to grafana-service.

Because the service now is of type NodePort, I am able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
[https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types]

In the scripts directory I therefor changed the content of file grafana.sh to:
[in bold, I highlighted the changes]

#!/bin/bash
echo "**** Begin installing Grafana"

#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/grafana-chart/*
helm create grafana-chart

rm -rf /vagrant/helmcharts/grafana-chart/templates/*
cp /vagrant/yaml/*grafana.yaml /vagrant/helmcharts/grafana-chart/templates

# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart grafana-chart"
helm install grafana-release ./grafana-chart

# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150

#List helm releases
echo "**** List helm releases"
helm list -d

#List pods
echo "**** List pods with namespace nl-amis-visualization"
kubectl get pods --namespace nl-amis-visualization

#List services
echo "**** List services with namespace nl-amis-visualization"
kubectl get service --namespace nl-amis-visualization

echo "**** Determine the IP of the minikube node"
nodeIP=$(kubectl get node minikube -o yaml | grep address: | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")
echo "---$nodeIP---"

echo "**** Via socat forward local port 3000 to port 30300 on the minikube node ($nodeIP)"
socat tcp-listen:3000,fork tcp:$nodeIP:30300 &

echo "**** End installing Grafana"

Again, on my Windows laptop, I used vagrant up.

With the following output:
[only showing the part about grafana]

    ubuntu_minikube_helm_elastic: **** Begin installing Grafana
    ubuntu_minikube_helm_elastic: **** Create Helm chart
    ubuntu_minikube_helm_elastic: Creating grafana-chart
    ubuntu_minikube_helm_elastic: WARNING: File "/vagrant/helmcharts/grafana-chart/.helmignore" already exists. Overwriting.
    ubuntu_minikube_helm_elastic: **** Install Helm chart grafana-chart
    ubuntu_minikube_helm_elastic: NAME: grafana-release
    ubuntu_minikube_helm_elastic: LAST DEPLOYED: Mon Mar 21 09:16:33 2022
    ubuntu_minikube_helm_elastic: NAMESPACE: default
    ubuntu_minikube_helm_elastic: STATUS: deployed
    ubuntu_minikube_helm_elastic: REVISION: 1
    ubuntu_minikube_helm_elastic: TEST SUITE: None
    ubuntu_minikube_helm_elastic: **** Waiting 2,5 minute ...
    ubuntu_minikube_helm_elastic: **** List helm releases
    ubuntu_minikube_helm_elastic: NAME                  NAMESPACE REVISION UPDATED                                 STATUS          CHART                      APP VERSION
    ubuntu_minikube_helm_elastic: namespace-release     default   1        2022-03-21 09:00:04.623690639 +0000 UTC deployed        namespace-chart-0.1.0      1.16.0
    ubuntu_minikube_helm_elastic: elasticsearch-release default   1        2022-03-21 09:00:35.937620941 +0000 UTC deployed        elasticsearch-chart-0.1.0  1.16.0
    ubuntu_minikube_helm_elastic: kibana-release        default   1        2022-03-21 09:03:07.758253888 +0000 UTC deployed        kibana-chart-0.1.0         1.16.0
    ubuntu_minikube_helm_elastic: logstash-release      default   1        2022-03-21 09:05:39.588711025 +0000 UTC deployed        logstash-chart-0.1.0       1.16.0
    ubuntu_minikube_helm_elastic: filebeat-release      default   1        2022-03-21 09:08:13.175288429 +0000 UTC deployed        filebeat-chart-0.1.0       1.16.0
    ubuntu_minikube_helm_elastic: mysql-release         default   1        2022-03-21 09:09:24.53031323 +0000 UTC  deployed        mysql-chart-0.1.0          1.16.0
    ubuntu_minikube_helm_elastic: booksservice-release  default   1        2022-03-21 09:13:59.647836159 +0000 UTC deployed        booksservice-chart-0.1.0   1.16.0
    ubuntu_minikube_helm_elastic: grafana-release       default   1        2022-03-21 09:16:33.88304826 +0000 UTC  deployed        grafana-chart-0.1.0        1.16.0
    ubuntu_minikube_helm_elastic: **** List pods with namespace nl-amis-visualization
    ubuntu_minikube_helm_elastic: NAME                      READY   STATUS    RESTARTS   AGE
    ubuntu_minikube_helm_elastic: grafana-d4d47f944-thbx8   1/1     Running   0          2m30s
    ubuntu_minikube_helm_elastic: **** List services with namespace nl-amis-visualization
    ubuntu_minikube_helm_elastic: NAME              TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
    ubuntu_minikube_helm_elastic: grafana-service   NodePort   10.111.132.102   <none>        3000:30300/TCP   2m30s
    ubuntu_minikube_helm_elastic: **** Determine the IP of the minikube node
    ubuntu_minikube_helm_elastic: ---10.0.2.15---
    ubuntu_minikube_helm_elastic: **** Via socat forward local port 3000 to port 30300 on the minikube node (10.0.2.15)
    ubuntu_minikube_helm_elastic: **** End installing Grafana

Grafana Dashboard log in

On my Windows laptop, in a Web Browser I started the Grafana Dashboard (via: localhost:3000), with the expected result:

To log in to Grafana for the first time:

  1. Open your web browser and go to http://localhost:3000/. The default HTTP port that Grafana listens to is 3000 unless you have configured a different port.
  2. On the login page, enter admin for username and password.
  3. Click Log in. If login is successful, then you will see a prompt to change the password.
  4. Click OK on the prompt, then change your password.


[https://grafana.com/docs/grafana/latest/getting-started/getting-started/#step-2-log-in]

And there was the Welcome screen:

Data sources

Grafana supports many different storage backends for your time series data (data source). Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.
[https://grafana.com/docs/grafana/latest/datasources/#data-sources]

One of the supported data sources is Elasticsearch and I opted for that one.

Grafana ships with advanced support for Elasticsearch. You can do many types of simple or complex Elasticsearch queries to visualize logs or metrics stored in Elasticsearch. You can also annotate your graphs with log events stored in Elasticsearch.

I followed the instructions from “Using Elasticsearch in Grafana” to set it up.
[https://grafana.com/docs/grafana/latest/datasources/elasticsearch/]

Adding the Elasticsearch data source

In the left side menu, I clicked on Configuration | Data Sources.

Next, I clicked on the button “Add data source” and search (via the Filter field) for Elasticsearch.

I selected the Elasticsearch data source.

I kept the default settings, but did change/fill in:

Remark about the Elasticsearch details, Index name:
In my demo environment the Logstash configuration was set up to create an Elasticsearch index with the format:

logstash-via-filebeat-%{+YYYY.MM.dd}

For example: logstash-via-filebeat-2022.03.21
[https://technology.amis.nl/continuous-delivery/containers/using-elastic-stack-filebeat-and-logstash-for-log-aggregation/]

Then, I clicked on button “Save & Test”, with the following output:

Elasticsearch error: Bad Gateway

So, I changed the value for HTTP, URL to:

http://elasticsearch-service.nl-amis-logging:9200

Remark about service name lookup:
A cluster-aware DNS server, such as CoreDNS, watches the Kubernetes API for new Services and creates a set of DNS records for each one. If DNS has been enabled throughout your cluster then all Pods should automatically be able to resolve Services by their DNS name.

For example, if you have a Service called my-service in a Kubernetes namespace my-ns, the control plane and the DNS Service acting together create a DNS record for my-service.my-ns. Pods in the my-ns namespace should be able to find the service by doing a name lookup for my-service (my-service.my-ns would also work).

Pods in other namespaces must qualify the name as my-service.my-ns. These names will resolve to the cluster IP assigned for the Service.
[https://kubernetes.io/docs/concepts/services-networking/service/#dns]

Then again, I clicked on button “Save & Test”, with the following output:

Index OK. Time field name OK.

Postman

In order for the Elasticsearch index to have some data, as I described in a previous article, I used Postman to add books to and retrieve books from the book catalog. I did this for version 1.0 and 2.0 of the BooksService application.
[https://technology.amis.nl/2019/09/15/using-elastic-stack-filebeat-for-log-aggregation/]

Grafana Explore

Grafana’s dashboard UI is all about building dashboards for visualization. Explore strips away the dashboard and panel options so that you can focus on the query. It helps you iterate until you have a working query and then think about building a dashboard.
[https://grafana.com/docs/grafana/latest/explore/]

From the menu on the left, I choose Explore:

Here, I made the following changes:

Group By: Terms
Select Field: kubernetes.labels.app.keyword

With the following result (showing the data in a Table):

The Elasticsearch query editor allows you to select multiple metrics and group by multiple terms or filters. Use the plus and minus icons to the right to add/remove metrics or group by clauses. Some metrics and group by clauses have options, click the option text to expand the row to view and edit metric or group by options.
[https://grafana.com/docs/grafana/latest/datasources/elasticsearch/]

Remark about the result:
In my demo environment, I setup Logstash in such a way, that only the logfile data from the containers for MySQL (mysql*.log) and my booksservices (booksservices*.log) were used in Elasticsearch.
[https://technology.amis.nl/continuous-delivery/containers/using-elastic-stack-filebeat-for-log-aggregation/]

This is an example Kubernetes manifest file (service-booksservice-dev-v1.0.yaml), I used in my demo environment:

kind: Service
apiVersion: v1
metadata:
  name: booksservice-v1-0-service
  namespace: nl-amis-development
  labels:
    app: booksservice
    version: "1.0"
    environment: development
spec:
  selector:
    app: booksservice
    version: "1.0"
    environment: development
  type: NodePort
  ports:
  - protocol: TCP
    nodePort: 30010
    port: 9190
    targetPort: 9090

This relates to the Term: kubernetes.labels.app.keyword

So, now I knew I had a working query that gave me some results. I now could use this query in a dashboard.

Grafana Dashboards

A dashboard is a set of one or more panels organized and arranged into one or more rows. Grafana ships with a variety of Panels. Grafana makes it easy to construct the right queries, and customize the display properties so that you can create the perfect dashboard for your need. Each panel can interact with data from any configured Grafana Data Source (currently Graphite, Prometheus, Elasticsearch, InfluxDB, OpenTSDB, MySQL, PostgreSQL, Microsoft SQL Server and AWS Cloudwatch).
[https://grafana.com/docs/grafana/latest/dashboards/]

From the menu on the left, I choose Dashboards | Browse:

Then I clicked on button “New Dashboard”:

Next, I clicked on “Add a new panel”:

Here, I made the following changes:

Group By: Terms
Select Field: kubernetes.labels.app.keyword

With the following result (showing no data):

On the top right, I opened the Visualization combo box:

I choose “Pie chart” as visualization:

In “Value options | Show”, I clicked on button “All values”:

In “Panel options | Title”, I changed the title to “Apps” and in “Pie chart | Labels”, I selected “Value”:

Then, I clicked on button “Save” and save the dashboard as “Dashboard1” :

With the following result:

Adding another panel to the dashboard

On the top right , I clicked on icon “Add Panel”, followed by “Add a new panel”:

In a similar way as before, I made the following changes:

Group By: Terms
Select Field: kubernetes.labels.environment.keyword

I choose “Gauge” as visualization, in “Value options | Show”, I clicked on button “All values” and I changed the title to “Environment”.

Next, I clicked on button “Save”. In the “Save dashboard” pop-up I added a note and clicked “Save” again.

With the following result:

As I expected, the label “environment” in my demo environment has the value development or testing. See, for example the Kubernetes manifest file (service-booksservice-dev-v1.0.yaml), I described in this article.

With this simple dashboard (with two panels with different visualizations), for me it was obvious enough that my demo environment with Grafana in combination with Elasticsearch (as the data source), was working. It now gives me to possibility to explore the functionality of Grafana even further (and eventually also in combination with other data sources).

You can find the code, belonging to my article, here:
https://github.com/marclameriks/amis-technology-blog-2022-03-1

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Next Post

Apache NiFi: Avoid these common pitfalls

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It has a powerful UI which can be used for both development and operations. In addition, the NiFi Registry is available to make promoting software from one environment to the next, easy. In order […]
%d bloggers like this: