For a demo, I needed an environment with Grafana.
Lucky for me, I had the configuration for such an environment using Vagrant and Oracle VirtualBox. In the past, I already set up such a demo environment, available within an Oracle VirtualBox appliance, as I described in a series of articles.
[https://technology.amis.nl/2019/09/15/using-elastic-stack-filebeat-for-log-aggregation/]
This demo environment was built with quite some files, and included:
- guest Operating System (Ubuntu)
- Docker
- Minikube
- Kubectl
- Helm
- Elasticsearch
- Filebeat
- Logstash
- Kibana
- MySQL
In this article, I will share with you the steps I took, to get Grafana in combination with Elasticsearch (as the data source), working on my demo environment. I also will describe how I created a simple dashboard with two panels with different visualizations.
Grafana
Dashboard anything. Observe everything.
Query, visualize, alert on, and understand your data no matter where it’s stored. With Grafana you can create, explore and share all of your data through beautiful, flexible dashboards.
[https://grafana.com/grafana/]
I will shortly describe some of the Grafana key features, for more information please see the Grafana website.
Grafana key features:
- Panels
From heatmaps to histograms. Graphs to geomaps. Grafana has fast and flexible visualizations that allows you to visualize your data, any way you want. - Plugins
Connect your tools and your teams with Grafana plugins. Data source plugins hook into existing data sources via APIs and render the data in real time without requiring you to migrate or ingest your data. - Alerts
With Grafana alerting, you can create, manage, and silence all of your alerts within one simple UI— allowing you to easily consolidate and centralize all of your alerts. - Transformations
Transformations allow you to rename, summarize, combine, and perform calculations across different queries and data sources. - Annotations
Annotate graphs with rich events from different data sources. Hover over events shows you the full event metadata and tags. - Panel editor
The panel editor makes it easy to configure, customize and explore all of your panels with a consistent UI for setting data options across all of your visualizations.
[https://grafana.com/grafana/]
Choose the version of Grafana that’s best for you
Get Grafana fully managed with Grafana Cloud or run on your own infrastructure with self-managed options.
- Open source
Centralize the analysis, visualization, and alerting for all of your data with Grafana.For users who prefer to set up, administer, and maintain their own installation.
- Cloud
Offered as a fully managed service, Grafana Cloud is the fastest way to adopt Grafana and includes a scalable, managed backend for metrics, logs, and traces.Managed and administered by Grafana Labs with free and paid options for individuals, teams, and large enterprises.
Includes a robust free tier with access to 10k metrics, 50GB logs and 50GB traces for 3 users.
- Enterprise
Grafana’s powerful visualization and alerting, enhanced with access to Enterprise data source plugins and built-in collaboration features.For organizations that have specific privacy or security requirements and need a self-managed environment.
[https://grafana.com/grafana/]
For my purpose I opted for the self-managed open source version.
Installing Grafana
I wanted to setup Grafana in my demo environment, which already included Minikube (a local Kubernetes cluster). On the Grafana website, I found an example Kubernetes manifest file.
[https://grafana.com/docs/grafana/latest/installation/kubernetes/?pg=oss-graf&plcmt=resources#deploy-grafana-on-kubernetes]
In line with how a previously set up my environment , from this example manifest file, I created the following manifest files:
- persistent-volume-claim-grafana.yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
- deployment-grafana.yaml
apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: securityContext: fsGroup: 472 supplementalGroups: - 0 containers: - name: grafana image: grafana/grafana:7.5.2 imagePullPolicy: IfNotPresent ports: - containerPort: 3000 name: http-grafana protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /robots.txt port: 3000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 2 livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3000 timeoutSeconds: 1 resources: requests: cpu: 250m memory: 750Mi volumeMounts: - mountPath: /var/lib/grafana name: grafana-pv volumes: - name: grafana-pv persistentVolumeClaim: claimName: grafana-pvc
- service-grafana.yaml
apiVersion: v1 kind: Service metadata: name: grafana spec: ports: - port: 3000 protocol: TCP targetPort: http-grafana selector: app: grafana sessionAffinity: None type: LoadBalancer
Vagrantfile
I changed the content of Vagrantfile to:
[in bold, I highlighted the changes]
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/focal64"
config.vm.define "ubuntu_minikube_helm_elastic" do |ubuntu_minikube_helm_elastic|
config.vm.network "forwarded_port",
guest: 3000,
host: 3000,
auto_correct: true
config.vm.network "forwarded_port",
guest: 8001,
host: 8001,
auto_correct: true
config.vm.network "forwarded_port",
guest: 5601,
host: 5601,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9200,
host: 9200,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9010,
host: 9010,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9020,
host: 9020,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9110,
host: 9110,
auto_correct: true
config.vm.provider "virtualbox" do |vb|
vb.name = "Ubuntu Minikube Helm Elastic Stack"
vb.memory = "8192"
vb.cpus = "2"
args = []
config.vm.provision "docker shell script", type: "shell",
path: "scripts/docker.sh",
args: args
args = []
config.vm.provision "minikube shell script", type: "shell",
path: "scripts/minikube.sh",
args: args
args = []
config.vm.provision "kubectl shell script", type: "shell",
path: "scripts/kubectl.sh",
args: args
args = []
config.vm.provision "helm shell script", type: "shell",
path: "scripts/helm.sh",
args: args
args = []
config.vm.provision "namespaces shell script", type: "shell",
path: "scripts/namespaces.sh",
args: args
args = []
config.vm.provision "elasticsearch shell script", type: "shell",
path: "scripts/elasticsearch.sh",
args: args
args = []
config.vm.provision "kibana shell script", type: "shell",
path: "scripts/kibana.sh",
args: args
args = []
config.vm.provision "logstash shell script", type: "shell",
path: "scripts/logstash.sh",
args: args
args = []
config.vm.provision "filebeat shell script", type: "shell",
path: "scripts/filebeat.sh",
args: args
args = []
config.vm.provision "mysql shell script", type: "shell",
path: "scripts/mysql.sh",
args: args
args = []
config.vm.provision "booksservices shell script", type: "shell",
path: "scripts/booksservices.sh",
args: args
args = []
config.vm.provision "grafana shell script", type: "shell",
path: "scripts/grafana.sh",
args: args
end
end
end
As you can see in this file, I used shell scripts to install the software needed for my demo environment. Besides already being able to use the Kubernetes Dashboard (in a Web Browser) on my Windows laptop (via port forwarding), I also wanted to be able to the Grafana Dashboard on my Windows laptop. So again, I used the forwarded_port configuration option, to forward a port on my host (Windows) to a port on my guest (Ubuntu).
Navigate to localhost:3000 in your browser and you should see a Grafana login page.
[https://grafana.com/docs/grafana/latest/installation/kubernetes/?pg=oss-graf&plcmt=resources#send-manifest-to-kubernetes-api-server]
Vagrant forwarded ports allow you to access a port on your host machine and have all data forwarded to a port on the guest machine, over either TCP or UDP.
[https://www.vagrantup.com/docs/networking/forwarded_ports.html]
In the scripts directory I created a file grafana.sh with the following content:
#!/bin/bash echo "**** Begin installing Grafana" #Create Helm chart echo "**** Create Helm chart" cd /vagrant cd helmcharts rm -rf /vagrant/helmcharts/grafana-chart/* helm create grafana-chart rm -rf /vagrant/helmcharts/grafana-chart/templates/* cp /vagrant/yaml/*grafana.yaml /vagrant/helmcharts/grafana-chart/templates # Install Helm chart cd /vagrant cd helmcharts echo "**** Install Helm chart grafana-chart" helm install grafana-release ./grafana-chart # Wait 2,5 minute echo "**** Waiting 2,5 minute ..." sleep 150 #List helm releases echo "**** List helm releases" helm list -d echo "**** Forward local port 3000 to port 3000 on the grafana service" kubectl port-forward service/grafana 3000:3000 </dev/null &>/dev/null & echo "**** End installing Grafana"
As you can see, I used Helm (the package manager for Kubernetes), to deploy the Kubernetes resources, as I did and described before.
[https://technology.amis.nl/continuous-delivery/containers/using-helm-the-package-manager-for-kubernetes-to-install-two-versions-of-a-restful-web-service-spring-boot-application-within-minikube/]
From the subdirectory named env on my Windows laptop, I opened a Windows Command Prompt (cmd) and typed: vagrant up
This command creates and configures guest machines according to your Vagrantfile.
[https://www.vagrantup.com/docs/cli/up.html]
With the following output (only showing the part about Grafana):
ubuntu_minikube_helm_elastic: **** Install Helm chart grafana-chart ubuntu_minikube_helm_elastic: NAME: grafana-release ubuntu_minikube_helm_elastic: LAST DEPLOYED: Sat Mar 12 16:13:52 2022 ubuntu_minikube_helm_elastic: NAMESPACE: default ubuntu_minikube_helm_elastic: STATUS: deployed ubuntu_minikube_helm_elastic: REVISION: 1 ubuntu_minikube_helm_elastic: TEST SUITE: None ubuntu_minikube_helm_elastic: **** Waiting 2,5 minute ... ubuntu_minikube_helm_elastic: **** List helm releases ubuntu_minikube_helm_elastic: NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ubuntu_minikube_helm_elastic: namespace-release default 1 2022-03-12 15:57:45.147992882 +0000 UTC deployed namespace-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: elasticsearch-release default 1 2022-03-12 15:58:16.392846066 +0000 UTC deployed elasticsearch-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: kibana-release default 1 2022-03-12 16:00:47.91021763 +0000 UTC deployed kibana-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: logstash-release default 1 2022-03-12 16:03:20.271988972 +0000 UTC deployed logstash-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: filebeat-release default 1 2022-03-12 16:05:52.295700964 +0000 UTC deployed filebeat-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: mysql-release default 1 2022-03-12 16:07:00.323238623 +0000 UTC deployed mysql-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: booksservice-release default 1 2022-03-12 16:11:19.453269408 +0000 UTC deployed booksservice-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: grafana-release default 1 2022-03-12 16:13:52.662794581 +0000 UTC deployed grafana-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: **** Forward local port 3000 to port 3000 on the grafana service ubuntu_minikube_helm_elastic: **** End installing Grafana
Grafana Dashboard
On my Windows laptop, after my demo environment was set up, in a Web Browser I tried to start the Grafana Dashboard (via: localhost:3000), with the following response:
This site can’t be reached The webpage at http://localhost:3000/ might be temporarily down or it may have moved permanently to a new web address. ERR_SOCKET_NOT_CONNECTED
In order to investigate the problem, I wanted to use the Kubernetes Dashboard.
Kubernetes Dashboard
In the Web Browser on my Windows laptop, I started the Kubernetes Dashboard:
With the following output:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"kubernetes-dashboard\" not found",
"reason": "NotFound",
"details": {
"name": "kubernetes-dashboard",
"kind": "services"
},
"code": 404
}
I remembered, that in my demo environment in order to use the Kubernetes Web UI (Dashboard), since the last VM upgrade, I had to do some manual steps.
[https://technology.amis.nl/continuous-delivery/containers/trouble-shooting-while-upgrading-a-vm-with-minikube-elasticsearch-and-kibana-using-the-vm-drivernone-option-on-my-windows-laptop-using-vagrant-and-oracle-virtualbox/]
I used vagrant ssh to open a Linux Command Prompt where I used the following command:
kubectl get pods --all-namespaces
With the following output:
NAMESPACE NAME READY STATUS RESTARTS AGE default grafana-756fb84d84-w5rcq 1/1 Running 0 12m kube-system coredns-78fcd69978-qlhl4 1/1 Running 0 31m kube-system etcd-minikube 1/1 Running 0 31m kube-system kube-apiserver-minikube 1/1 Running 0 31m kube-system kube-controller-manager-minikube 1/1 Running 0 31m kube-system kube-proxy-bhvxv 1/1 Running 0 31m kube-system kube-scheduler-minikube 1/1 Running 0 31m kube-system storage-provisioner 1/1 Running 0 31m nl-amis-development booksservice-v1.0-7998bd45ff-6jttk 1/1 Running 0 15m nl-amis-development booksservice-v1.0-7998bd45ff-p7whh 1/1 Running 0 15m nl-amis-development booksservice-v2.0-6649c5d4d8-dprgr 1/1 Running 0 15m nl-amis-development booksservice-v2.0-6649c5d4d8-n8h24 1/1 Running 0 15m nl-amis-logging elasticsearch-bc5f6d549-wdwq6 1/1 Running 0 28m nl-amis-logging filebeat-daemonset-972s9 1/1 Running 0 20m nl-amis-logging kibana-f4665d556-vhsm2 1/1 Running 0 25m nl-amis-logging logstash-7d85759479-76bzk 1/1 Running 0 23m nl-amis-testing booksservice-v1.0-b7d5d44f4-djjsh 1/1 Running 0 15m nl-amis-testing booksservice-v1.0-b7d5d44f4-t699h 1/1 Running 0 15m nl-amis-testing mysql-787c788fd6-nzx86 1/1 Running 0 19m
As I expected the Kubernetes Dashboard was not yet deployed.
I used vagrant ssh to open a Linux Command Prompt where I used the following command:
minikube dashboard --url
With the following output:
Enabling dashboard ... ▪ Using image kubernetesui/dashboard:v2.3.1 ▪ Using image kubernetesui/metrics-scraper:v1.0.7 Verifying dashboard health ... Launching proxy ... Verifying proxy health ... http://127.0.0.1:40061/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
Then, to stop this command, I typed Ctrl-C to return to my Linux Command Prompt.
Remark about getting just the dashboard URL:
If you don’t want to open a web browser, the dashboard command can also simply emit a URL:
minikube dashboard --url
[https://minikube.sigs.k8s.io/docs/handbook/dashboard/]
Next, I used vagrant ssh to open a Linux Command Prompt where I used the following command:
kubectl get pods --all-namespaces
With the following output:
NAMESPACE NAME READY STATUS RESTARTS AGE default grafana-756fb84d84-w5rcq 1/1 Running 0 18m kube-system coredns-78fcd69978-qlhl4 1/1 Running 0 36m kube-system etcd-minikube 1/1 Running 0 37m kube-system kube-apiserver-minikube 1/1 Running 0 37m kube-system kube-controller-manager-minikube 1/1 Running 0 37m kube-system kube-proxy-bhvxv 1/1 Running 0 36m kube-system kube-scheduler-minikube 1/1 Running 0 37m kube-system storage-provisioner 1/1 Running 0 37m kubernetes-dashboard dashboard-metrics-scraper-5594458c94-fjssq 1/1 Running 0 3m8s kubernetes-dashboard kubernetes-dashboard-654cf69797-g49k2 1/1 Running 0 3m8s nl-amis-development booksservice-v1.0-7998bd45ff-6jttk 1/1 Running 0 20m nl-amis-development booksservice-v1.0-7998bd45ff-p7whh 1/1 Running 0 20m nl-amis-development booksservice-v2.0-6649c5d4d8-dprgr 1/1 Running 0 20m nl-amis-development booksservice-v2.0-6649c5d4d8-n8h24 1/1 Running 0 20m nl-amis-logging elasticsearch-bc5f6d549-wdwq6 1/1 Running 0 33m nl-amis-logging filebeat-daemonset-972s9 1/1 Running 0 26m nl-amis-logging kibana-f4665d556-vhsm2 1/1 Running 0 31m nl-amis-logging logstash-7d85759479-76bzk 1/1 Running 0 28m nl-amis-testing booksservice-v1.0-b7d5d44f4-djjsh 1/1 Running 0 20m nl-amis-testing booksservice-v1.0-b7d5d44f4-t699h 1/1 Running 0 20m nl-amis-testing mysql-787c788fd6-nzx86 1/1 Running 0 25m
I could see that the Pods for the dashboard were running.
minikube.sh
In order to avoid these manual steps, I changed the content of my minikube.sh script to:
[in bold, I highlighted the changes]
#!/bin/bash echo "**** Begin downloading minikube" #Kubernetes 1.22.3 requires conntrack to be installed in root's path sudo apt install -y conntrack #Download a static binary curl -o minikube https://storage.googleapis.com/minikube/releases/v1.24.0/minikube-linux-amd64 chmod +x minikube #Add the Minikube executable to your path sudo cp minikube /usr/local/bin/ rm minikube echo "**** End downloading minikube" echo "**** Begin starting a Cluster" sudo sysctl fs.protected_regular=0 #Start a Cluster minikube start \ --vm-driver=none \ --extra-config=kubeadm.node-name=minikube \ --extra-config=kubelet.hostname-override=minikube #To use kubectl or minikube commands as your own user, you may need to relocate them. sudo cp -R /root/.kube /root/.minikube /home/vagrant sudo chown -R vagrant /home/vagrant/.kube /home/vagrant/.minikube sed -i 's/root/home\/vagrant/g' /home/vagrant/.kube/config echo "**** End starting a Cluster" echo "**** Begin preparing dashboard" minikube dashboard --url </dev/null &>/dev/null & echo "**** End preparing dashboard" minikube kubectl -- get pods -A
Remark about the background process and redirecting the standard IO streams:
To run a command in the background, add the ampersand symbol (&) at the end of the command.
The shell job ID (surrounded with brackets) and process ID will be printed on the terminal.
Before a command is executed, its standard input (file descriptor 0), standard output (file descriptor 1) and standard error output (file descriptor 2) may be redirected using a special notation interpreted by the shell.
Redirection of input causes the file whose name results from the expansion of word to be opened for reading on file descriptor n, or the standard input (file descriptor 0) if n is not specified.
The following construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be redirected to the file whose name is the expansion of word.
[https://www.gnu.org/software/bash/manual/html_node/Redirections.html]
The null device /dev/null is a device file that discards all data written to it but reports that the write operation succeeded. The null device is typically used for disposing of unwanted output streams of a process, or as a convenient empty file for input streams. This is usually done by redirection.
[https://en.wikipedia.org/wiki/Null_device]
Remark:
In order to stop the running machine and destroy its resources, I used the following command on the Windows Command Prompt: vagrant destroy
I won’t mention this step for readability, but after each problem fix, I used it.
Again, on my Windows laptop, I used vagrant up and looked at the output and then started the Kubernetes Web UI (Dashboard):
I could see that the service grafana was pending (see orange dot).
By clicking on this service, I could also see that the Pod was running.
I used vagrant ssh to open a Linux Command Prompt where I used the following command:
kubectl get services
With the following output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.100.85.101 <pending> 3000:30066/TCP 7h23m kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7h42m
As it turned out, if you are using a custom Kubernetes Cluster (using minikube, kubeadm or the like), there is no LoadBalancer integrated with the default setup. LoadBalancer is a valid service type but it is being used in a non-compatible platform (at least by default). In order to use LoadBalancer you must have a platform that can provide external IPs to the pods.
If you are using minikube the run “minikube tunnel”. Now when you check your services you will get the public ip.
[https://stackoverflow.com/questions/44110876/kubernetes-service-external-ip-pending]
There are two major categories of services in Kubernetes:
- NodePort
- LoadBalancer
minikube supports either.
NodePort access
A NodePort service is the most basic way to get external traffic directly to your service. NodePort, as the name implies, opens a specific port, and any traffic that is sent to this port is forwarded to the service.
LoadBalancer access
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets its own IP address.
Using minikube tunnel
Services of type LoadBalancer can be exposed via the minikube tunnel command. It must be run in a separate terminal window to keep the LoadBalancer running. Ctrl-C in the terminal can be used to terminate the process at which time the network routes will be cleaned up.
[https://minikube.sigs.k8s.io/docs/handbook/accessing/]
In my demo environment I already used NodePort access for services. So, I wanted to try out the steps described for LoadBalancer access.
Again, I used vagrant ssh to open an extra Linux Command Prompt where I used the following command:
minikube tunnel
With the following output:
Status: machine: minikube pid: 57801 route: 10.96.0.0/12 -> 10.0.2.15 minikube: Running services: [grafana] errors: minikube: no errors router: no errors loadbalancer emulator: no errors
Then, to stop this command, I typed Ctrl-C to return to my Linux Command Prompt.
I wanted to use port forwarding. So, in VirtualBox, for my appliance I changed the Network settings.
For Adapter 1, by choosing “Advanced” and clicking on button “Port Forwarding”, I added a Port Forwarding Rule, this time for Host Port 3001, which I forwarded to Guest Port 3001.
Remark:
Because I already set everything up for port 3000 (see file grafana.sh) to be forwarded, I knew that port was already in use, so now I temporarily used port 3001.
In my other Linux Command Prompt, again I used the following command:
kubectl get services
With the following output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.100.85.101 10.100.85.101 3000:30066/TCP 7h31m kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7h50m
Then I used the following command:
socat tcp-listen:3001,fork tcp:10.100.85.101:3000 &
With the following output:
[1] 59758
Then, on my Windows laptop, I refreshed the Kubernetes Web UI (Dashboard):
I could see that the service grafana was running (see green dot).
By clicking on this service, I could also see that the Pod was running.
Grafana Dashboard
Next, on my Windows laptop, in a Web Browser I started the Grafana Dashboard (via: localhost:3000), with the expected result:
So, using the LoadBalancer access for service Grafana was now working.
However, for my demo environment, I wanted to keep things simple, so I decided to use NodePort access. I noticed that the namespace default was used for the Grafana resources and I wanted to change that also. And, lucky for me, version 8.4.4 was just released (March 16, 2022), so I could use that version.
[https://grafana.com/grafana/download?edition=oss&pg=oss-graf&plcmt=resources&platform=docker]
Adding another namespace
I did need another namespace, so I added to the yaml directory a file namespace-visualization.yaml with the following content:
apiVersion: v1 kind: Namespace metadata: name: "nl-amis-visualization" labels: name: "nl-amis-visualization"
Changing Kubernetes manifest files
In line with how a previously set up my environment, I changed the content of the following manifest files (with my own namespace and labels):
[in bold, I highlighted the changes]
- persistent-volume-claim-grafana.yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc namespace: nl-amis-visualization spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
- deployment-grafana.yaml
apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana namespace: nl-amis-visualization spec: selector: matchLabels: app: grafana version: "8.4.4" environment: visualization template: metadata: labels: app: grafana version: "8.4.4" environment: visualization spec: securityContext: fsGroup: 472 supplementalGroups: - 0 containers: - name: grafana image: grafana/grafana-oss:8.4.4 imagePullPolicy: IfNotPresent ports: - containerPort: 3000 readinessProbe: failureThreshold: 3 httpGet: path: /robots.txt port: 3000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 2 livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3000 timeoutSeconds: 1 resources: requests: cpu: 250m memory: 750Mi volumeMounts: - mountPath: /var/lib/grafana name: grafana-pv volumes: - name: grafana-pv persistentVolumeClaim: claimName: grafana-pvc
- service-grafana.yaml
apiVersion: v1 kind: Service metadata: name: grafana-service namespace: nl-amis-visualization labels: app: grafana version: "8.4.4" environment: visualization spec: ports: - nodePort: 30300 port: 3000 targetPort: 3000 selector: app: grafana version: "8.4.4" environment: visualization sessionAffinity: None type: NodePort
Again, on my Windows laptop, I used vagrant up and looked at the output and then started the
Grafana Dashboard (via: localhost:3000), with the following response:
This site can’t be reached The webpage at http://localhost:3000/ might be temporarily down or it may have moved permanently to a new web address. ERR_SOCKET_NOT_CONNECTED
It looked like the port forwarding went wrong, probably due to the namespace change.
Remember I used the following:
echo "**** Forward local port 3000 to port 3000 on the grafana service" kubectl port-forward service/grafana 3000:3000 </dev/null &>/dev/null &
But the service/grafana is now no longer in the namespace default and I also changed the name of the service to grafana-service.
Because the service now is of type NodePort, I am able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
[https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types]
In the scripts directory I therefor changed the content of file grafana.sh to:
[in bold, I highlighted the changes]
#!/bin/bash
echo "**** Begin installing Grafana"
#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/grafana-chart/*
helm create grafana-chart
rm -rf /vagrant/helmcharts/grafana-chart/templates/*
cp /vagrant/yaml/*grafana.yaml /vagrant/helmcharts/grafana-chart/templates
# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart grafana-chart"
helm install grafana-release ./grafana-chart
# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150
#List helm releases
echo "**** List helm releases"
helm list -d
#List pods
echo "**** List pods with namespace nl-amis-visualization"
kubectl get pods --namespace nl-amis-visualization
#List services
echo "**** List services with namespace nl-amis-visualization"
kubectl get service --namespace nl-amis-visualization
echo "**** Determine the IP of the minikube node"
nodeIP=$(kubectl get node minikube -o yaml | grep address: | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")
echo "---$nodeIP---"
echo "**** Via socat forward local port 3000 to port 30300 on the minikube node ($nodeIP)"
socat tcp-listen:3000,fork tcp:$nodeIP:30300 &
echo "**** End installing Grafana"
Again, on my Windows laptop, I used vagrant up.
With the following output:
[only showing the part about grafana]
ubuntu_minikube_helm_elastic: **** Begin installing Grafana ubuntu_minikube_helm_elastic: **** Create Helm chart ubuntu_minikube_helm_elastic: Creating grafana-chart ubuntu_minikube_helm_elastic: WARNING: File "/vagrant/helmcharts/grafana-chart/.helmignore" already exists. Overwriting. ubuntu_minikube_helm_elastic: **** Install Helm chart grafana-chart ubuntu_minikube_helm_elastic: NAME: grafana-release ubuntu_minikube_helm_elastic: LAST DEPLOYED: Mon Mar 21 09:16:33 2022 ubuntu_minikube_helm_elastic: NAMESPACE: default ubuntu_minikube_helm_elastic: STATUS: deployed ubuntu_minikube_helm_elastic: REVISION: 1 ubuntu_minikube_helm_elastic: TEST SUITE: None ubuntu_minikube_helm_elastic: **** Waiting 2,5 minute ... ubuntu_minikube_helm_elastic: **** List helm releases ubuntu_minikube_helm_elastic: NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ubuntu_minikube_helm_elastic: namespace-release default 1 2022-03-21 09:00:04.623690639 +0000 UTC deployed namespace-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: elasticsearch-release default 1 2022-03-21 09:00:35.937620941 +0000 UTC deployed elasticsearch-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: kibana-release default 1 2022-03-21 09:03:07.758253888 +0000 UTC deployed kibana-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: logstash-release default 1 2022-03-21 09:05:39.588711025 +0000 UTC deployed logstash-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: filebeat-release default 1 2022-03-21 09:08:13.175288429 +0000 UTC deployed filebeat-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: mysql-release default 1 2022-03-21 09:09:24.53031323 +0000 UTC deployed mysql-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: booksservice-release default 1 2022-03-21 09:13:59.647836159 +0000 UTC deployed booksservice-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: grafana-release default 1 2022-03-21 09:16:33.88304826 +0000 UTC deployed grafana-chart-0.1.0 1.16.0 ubuntu_minikube_helm_elastic: **** List pods with namespace nl-amis-visualization ubuntu_minikube_helm_elastic: NAME READY STATUS RESTARTS AGE ubuntu_minikube_helm_elastic: grafana-d4d47f944-thbx8 1/1 Running 0 2m30s ubuntu_minikube_helm_elastic: **** List services with namespace nl-amis-visualization ubuntu_minikube_helm_elastic: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ubuntu_minikube_helm_elastic: grafana-service NodePort 10.111.132.102 <none> 3000:30300/TCP 2m30s ubuntu_minikube_helm_elastic: **** Determine the IP of the minikube node ubuntu_minikube_helm_elastic: ---10.0.2.15--- ubuntu_minikube_helm_elastic: **** Via socat forward local port 3000 to port 30300 on the minikube node (10.0.2.15) ubuntu_minikube_helm_elastic: **** End installing Grafana
Grafana Dashboard log in
On my Windows laptop, in a Web Browser I started the Grafana Dashboard (via: localhost:3000), with the expected result:
To log in to Grafana for the first time:
- Open your web browser and go to http://localhost:3000/. The default HTTP port that Grafana listens to is 3000 unless you have configured a different port.
- On the login page, enter admin for username and password.
- Click Log in. If login is successful, then you will see a prompt to change the password.
- Click OK on the prompt, then change your password.
[https://grafana.com/docs/grafana/latest/getting-started/getting-started/#step-2-log-in]
And there was the Welcome screen:
Data sources
Grafana supports many different storage backends for your time series data (data source). Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.
[https://grafana.com/docs/grafana/latest/datasources/#data-sources]
One of the supported data sources is Elasticsearch and I opted for that one.
Grafana ships with advanced support for Elasticsearch. You can do many types of simple or complex Elasticsearch queries to visualize logs or metrics stored in Elasticsearch. You can also annotate your graphs with log events stored in Elasticsearch.
I followed the instructions from “Using Elasticsearch in Grafana” to set it up.
[https://grafana.com/docs/grafana/latest/datasources/elasticsearch/]
Adding the Elasticsearch data source
In the left side menu, I clicked on Configuration | Data Sources.
Next, I clicked on the button “Add data source” and search (via the Filter field) for Elasticsearch.
I selected the Elasticsearch data source.
I kept the default settings, but did change/fill in:
- Name: Elasticsearch_logstash-via-filebeat
- HTTP, URL: http:localhost:9200
The HTTP protocol, IP, and port of your Elasticsearch server. - Elasticsearch details, Index name: [logstash-via-filebeat-]YYYY.MM.DD
- Elasticsearch details, Pattern: Daily
- Elasticsearch details, Version: 8.0+
Remark about the Elasticsearch details, Index name:
In my demo environment the Logstash configuration was set up to create an Elasticsearch index with the format:
logstash-via-filebeat-%{+YYYY.MM.dd}
For example: logstash-via-filebeat-2022.03.21
[https://technology.amis.nl/continuous-delivery/containers/using-elastic-stack-filebeat-and-logstash-for-log-aggregation/]
Then, I clicked on button “Save & Test”, with the following output:
Elasticsearch error: Bad Gateway
So, I changed the value for HTTP, URL to:
http://elasticsearch-service.nl-amis-logging:9200
Remark about service name lookup:
A cluster-aware DNS server, such as CoreDNS, watches the Kubernetes API for new Services and creates a set of DNS records for each one. If DNS has been enabled throughout your cluster then all Pods should automatically be able to resolve Services by their DNS name.
For example, if you have a Service called my-service in a Kubernetes namespace my-ns, the control plane and the DNS Service acting together create a DNS record for my-service.my-ns. Pods in the my-ns namespace should be able to find the service by doing a name lookup for my-service (my-service.my-ns would also work).
Pods in other namespaces must qualify the name as my-service.my-ns. These names will resolve to the cluster IP assigned for the Service.
[https://kubernetes.io/docs/concepts/services-networking/service/#dns]
Then again, I clicked on button “Save & Test”, with the following output:
Index OK. Time field name OK.
Postman
In order for the Elasticsearch index to have some data, as I described in a previous article, I used Postman to add books to and retrieve books from the book catalog. I did this for version 1.0 and 2.0 of the BooksService application.
[https://technology.amis.nl/2019/09/15/using-elastic-stack-filebeat-for-log-aggregation/]
Grafana Explore
Grafana’s dashboard UI is all about building dashboards for visualization. Explore strips away the dashboard and panel options so that you can focus on the query. It helps you iterate until you have a working query and then think about building a dashboard.
[https://grafana.com/docs/grafana/latest/explore/]
From the menu on the left, I choose Explore:
Here, I made the following changes:
Group By: Terms
Select Field: kubernetes.labels.app.keyword
With the following result (showing the data in a Table):
The Elasticsearch query editor allows you to select multiple metrics and group by multiple terms or filters. Use the plus and minus icons to the right to add/remove metrics or group by clauses. Some metrics and group by clauses have options, click the option text to expand the row to view and edit metric or group by options.
[https://grafana.com/docs/grafana/latest/datasources/elasticsearch/]
Remark about the result:
In my demo environment, I setup Logstash in such a way, that only the logfile data from the containers for MySQL (mysql*.log) and my booksservices (booksservices*.log) were used in Elasticsearch.
[https://technology.amis.nl/continuous-delivery/containers/using-elastic-stack-filebeat-for-log-aggregation/]
This is an example Kubernetes manifest file (service-booksservice-dev-v1.0.yaml), I used in my demo environment:
kind: Service apiVersion: v1 metadata: name: booksservice-v1-0-service namespace: nl-amis-development labels: app: booksservice version: "1.0" environment: development spec: selector: app: booksservice version: "1.0" environment: development type: NodePort ports: - protocol: TCP nodePort: 30010 port: 9190 targetPort: 9090
This relates to the Term: kubernetes.labels.app.keyword
So, now I knew I had a working query that gave me some results. I now could use this query in a dashboard.
Grafana Dashboards
A dashboard is a set of one or more panels organized and arranged into one or more rows. Grafana ships with a variety of Panels. Grafana makes it easy to construct the right queries, and customize the display properties so that you can create the perfect dashboard for your need. Each panel can interact with data from any configured Grafana Data Source (currently Graphite, Prometheus, Elasticsearch, InfluxDB, OpenTSDB, MySQL, PostgreSQL, Microsoft SQL Server and AWS Cloudwatch).
[https://grafana.com/docs/grafana/latest/dashboards/]
From the menu on the left, I choose Dashboards | Browse:
Then I clicked on button “New Dashboard”:
Next, I clicked on “Add a new panel”:
Here, I made the following changes:
Group By: Terms
Select Field: kubernetes.labels.app.keyword
With the following result (showing no data):
On the top right, I opened the Visualization combo box:
I choose “Pie chart” as visualization:
In “Value options | Show”, I clicked on button “All values”:
In “Panel options | Title”, I changed the title to “Apps” and in “Pie chart | Labels”, I selected “Value”:
Then, I clicked on button “Save” and save the dashboard as “Dashboard1” :
With the following result:
Adding another panel to the dashboard
On the top right , I clicked on icon “Add Panel”, followed by “Add a new panel”:
In a similar way as before, I made the following changes:
Group By: Terms
Select Field: kubernetes.labels.environment.keyword
I choose “Gauge” as visualization, in “Value options | Show”, I clicked on button “All values” and I changed the title to “Environment”.
Next, I clicked on button “Save”. In the “Save dashboard” pop-up I added a note and clicked “Save” again.
With the following result:
As I expected, the label “environment” in my demo environment has the value development or testing. See, for example the Kubernetes manifest file (service-booksservice-dev-v1.0.yaml), I described in this article.
With this simple dashboard (with two panels with different visualizations), for me it was obvious enough that my demo environment with Grafana in combination with Elasticsearch (as the data source), was working. It now gives me to possibility to explore the functionality of Grafana even further (and eventually also in combination with other data sources).
You can find the code, belonging to my article, here:
https://github.com/marclameriks/amis-technology-blog-2022-03-1
