For training and demo purposes, on my windows laptop, I needed an environment with a guest Operating System, Docker and Minikube available within an Oracle VirtualBox appliance.
So, in a series of articles up till now, I described the following:
- The steps I took, to get Docker and Minikube (using the –vm-driver=none option) installed on top of an Ubuntu guest Operating System within an Oracle VirtualBox appliance, with the help of Vagrant.
[https://technology.amis.nl/2019/02/12/rapidly-spinning-up-a-vm-with-ubuntu-docker-and-minikube-using-the-vm-drivernone-option-on-my-windows-laptop-using-vagrant-and-oracle-virtualbox/] - How I created two versions of a RESTful Web Service Spring Boot application that, later on (in another article), I would be running in Minikube.
The Spring Boot application I created in the article is a book service. With this service you can add, update, delete and retrieve books from a catalog.
The application uses an H2 in-memory database but is also prepared for using an external MySQL database. For demo purposes I created a 1.0 and 2.0 version of the application.
The latter has some additional fields representing a book.
[https://technology.amis.nl/2019/02/26/building-a-restful-web-service-with-spring-boot-using-an-h2-in-memory-database-and-also-an-external-mysql-database/] - How these applications were used in Minikube, together with an external “Dockerized” MySQL database.
[https://technology.amis.nl/2019/03/05/using-a-restful-web-service-spring-boot-application-in-minikube-together-with-an-external-dockerized-mysql-database/] - How I used Helm, the package manager for Kubernetes, to install two versions of a RESTful Web Service Spring Boot application, together with an external MySQL database, within Minikube.
[https://technology.amis.nl/2019/03/12/using-helm-the-package-manager-for-kubernetes-to-install-two-versions-of-a-restful-web-service-spring-boot-application-within-minikube/]
In this article I will describe how I installed ElasticSearch, Fluentd and Kibana (EFK) in order to do log aggregation, later on (described in another article).
Also, I wanted it to be possible to set up my demo environment from scratch, including some data (books in the catalog), without the manual steps I had to do up till now.
For example, the manual steps for:
- installing Helm, Tiller and socat.
- creating Docker images.
- setting up a MySQL database named test.
- installing several Helm charts.
Besides already being able to use the Kubernetes Dashboard (in a Web Browser) on my Windows laptop (via port forwarding), I also wanted to be able to use Postman (for sending requests) and the Kibana Dashboard on my Windows laptop, whilst all the software is running in the VirtualBox appliance (created with Vagrant).
So, in this article I will also describe how I used Vagrant and shell scripts to further automate setting up my demo environment from scratch, including ElasticSearch, Fluentd and Kibana.
Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.
[https://www.elastic.co/products/elasticsearch]
Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time.
[https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html]
Fluentd
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
[https://www.fluentd.org/]
Fluentd Events
Fluentd’s input sources (where all the data comes from) are enabled by selecting and configuring the desired input plugins using source directives
The source submits events into the Fluentd’s routing engine. An event consists of three entities: tag, time and record.
- The tag is a string separated by ‘.’s (e.g. myapp.access), and is used as the directions for Fluentd’s internal routing engine. → Where an event comes from
- The time field is specified by input plugins, and it must be in the Unix time format. → When an event happens.
- The record is a JSON object. → Actual log content.
The input plugin has the responsibility for generating a Fluentd event from data sources.
Here is a brief overview of the life of a Fluentd event:
[https://docs.fluentd.org/v1.0/articles/config-file#introduction:-the-life-of-a-fluentd-event]
A configuration file allows the user to control the input and output behavior of Fluentd by (1) selecting input and output plugins and (2) specifying the plugin parameters. The file is required for Fluentd to operate properly.
If you’re using the Docker container, the default location is located at /fluentd/etc/fluent.conf.
[https://docs.fluentd.org/v1.0/articles/config-file#config-file-location]
Kibana
Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack.
[https://www.elastic.co/products/kibana]
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.
Kibana makes it easy to understand large volumes of data. It’s simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.
[https://www.elastic.co/guide/en/kibana/current/introduction.html]
Adding logging to the BooksService application
In order to be able to, later on, demonstrate log aggregation via ElasticSearch, Fluentd and Kibana (EFK), I added some extra logging to BookService.java.
package nl.amis.demo.services.books_service.application;
import nl.amis.demo.services.books_service.BooksServiceApplication;
import nl.amis.demo.services.books_service.domain.Book;
import nl.amis.demo.services.books_service.infrastructure.persistence.BookRepository;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
@Service
public class BookService {
private static final Logger logger = LoggerFactory.getLogger(BookService.class);
@Autowired
private BookRepository bookRepository;
public List getAllBooks() {
logger.info("\n----Begin logging BookService.getAllBooks----");
List books = new ArrayList<>();
bookRepository.findAll().forEach(books::add);
logger.info("\n---- " + books.size() + " books found ----");
logger.info("\n----End logging BookService.getAllBooks----");
return books;
}
public Book getBook(String id) {
return bookRepository.findById(id).orElseGet(Book::new);
}
public void addBook(Book whiskey) {
bookRepository.save(whiskey);
}
public void updateBook(String id, Book whiskey) {
bookRepository.save(whiskey);
}
public void deleteBook(String id) {
bookRepository.deleteById(id);
}
}
So, each time the getAllBooks method is called, this becomes visible in the log file.
Changing the testing profile
As you may remember, in the testing profile of the BooksService application, the connection to the external MySQL database was configured.
By using the spring.jpa.hibernate.ddl-auto property and setting it to value create, database dropping will be generated followed by database creation (by Hybernate).
[https://docs.spring.io/spring-boot/docs/current/reference/html/howto-database-initialization.html]
[https://docs.jboss.org/hibernate/orm/5.2/userguide/html_single/Hibernate_User_Guide.html#domain-model, see paragraph “Automatic schema generation”]
Of course, this automatic schema generation is very convenient in a development environment. In my case the test database and the book table are automatically created.
Unfortunately, this has as side effect that after stopping the container that runs the BooksService application and starting a new one, the previously inserted data (books in the catalog) is gone.
In order to confirm what actually happens, I used a mysql-client.
kubectl --namespace=nl-amis-testing run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql-service.nl-amis-testing -ppassword If you don't see a command prompt, try pressing enter.
I had to click on the Enter button.
mysql> show databases;
With the following output:
mysql> select create_time from information_schema.tables where table_schema = 'test';
With the following output:
mysql> select create_time from information_schema.tables where table_schema = 'test' and table_name = 'book';
With the following output:
mysql> select * from book;
With the following output:
Next, via the Minikube Dashboard I deleted one of the booksservice-v1.0 pods in the nl-amis-testing namespace. The ReplicaSet ensures that 2 pod replicas are running at any given time. So, a new pod is automatically created and the BooksService application started.
After that I repeated the SQL statements via mysql-client.
mysql> select create_time from information_schema.tables where table_schema = 'test';
With the following output:
mysql> select create_time from information_schema.tables where table_schema = 'test' and table_name = 'book';
With the following output:
mysql> select * from book;
With the following output:
This confirms that when the BooksService application is started (in a newly created pod), the test database and the book table are automatically created and therefor the data is gone. To avoid this behavior, I chose to use the value none for the spring.jpa.hibernate.ddl-auto property.
Because later on, I had to create the book table myself, I wanted to know the metadata.
mysql> desc book;
With the following output:
mysql> show create table book;
With the following output:
mysql> exit Bye pod "mysql-client" deleted
I changed the content of application-testing.properties to:
logging.level.root=INFO server.port=9091 nl.amis.environment=testing spring.datasource.url=jdbc:mysql://localhost:3306/test?allowPublicKeyRetrieval=true&useSSL=false spring.datasource.username=root spring.datasource.password=password spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect spring.jpa.hibernate.ddl-auto=none
With the changed logging and testing profile in place, within IntelliJ IDEA, I created a new Executable jar for each of the two versions of the BooksService application.
When the new jar was used in Minikube and a rest call was made to get all books, this was part of the log from the booksservice-v1-0-container:
So now, because of the changed testing profile, the book table isn’t automatically generated anymore, hence the SQL error message “Table ‘test.book’ doesn’t exist”.
In the scripts directory I created a file mysql.sh (see later on in this article) were I created the book table myself.
Vagrantfile
To further automate setting up my demo environment, I changed the content of Vagrantfile to:
[in bold, I highlighted the changes]
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.define "ubuntu_minikube_helm_efk" do |ubuntu_minikube_helm_efk|
config.vm.network "forwarded_port",
guest: 8001,
host: 8001,
auto_correct: true
config.vm.network "forwarded_port",
guest: 5601,
host: 5601,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9200,
host: 9200,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9010,
host: 9010,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9020,
host: 9020,
auto_correct: true
config.vm.network "forwarded_port",
guest: 9110,
host: 9110,
auto_correct: true
config.vm.provider "virtualbox" do |vb|
vb.name = "Ubuntu Minikube Helm EFK"
vb.memory = "8192"
vb.cpus = "1"
args = []
config.vm.provision "shell",
path: "scripts/docker.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/minikube.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/kubectl.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/helm.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/namespaces.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/elasticsearch.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/kibana.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/fluentd.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/mysql.sh",
args: args
args = []
config.vm.provision "shell",
path: "scripts/booksservices.sh",
args: args
end
end
end
As you can see in this file I used extra shell scripts to install the software needed for my demo environment. Besides already being able to use the Kubernetes Dashboard (in a Web Browser) on my Windows laptop (via port forwarding), I also wanted to be able to use Postman (for sending requests) and the Kibana Dashboard on my Windows laptop. So again, I used the forwarded_port configuration option, to forward a port on my host (Windows) to a port on my guest (Ubuntu).
Vagrant forwarded ports allow you to access a port on your host machine and have all data forwarded to a port on the guest machine, over either TCP or UDP.
[https://www.vagrantup.com/docs/networking/forwarded_ports.html]
From the subdirectory named env on my Windows laptop, I opened a Windows Command Prompt (cmd) and typed: vagrant up
This command creates and configures guest machines according to your Vagrantfile.
[https://www.vagrantup.com/docs/cli/up.html]
In the table below, you can see an overview of the Pods in the demo environment, this Vagrantfile will create:
| Spring Boot application | Pod | Namespace | Service ports | |||
| Environment | Database | nodePort | port | targetPort | ||
| DEV | H2 in memory | booksservice-v1.0-* | nl-amis-development | 30010 | 9190 | 9090 |
| booksservice-v2.0-* | nl-amis-development | 30020 | 9190 | 9090 | ||
| TST | MySQL | booksservice-v1.0-* | nl-amis-testing | 30110 | 9191 | 9091 |
| – | – | mysql-* | nl-amis-testing | – | 3306 | – |
| – | – | elasticsearch-* | nl-amis-logging | 30200 | 9200 | 9200 |
| – | – | fluentd-* | nl-amis-logging | – | – | – |
| – | – | kibana-* | nl-amis-logging | 30601 | 5601 | 5601 |
Now I am going to describe the content of the extra shell scripts. They are mainly used to automate the manual steps, I already described in my previous articles.
You will find that I used the sleep command, in order to wait for a Pod to reach the “Running” status. Again, I used Helm, the package manager for Kubernetes, to install Kubernetes Objects.
[https://technology.amis.nl/2019/03/12/using-helm-the-package-manager-for-kubernetes-to-install-two-versions-of-a-restful-web-service-spring-boot-application-within-minikube/]
Besides using kubctl proxy (for exposing the Kubernetes Dashboard), I used socat to forward a local port to a port on the minikube node.
Remark:
Using kubectl port-forward for this, wasn’t an option because this only works on Pods.
[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward]
Installing helm
In the scripts directory I created a file helm.sh with the following content:
#!/bin/bash echo "**** Begin installing Helm" #Install socat sudo apt-get install socat #Install Helm client-side sudo snap install helm --classic #Install Tiller (Helm server-side) helm init # Wait 2,5 minute echo "**** Waiting 2,5 minute ..." sleep 150 #List pods echo "**** List pods with namespace kube-system" kubectl get pods --namespace kube-system echo "**** End installing Helm"
Installing namespaces
In my previous article I already described the use of a namespace-chart helm chart.
[https://technology.amis.nl/2019/03/12/using-helm-the-package-manager-for-kubernetes-to-install-two-versions-of-a-restful-web-service-spring-boot-application-within-minikube/]
I did need another namespace, so I added to the yaml directory a file namespace-logging.yaml with the following content:
apiVersion: v1 kind: Namespace metadata: name: "nl-amis-logging" labels: name: "nl-amis-logging"
In the scripts directory I created a file namespaces.sh with the following content:
#!/bin/bash echo "**** Begin installing namespaces" #Create Helm chart echo "**** Create Helm chart" cd /vagrant cd helmcharts rm -rf /vagrant/helmcharts/namespace-chart/* helm create namespace-chart rm -rf /vagrant/helmcharts/namespace-chart/templates/* cp /vagrant/yaml/namespace*.yaml /vagrant/helmcharts/namespace-chart/templates # Install Helm chart cd /vagrant cd helmcharts echo "**** Install Helm chart namespace-chart" helm install ./namespace-chart --name namespace-release # Wait 30 seconds echo "**** Waiting 30 seconds ..." sleep 30 #List helm releases echo "**** List helm releases" helm list -d #List namespaces echo "**** List namespaces" kubectl get namespaces echo "**** End installing namespaces"
Installing elasticsearch
I added to the yaml directory a file deployment-elasticsearch.yaml with the following content:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: elasticsearch
namespace: nl-amis-logging
labels:
app: elasticsearch
version: "7.0.0"
environment: logging
spec:
selector:
matchLabels:
app: elasticsearch
version: "7.0.0"
environment: logging
template:
metadata:
labels:
app: elasticsearch
version: "7.0.0"
environment: logging
spec:
containers:
- name: elasticsearch-container
image: docker.elastic.co/elasticsearch/elasticsearch:7.0.0
env:
- name: discovery.type
value: single-node
ports:
- containerPort: 9200
volumeMounts:
- name: storage
mountPath: /data
volumes:
- name: storage
emptyDir: {}
Remark:
This deployment is based on the following command line:
docker run -p 9200:9200 -p 9300:9300 -e “discovery.type=single-node” docker.elastic.co/elasticsearch/elasticsearch:7.0.0
The replicas are set to 1, so the ReplicaSet ensures that 1 pod replica is running at any given time.
The ReplicaSet manages all the pods with labels that match the selector. In my case these labels are:
| Label key | Label value |
| app | elasticsearch |
| version | 7.0.0 |
| environment | logging |
I added to the yaml directory a file service-elasticsearch.yaml with the following content:
apiVersion: v1 kind: Service metadata: name: elasticsearch-service namespace: nl-amis-logging labels: app: elasticsearch version: "7.0.0" environment: logging spec: type: NodePort selector: app: elasticsearch version: "7.0.0" environment: logging ports: - nodePort: 30200 port: 9200 targetPort: 9200
In the scripts directory I created a file elasticsearch.sh with the following content:
#!/bin/bash
echo "**** Begin installing Elasticsearch"
#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/elasticsearch-chart/*
helm create elasticsearch-chart
rm -rf /vagrant/helmcharts/elasticsearch-chart/templates/*
cp /vagrant/yaml/*elasticsearch.yaml /vagrant/helmcharts/elasticsearch-chart/templates
# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart elasticsearch-chart"
helm install ./elasticsearch-chart --name elasticsearch-release
# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150
#List helm releases
echo "**** List helm releases"
helm list -d
#List pods
echo "**** List pods with namespace nl-amis-logging"
kubectl get pods --namespace nl-amis-logging
#List services
echo "**** List services with namespace nl-amis-logging"
kubectl get service --namespace nl-amis-logging
echo "**** Determine the IP of the minikube node"
nodeIP=$(kubectl get node minikube -o yaml | grep address: | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")
echo "---$nodeIP---"
echo "**** Via socat forward local port 9200 to port 30200 on the minikube node ($nodeIP)"
socat tcp-listen:9200,fork tcp:$nodeIP:30200 &
echo "**** Send a request to Elasticsearch"
curl -XGET http://localhost:9200/_count?pretty
echo "**** End installing Elasticsearch"
Remark:
On my Windows laptop, after my demo environment is set up, in a Web Browser I can use: http://localhost:9200/_count?pretty
With for example the following result:
Installing kibana
I added to the yaml directory a file deployment-kibana.yaml with the following content:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kibana namespace: nl-amis-logging spec: selector: matchLabels: app: kibana version: "7.0.0" environment: logging template: metadata: labels: app: kibana version: "7.0.0" environment: logging spec: containers: - name: kibana-container image: docker.elastic.co/kibana/kibana:7.0.0 env: - name: ELASTICSEARCH_HOSTS value: http://elasticsearch-service.nl-amis-logging:9200 ports: - containerPort: 5601
Remark:
This deployment is based on the following command line:
docker pull docker.elastic.co/kibana/kibana:7.0.0
[https://www.elastic.co/guide/en/kibana/current/docker.html#pull-image]
The default settings configure Kibana to run on localhost:5601.
[https://www.elastic.co/guide/en/kibana/7.0/settings.html#settings]
Under Docker, Kibana can be configured via environment variables.
[https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config]
The following settings have different default values when using the Docker images:
| Environment Variable | Kibana Setting | Default value |
| SERVER_NAME | server.name | kibana |
| SERVER_HOST | server.host | “0” |
| ELASTICSEARCH_HOSTS | elasticsearch.hosts | http://elasticsearch:9200 |
| XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED | xpack.monitoring.ui.container.elasticsearch.enabled | true |
These settings are defined in the default kibana.yml. They can be overridden with a custom kibana.yml or via environment variables.
[https://www.elastic.co/guide/en/kibana/current/docker.html#docker-defaults]
I changed the value of the environment variable ELASTICSEARCH_HOSTS to: http://elasticsearch-service.nl-amis-logging:9200
The replicas are set to 1, so the ReplicaSet ensures that 1 pod replica is running at any given time.
The ReplicaSet manages all the pods with labels that match the selector. In my case these labels are:
| Label key | Label value |
| app | kibana |
| version | 7.0.0 |
| environment | logging |
I added to the yaml directory a file service-kibana.yaml with the following content:
apiVersion: v1 kind: Service metadata: name: kibana-service namespace: nl-amis-logging labels: app: kibana version: "7.0.0" environment: logging spec: type: NodePort selector: app: kibana version: "7.0.0" environment: logging ports: - nodePort: 30601 port: 5601 targetPort: 5601
In the scripts directory I created a file kibana.sh with the following content:
#!/bin/bash
echo "**** Begin installing Kibana"
#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/kibana-chart/*
helm create kibana-chart
rm -rf /vagrant/helmcharts/kibana-chart/templates/*
cp /vagrant/yaml/*kibana.yaml /vagrant/helmcharts/kibana-chart/templates
# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart kibana-chart"
helm install ./kibana-chart --name kibana-release
# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150
#List helm releases
echo "**** List helm releases"
helm list -d
#List pods
echo "**** List pods with namespace nl-amis-logging"
kubectl get pods --namespace nl-amis-logging
#List services
echo "**** List services with namespace nl-amis-logging"
kubectl get service --namespace nl-amis-logging
echo "**** Determine the IP of the minikube node"
nodeIP=$(kubectl get node minikube -o yaml | grep address: | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")
echo "---$nodeIP---"
echo "**** Via socat forward local port 5601 to port 30601 on the minikube node ($nodeIP)"
socat tcp-listen:5601,fork tcp:$nodeIP:30601 &
echo "**** End installing Kibana"
Remark:
On my Windows laptop, after my demo environment is set up, in a Web Browser I can start the Kibana Dashboard via: http://localhost:5601/app/kibana
Installing fluentd
I had a look at “Kubernetes Logging with Fluentd”, which has a reference to the following Git repository:
[https://docs.fluentd.org/v0.12/articles/kubernetes-fluentd]
[https://github.com/fluent/fluentd-kubernetes-daemonset]
From this “Fluentd daemonset for Kubernetes and it’s Docker image” Git repository, I extracted the file fluentd-daemonset-elasticsearch.yaml with the following content:
[https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml]
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:elasticsearch env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-logging" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" # Option to configure elasticsearch plugin with self signed certs # ================================================================ - name: FLUENT_ELASTICSEARCH_SSL_VERIFY value: "true" # X-Pack Authentication # ===================== - name: FLUENT_ELASTICSEARCH_USER value: "elastic" - name: FLUENT_ELASTICSEARCH_PASSWORD value: "changeme" # Logz.io Authentication # ====================== - name: LOGZIO_TOKEN value: "ThisIsASuperLongToken" - name: LOGZIO_LOGTYPE value: "kubernetes" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
From this “Fluentd daemonset for Kubernetes and it’s Docker image” Git repository, I also extracted the file fluentd-daemonset-elasticsearch-rbac.yaml with the following content:
[https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml]
[In bold, I highlighted the differences with file fluentd-daemonset-elasticsearch.yaml]
--- apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluentd namespace: kube-system rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: kube-system --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:elasticsearch env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-logging" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" # X-Pack Authentication # ===================== - name: FLUENT_ELASTICSEARCH_USER value: "elastic" - name: FLUENT_ELASTICSEARCH_PASSWORD value: "changeme" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
This file uses Role-based access control (RBAC). See for more information, for example:
https://kubernetes.io/docs/reference/access-authn-authz/authorization/
Based on the extracted file fluentd-daemonset-elasticsearch-rbac.yaml I created my own yaml files for installing fluentd.
I used as namespace nl-amis-logging.
I added to the yaml directory a file serviceaccount-fluentd.yaml with the following content:
apiVersion: v1 kind: ServiceAccount metadata: name: fluentd-serviceaccount namespace: nl-amis-logging
I added to the yaml directory a file clusterrole-fluentd.yaml with the following content:
apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluentd-clusterrole namespace: nl-amis-logging rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch
I added to the yaml directory a file clusterrolebinding-fluentd.yaml with the following content:
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: fluentd-clusterrolebinding namespace: nl-amis-logging roleRef: kind: ClusterRole name: fluentd-clusterrole apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd-serviceaccount namespace: nl-amis-logging
I added to the yaml directory a file daemonset-fluentd.yaml with the following content:
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: nl-amis-logging labels: app: fluentd version: "1.0" environment: logging spec: template: metadata: labels: app: fluentd version: "1.0" environment: logging spec: serviceAccount: fluentd-serviceaccount serviceAccountName: fluentd-serviceaccount tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-container image: fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-elasticsearch-1.7 env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-service.nl-amis-logging" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENTD_SYSTEMD_CONF value: "disable" - name: FLUENTD_CONF value: "custom-config/fluentd.conf" # X-Pack Authentication # ===================== - name: FLUENT_ELASTICSEARCH_USER value: "elastic" - name: FLUENT_ELASTICSEARCH_PASSWORD value: "changeme" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluentd-config-volume mountPath: /fluentd/etc/custom-config terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluentd-config-volume configMap: name: fluentd-configmap
A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.
A typical use of a DaemonSet is running a logs collection daemon on every node, such as fluentd or logstash.
[https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/]
Remark about the image:
As mentioned earlier, from the Git repository, I extracted the file fluentd-daemonset-elasticsearch-rbac.yaml which uses the following image:
fluent/fluentd-kubernetes-daemonset:elasticsearch
After I created the fluentd DaemonSet, the log file from the fluentd-container Container gave the following output:
…
[info]: reading config file path=”/fluentd/etc/fluent.conf”
[info]: starting fluentd-0.12.43
…
So, the version is: 0.12.43
In the Git repository README.md, the section about “Image versions” mentions:
The following repository expose images based on Alpine Linux and Debian. For production environments we strongly suggest to use Debian images.
Fluentd versioning is as follows:
| Series | Description |
| v1.x | stable |
| v0.12 | Old stable, no longer updated |
[https://github.com/fluent/fluentd-kubernetes-daemonset]
Of course, I wanted to use a more recent version. In the Git repository README.md, the section about “Supported tags and respective Dockerfile links”, Image versions mentions the tags:
v1.3.3-debian-elasticsearch-1.7,v1.3-debian-elasticsearch-1
So, I chose the image: fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-elasticsearch-1.7
Remark about the env FLUENT_ELASTICSEARCH_*:
Because Fluentd has to be able to send events to Elasticsearch, it has to know it’s url, in the format: <SCHEME>://<HOST><PORT>.
Remark about volumes and volumeMounts:
To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).
A process in a container sees a filesystem view composed from their Docker image and volumes. The Docker image is at the root of the filesystem hierarchy, and any volumes are mounted at the specified paths within the image.
A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod.
[https://kubernetes.io/docs/concepts/storage/volumes/#hostpath]
Remark about systemd:
If you don’t setup systemd in the container, fluentd shows following messages by default configuration.
[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s
You can suppress these messages by setting disable to FLUENTD_SYSTEMD_CONF environment variable in your kubernetes configuration.
[https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/README.md]
Remark about the fluentd config file:
If you’re using the Docker container, the default location is located at /fluentd/etc/fluent.conf.
You can change default configuration file location via FLUENT_CONF.
[https://docs.fluentd.org/v1.0/articles/config-file#config-file-location]
As you can see, I changed the default configuration file location (via FLUENT_CONF) to: /fluentd/etc/custom-config/fluentd.conf
There for, in the vagrant directory I created a subdirectory structure configmaps/configmap-fluentd with a file fluentd.conf with the following content:
<source>
@type tail
@id in_tail_booksservice_logs
path "/var/log/containers/booksservice*.log"
pos_file "/var/log/ fluentd-booksservice.log.pos"
tag "kubernetes.*"
read_from_head true
<parse>
@type "json"
time_format "%Y-%m-%dT%H:%M:%S.%NZ"
time_type string
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<filter kubernetes.**>
@type parser
key_name log
reserve_data true
<parse>
@type multiline
format_firstline /\d{4}-\d{2}-\d{2}/
format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3})\s+(?<level>[^\s]+)\s+(\[(?<service>[^,]*),(?<trace>[^,]*),(?<span>[^,]*),(?<exportable>[^\]]*)\]\s+)?(?<pid>\d+)\s+---\s+\[(?<thread>[^\]]+)\]\s+(?<source>[^\s]+)\s*:\s+(?<message>.*)/
time_format %Y-%m-%d %H:%M:%S.%N
</parse>
</filter>
<match **>
@type elasticsearch
@id out_es
@log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"
type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
<buffer>
flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true
</buffer>
</match>
Remark:
As a base for this file, I used the information mentioned in the article from my colleague from First8 (a sister company within the Conclusion ecosystem).
[https://technology.first8.nl/kubernetes-logging/]
I used parts of the content of the default configuration file /fluentd/etc/fluent.conf (see, later on in this article) and also the match part from the file fluent.conf, I extracted from the “Fluentd daemonset for Kubernetes and it’s Docker image” Git repository, I mentioned and used earlier.
[https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.3/debian-elasticsearch/conf/fluent.conf]
I only wanted the get the logging from the booksservice containers, so I made some changes, which in the file above, I highlighted in bold.
Remark about using a ConfigMap:
By using a ConfigMap, you can provide configuration data to an application without storing it in the container image or hardcoding it into the pod specification.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/]
I created a ConfigMap that holds the fluentd config file using:
kubectl create configmap fluentd-configmap --from-file=/vagrant/configmaps/configmap-fluentd --namespace nl-amis-logging
Next, I added labels to the ConfigMap using:
kubectl label configmap fluentd-configmap --namespace nl-amis-logging app=fluentd kubectl label configmap fluentd-configmap --namespace nl-amis-logging version="1.0" kubectl label configmap fluentd-configmap --namespace nl-amis-logging environment=logging
A ConfigMap can be created via a yaml file, but not if you want to use the from-file option, because kubernetes isn’t aware of the local file’s path.
[https://stackoverflow.com/questions/51268488/kubernetes-configmap-set-from-file-in-yaml-configuration]
You must create a ConfigMap before referencing it in a Pod specification (unless you mark the ConfigMap as “optional”). If you reference a ConfigMap that doesn’t exist, the Pod won’t start.
ConfigMaps reside in a specific namespace. A ConfigMap can only be referenced by pods residing in the same namespace.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/]
When you create a ConfigMap using –from-file, the filename becomes a key stored in the data section of the ConfigMap. The file contents become the key’s value.
[https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-volume]
In the scripts directory I created a file fluentd.sh with the following content:
#!/bin/bash echo "**** Begin installing Fluentd" #Create ConfigMap before creating DaemonSet kubectl create configmap fluentd-configmap --from-file=/vagrant/configmaps/configmap-fluentd --namespace nl-amis-logging #Label ConfigMap kubectl label configmap fluentd-configmap --namespace nl-amis-logging app=fluentd kubectl label configmap fluentd-configmap --namespace nl-amis-logging version="1.0" kubectl label configmap fluentd-configmap --namespace nl-amis-logging environment=logging #List configmaps echo "**** List configmap fluentd-configmap with namespace nl-amis-logging" kubectl get configmaps fluentd-configmap --namespace nl-amis-logging -o yaml #Create Helm chart echo "**** Create Helm chart" cd /vagrant cd helmcharts rm -rf /vagrant/helmcharts/fluentd-chart/* helm create fluentd-chart rm -rf /vagrant/helmcharts/fluentd-chart/templates/* cp /vagrant/yaml/*fluentd.yaml /vagrant/helmcharts/fluentd-chart/templates # Install Helm chart cd /vagrant cd helmcharts echo "**** Install Helm chart fluentd-chart" helm install ./fluentd-chart --name fluentd-release # Wait 2,5 minute echo "**** Waiting 2,5 minute ..." sleep 150 echo "**** Check if a certain action (list) on a resource (pods) is allowed for a specific user (system:serviceaccount:nl-amis-logging:fluentd-serviceaccount) ****" kubectl auth can-i list pods --as="system:serviceaccount:nl-amis-logging:fluentd-serviceaccount" --namespace nl-amis-logging #List helm releases echo "**** List helm releases" helm list -d #List pods echo "**** List pods with namespace nl-amis-logging" kubectl get pods --namespace nl-amis-logging echo "**** End installing Fluentd"
Remark:
You can list the configmap using:
kubectl get configmaps fluentd-configmap --namespace nl-amis-logging -o yaml
With the following output:
apiVersion: v1
data:
fluentd.conf: |-
<source>
…
So here you can see, that the filename is fluentd.conf.
Getting a shell to the running container gave the following output:
vagrant@ubuntu-xenial:/vagrant$ kubectl exec -it fluentd-zv5w5 –namespace nl-amis-logging — ls -latr /fluentd/etc
total 32
-rw-r–r– 1 root root 1199 Apr 12 18:03 systemd.conf
-rw-r–r– 1 root root 421 Apr 12 18:03 prometheus.conf
-rw-r–r– 1 root root 4848 Apr 12 18:03 kubernetes.conf
-rw-r–r– 1 root root 1871 Apr 12 18:03 fluent.conf
-rw-r–r– 1 root root 0 Apr 12 18:08 disable.conf
drwxr-xr-x 1 fluent fluent 4096 Apr 12 18:08 ..
drwxrwxrwx 3 root root 4096 Apr 21 19:04 custom-config
drwxr-xr-x 1 fluent fluent 4096 Apr 21 19:05 .
vagrant@ubuntu-xenial:/vagrant$
Remark:
The double dash symbol “–” is used to separate the arguments you want to pass to the command from the kubectl arguments.
[https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/]
Before setting the FLUENT_CONF variable, and after I created the fluentd DaemonSet, the log file from the fluentd-container Container gave the following output
…
[info]: parsing config file is succeeded path=“/fluentd/etc/fluent.conf”
[info]: using configuration file: <ROOT>
<source>
@type prometheus
bind “0.0.0.0”
port 24231
metrics_path “/metrics”
</source>
<source>
@type prometheus_output_monitor
</source>
<match fluent.**>
@type null
</match>
<source>
@type tail
@id in_tail_container_logs
path “/var/log/containers/*.log”
pos_file “/var/log/fluentd-containers.log.pos”
tag “kubernetes.*”
read_from_head true
<parse>
@type “json”
time_format “%Y-%m-%dT%H:%M:%S.%NZ”
time_type string
</parse>
</source>
<source>
@type tail
@id in_tail_minion
path “/var/log/salt/minion”
pos_file “/var/log/fluentd-salt.pos”
tag “salt”
<parse>
@type “regexp”
expression /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
time_format “%Y-%m-%d %H:%M:%S”
</parse>
</source>
<source>
@type tail
@id in_tail_startupscript
path “/var/log/startupscript.log”
pos_file “/var/log/fluentd-startupscript.log.pos”
tag “startupscript”
<parse>
@type “syslog”
</parse>
</source>
<source>
@type tail
@id in_tail_docker
path “/var/log/docker.log”
pos_file “/var/log/fluentd-docker.log.pos”
tag “docker”
<parse>
@type “regexp”
expression /^time=”(?<time>[^)]*)” level=(?<severity>[^ ]*) msg=”(?<message>[^”]*)”( err=”(?<error>[^”]*)”)?( statusCode=($<status_code>\d+))?/
</parse>
</source>
<source>
@type tail
@id in_tail_etcd
path “/var/log/etcd.log”
pos_file “/var/log/fluentd-etcd.log.pos”
tag “etcd”
<parse>
@type “none”
</parse>
</source>
<source>
@type tail
@id in_tail_kubelet
multiline_flush_interval 5s
path “/var/log/kubelet.log”
pos_file “/var/log/fluentd-kubelet.log.pos”
tag “kubelet”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_kube_proxy
multiline_flush_interval 5s
path “/var/log/kube-proxy.log”
pos_file “/var/log/fluentd-kube-proxy.log.pos”
tag “kube-proxy”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_kube_apiserver
multiline_flush_interval 5s
path “/var/log/kube-apiserver.log”
pos_file “/var/log/fluentd-kube-apiserver.log.pos”
tag “kube-apiserver”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_kube_controller_manager
multiline_flush_interval 5s
path “/var/log/kube-controller-manager.log”
pos_file “/var/log/fluentd-kube-controller-manager.log.pos”
tag “kube-controller-manager”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_kube_scheduler
multiline_flush_interval 5s
path “/var/log/kube-scheduler.log”
pos_file “/var/log/fluentd-kube-scheduler.log.pos”
tag “kube-scheduler”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_rescheduler
multiline_flush_interval 5s
path “/var/log/rescheduler.log”
pos_file “/var/log/fluentd-rescheduler.log.pos”
tag “rescheduler”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_glbc
multiline_flush_interval 5s
path “/var/log/glbc.log”
pos_file “/var/log/fluentd-glbc.log.pos”
tag “glbc”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_cluster_autoscaler
multiline_flush_interval 5s
path “/var/log/cluster-autoscaler.log”
pos_file “/var/log/fluentd-cluster-autoscaler.log.pos”
tag “cluster-autoscaler”
<parse>
@type “kubernetes”
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format “%m%d %H:%M:%S.%N”
</parse>
</source>
<source>
@type tail
@id in_tail_kube_apiserver_audit
multiline_flush_interval 5s
path “/var/log/kubernetes/kube-apiserver-audit.log”
pos_file “/var/log/kube-apiserver-audit.log.pos”
tag “kube-apiserver-audit”
<parse>
@type “multiline”
format_firstline “/^\\S+\\s+AUDIT:/”
format1 /^(?<time>\S+) AUDIT:(?: (?:id=”(?<id>(?:[^”\\]|\\.)*)”|ip=”(?<ip>(?:[^”\\]|\\.)*)”|method=”(?<method>(?:[^”\\]|\\.)*)”|user=”(?<user>(?:[^”\\]|\\.)*)”|groups=”(?<groups>(?:[^”\\]|\\.)*)”|as=”(?<as>(?:[^”\\]|\\.)*)”|asgroups=”(?<asgroups>(?:[^”\\]|\\.)*)”|namespace=”(?<namespace>(?:[^”\\]|\\.)*)”|uri=”(?<uri>(?:[^”\\]|\\.)*)”|response=”(?<response>(?:[^”\\]|\\.)*)”|\w+=”(?:[^”\\]|\\.)*”))*/
time_format “%Y-%m-%dT%T.%L%Z”
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<match **>
@type elasticsearch
@id out_es
@log_level “info”
include_tag_key true
host “elasticsearch-service.nl-amis-logging”
port 9200
path “”
scheme http
ssl_verify true
ssl_version TLSv1
user “elastic”
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
logstash_prefix “logstash”
logstash_format true
index_name “logstash”
type_name “fluentd”
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
</ROOT>
[info]: starting fluentd-1.3.3 pid=6 ruby=”2.3.3″
…
Installing mysql
In my previous article I already described setting up the following yaml files:
- persistent-volume-mysql.yaml
- persistent-volume-claim-mysql.yaml
- deployment-mysql.yaml
- service-mysql.yaml
Instead of manually starting the kubectl commands as described in that article, now I automated it, so in the scripts directory I created a file mysql.sh with the following content:
#!/bin/bash echo "**** Begin installing MySQL" #Install mysql-client sudo apt-get install -y mysql-client #Create Helm chart echo "**** Create Helm chart" cd /vagrant cd helmcharts rm -rf /vagrant/helmcharts/mysql-chart/* helm create mysql-chart rm -rf /vagrant/helmcharts/mysql-chart/templates/* cp /vagrant/yaml/*mysql.yaml /vagrant/helmcharts/mysql-chart/templates # Install Helm chart cd /vagrant cd helmcharts echo "**** Install Helm chart mysql-chart" helm install ./mysql-chart --name mysql-release # Wait 1 minute echo "**** Waiting 1 minute ..." sleep 60 #List helm releases echo "**** List helm releases" helm list -d #List pods echo "**** List pods with namespace nl-amis-testing" kubectl get pods --namespace nl-amis-testing #List services echo "**** List services with namespace nl-amis-testing" kubectl get service --namespace nl-amis-testing echo "**** Begin preparing mysql database 'test'" echo "**** Forward local port 3306 to port 3306 on the mysql-service service" #/bin/bash -c "kubectl port-forward service/mysql-service 3306 --namespace nl-amis-testing &" kubectl port-forward service/mysql-service 3306 --namespace nl-amis-testing /dev/null & # Wait 2,5 minute echo "**** Waiting 2,5 minute ..." sleep 150 mysql -h 127.0.0.1 -uroot -ppassword -e "create database if not exists test" mysql -h 127.0.0.1 -uroot -ppassword -e "show databases;" echo "**** Creating table book" mysql -h 127.0.0.1 -uroot -ppassword -e "use test; CREATE TABLE book ( id varchar(255) NOT NULL, author varchar(255) DEFAULT NULL, isbn13 varchar(255) DEFAULT NULL, language varchar(255) DEFAULT NULL, num_of_pages int(11) NOT NULL, price double NOT NULL, title varchar(255) DEFAULT NULL, type varchar(255) DEFAULT NULL, PRIMARY KEY (id) );" mysql -h 127.0.0.1 -uroot -ppassword -e "use test; desc book;" echo "**** End preparing mysql database 'test'" echo "**** End installing MySQL"
Remark about the database:
Here you can see that I created the test database with the book table myself (based on the metadata, I got earlier).
I installed the mysql-client for this on the guest operating system and used the kubectl port-forward option to forward the local port on the guest operating system to the port on the Kubernetes cluster.
Remark about </dev/null &>/dev/null &:
There are, by default, three “standard” files open when you run a program, standard input (stdin), standard output (stdout), and standard error (stderr).
In Unix, those are associated with “file descriptors” (stdin = 0, stdout = 1, stderr = 2).
The shell gives you the ability to “redirect” file descriptors:
- The > operator redirects output.
- The < operator redirects input.
/dev/null is the so-called null device, which is a special device which discards the information written to it.
[https://askubuntu.com/questions/12098/what-does-outputting-to-dev-null-accomplish-in-bash-scripts]
program_name </dev/null &>/dev/null &
The above command line means:
- run program_name.
- redirect standard input from /dev/null (</dev/null).
- redirect both file descriptors 1 and 2 (stdout and stderr) to /dev/null (&>/dev/null).
- run the program in the background (&).
Installing booksservices
In my previous article I already described setting up the following yaml files:
- deployment-booksservice-dev-v1.0.yaml
- deployment-booksservice-dev-v2.0.yaml
- deployment-booksservice-tst-v1.0.yaml
- service-booksservice-dev-v1.0.yaml
- service-booksservice-dev-v2.0.yaml
- service-booksservice-tst-v1.0.yaml
Instead of manually starting the kubectl commands as described in that article, now I automated it, so in the scripts directory I created a file booksservices.sh with the following content:
#!/bin/bash
echo "**** Begin installing booksservices"
#Create Helm chart
echo "**** Create Helm chart"
cd /vagrant
cd helmcharts
rm -rf /vagrant/helmcharts/booksservice-chart/*
helm create booksservice-chart
rm -rf /vagrant/helmcharts/booksservice-chart/templates/*
cp /vagrant/yaml/*booksservice*.yaml /vagrant/helmcharts/booksservice-chart/templates
# Create Docker images
echo "**** Docker images"
cd /vagrant
cd applications
cd books_service_1.0
docker build -t booksservice:v1.0 .
cd ..
cd books_service_2.0
docker build -t booksservice:v2.0 .
# Wait 30 seconds
echo "**** Waiting 30 seconds ..."
sleep 30
# Install Helm chart
cd /vagrant
cd helmcharts
echo "**** Install Helm chart booksservice-chart"
helm install ./booksservice-chart --name booksservice-release
# Wait 2,5 minute
echo "**** Waiting 2,5 minute ..."
sleep 150
#List helm releases
echo "**** List helm releases"
helm list -d
#List pods
echo "**** List pods with namespace nl-amis-testing"
kubectl get pods --namespace nl-amis-testing
#List services
echo "**** List services with namespace nl-amis-testing"
kubectl get service --namespace nl-amis-testing
echo "**** Determine the IP of the minikube node"
nodeIP=$(kubectl get node minikube -o yaml | grep address: | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")
echo "---$nodeIP---"
echo "**** Via socat forward local port 9010 to port 30010 on the minikube node ($nodeIP)"
socat tcp-listen:9010,fork tcp:$nodeIP:30010 &
echo "**** Via socat forward local port 9020 to port 30020 on the minikube node ($nodeIP)"
socat tcp-listen:9020,fork tcp:$nodeIP:30020 &
echo "**** Via socat forward local port 9110 to port 30110 on the minikube node ($nodeIP)"
socat tcp-listen:9110,fork tcp:$nodeIP:30110 &
echo "**** Add books"
curl --header "Content-Type: application/json" --request POST --data '{"id": 1, "title": "The Threat: How the FBI Protects America in the Age of Terror and Trump", "author": "Andrew G. McCabe", "type": "Hardcover", "price": 17.99, "numOfPages": 288, "language": "English", "isbn13": "978-1250207579"}' http://localhost:9110/books
curl --header "Content-Type: application/json" --request POST --data '{"id": 2, "title": "Becoming", "publishDate": "2018-11-13", "author": "Michelle Obama", "type": "Hardcover", "price": 17.88, "numOfPages": 448, "publisher": "Crown Publishing Group; First Edition edition", "language": "English", "isbn13": "978-1524763138"}' http://localhost:9110/books
echo "**** Get books"
curl http://localhost:9110/books
echo ""
echo "**** List the books in the database"
mysql -h 127.0.0.1 -uroot -ppassword -e "show databases;"
mysql -h 127.0.0.1 -uroot -ppassword -e "use test; select * from book;"
echo "**** End installing booksservices"
Remark:
Here you can see that via curl commands, I added two books to the book catalog.
Postman
Remember that on my Windows laptop, I also wanted to be able to use Postman (for sending requests), via port forwarding this was made possible.
So, I used Postman to add two extra books to the book catalog.
From Postman I invoked a request named “PostBook3Request” (with method “POST” and URL “http://locahost:9110/books”) and a response with “Status 200 OK” was shown:
Next, from Postman I invoked a request named “PostBook4Request” (with method “POST” and URL “http://locahost:9110/books”) and a response with “Status 200 OK” was shown:
Then I wanted to check if the books were added to the catalog.
From Postman I invoked a request named “GetAllBooksRequest” (with method “GET” and URL “http://locahost:9110/books”) and a response with “Status 200 OK” was shown:
As you can see, both extra books that I added, were also returned from the catalog.
Kubernetes Dashboard
On my Windows laptop, after my demo environment is set up, in a Web Browser I can start the Kubernetes Dashboard via: http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/#!/node?namespace=_all
Check via the Kubernetes Web UI (Dashboard) that the deployment is created (in the nl-amis-logging namespace):
Navigate to Workloads | Pods:
Check via the Kubernetes Web UI (Dashboard) that the deployment is created (in the nl-amis-development namespace):
Navigate to Workloads | Pods:
Check via the Kubernetes Web UI (Dashboard) that the deployment is created (in the nl-amis-testing namespace):
Navigate to Workloads | Pods:
Kibana Dashboard
On my Windows laptop, after my demo environment is set up, in a Web Browser I can start the Kibana Dashboard via: http://localhost:5601/app/kibana
I quickly checked if I could see some logging from the booksservice containers, and this was the case.
In the example below you can see, for example:
In a next article I will dive into using Kibana.
To conclude this blog, I will sum up the script files I used to get it all working:
- Vagrantfile
- docker.sh
- minikube.sh
- kubectl.sh
- helm.sh
- namespaces.sh
- elasticsearch.sh
- kibana.sh
- fluentd.sh
- mysql.sh
- booksservices.sh
In this article I described how I used Vagrant and shell scripts to further automate setting up my demo environment from scratch, including ElasticSearch, Fluentd and Kibana within Minikube.
In a next article I will dive into using ElasticSearch, Fluentd and Kibana in order to do, for example, log aggregation.
