I frequently want to demonstrate the working of Apache Kafka and/or provide colleagues and other audiences with an environment for a workshop that involves Apache Kafka. I have been struggling to find a good way to get Kafka to function properly on Minikube – the standalone, single node Kubernetes cluster for experiments. This weekend, I have worked with a set of K8S resource files that gives me what I need. The GitHub repository created by Diego Alvarez turned out to be a very lucky find: https://github.com/d1egoaz/minikube-kafka-cluster .
In a few steps, I can get start a Kafka environment in Minikube, either on my laptop or in a Katacoda environment.
Follow these instructions for getting a Kafka Cluster running on your Kubernetes cluster.
git clone https://github.com/d1egoaz/minikube-kafka-cluster
cd minikube-kafka-cluster
Then execute these four statements to create the Kubernetes resources that run the Kafka cluster.
kubectl apply -f 00-namespace/
kubectl apply -f 01-zookeeper/
kubectl apply -f 02-kafka/
kubectl apply -f 03-yahoo-kafka-manager/
In the Kubernetes dashboard, switch to the kafka-ca1 namespace and look what has been created: Note: it will take some time for all Pods to be running correctly. After a little while, things are likely to get right – unless the minikube cluster is starved of physical resources (memory and disk).
Pods were created as well as services, deployments and stateful sets. Zookeeper has been installed as well as three Kafka Broker Pods. Together these form the Kafka Cluster – a pretty powerful event hub.
At this point, the Kafka Cluster is running. I can check the pods and services in the Kubernetes Dashboard as well as through kubectl on the command line.
Run Kafka Manager
Execute at the command line
kubectl get svc –namespace=kafka-ca1
This should list the services in kafka-ca1 namespace, one of which is kafka-manager. We are looking for the port at which this service is exposed:
Using ‘minikube ip’ we can learn the IP address for the minikube cluster.
In your browser, you can open the Kafka Manager using the minikube cluster IP and the kafka-manager service port:
Open the Cluster menu and click on Add Cluster:
Add new cluster, and use the following data for Cluster Zookeeper Hosts:
zookeeper-service:2181 and press Save. Note: this is not the creation of new Kafka cluster but instead the registration with Kafka Manager of an existing cluster.
Now you can use Kafka Manager to manage and monitor the Kafka Cluster.
Create Topic
As our next step, we need to create a Topic on the Kafka Cluster. Click on Topics, the on Create:
Set the name to the new topic to event-bus. Accept all other default settings and press Create.
Run an Application with Kafka Interaction on Kubernetes
Run a Pod on Kubernetes that starts a Node.JS application that listens to the Event Bus topic on the Kafka Broker.
git clone https://github.com/lucasjellema/2019-fontys-business-and-IT-agility-through-microservice-architecture
cd 2019-fontys-business-and-IT-agility-through-microservice-architecture/4-kafka
You can run the application using the EventBusListenerPod.yaml file in directory 4-kafka/event-bus-listener. Before you run the yaml file, check out its contents. Take note of the ports, the environment variables such as KAFKA_TOPIC (set to event-bus) and KAFKA_HOST (set to zookeeper-service.kafka-ca1, the fully qualified name of the name of the Zookeeper service).
Now run from the specified directory:
kubectl create -f EventBusListenerPod.yaml -f EventBusListenerService.yaml
You can check in the Kubernetes Dashboard whether the Pod has started running successfully and is now listening to a Kafka Topic:
Using the port assigned to service
vou can access the event bus listener application from the browser on your laptop (using the IP address of the minikube cluster) and get a list of all events consumed thusfar from topic event-bus – which are probably none at this stage:
Kafka Manager has recognized the listener or consumer:
Application on Kubernetes that Publishes Events
As a next step, we will run an application that takes input from you – the user – through simple HTTP requests and that turns each request into an event published on the event bus. The event bus listener application that we launched in the previous section will consume all those events and make them available through the browser. That means we realize communication between two applications that are completely unaware of each other and only need to know about the common platform event-bus capability.
Directory 4-kafka\event-bus-publisher contains the sources for this application. Deploy Pod ad Services for the Event Bus Publisher:
kubectl create -f EventBusPublisherPod.yaml EventBusPublisherService.yaml
Check in Kubernetes Dashboard or through kubernetes get pods and kubernetes get services if the creation is complete – and what the port is that was assigned to the EventBusPublisherService.
With http requests from your browser – or using CURL or Postman – you can trigger the Event Bus Publisher microservice into publishing events to the Event Bus:
http://<Kubernetes IP>:<port assigned to Kubernetes service>/publish?area=greenland&importance=high&color=orange
The Event Bus Listener application is still running. Through the browser – or by looking at the logs for the Pod – you can check if it has consumed the event that was just published through the Event Bus Publisher application.
When the HTTP Request to the Event Bus Publisher results in a message returned from the Event Bus Listener application, then we have succeeded.
Run Apache Kafka on Katacoda
I am a fan of Katacoda. Katacoda is a cloud platform that provides browser based access to many playgrounds and scenarios for learning about and experimenting with various technologies. Using Katacoda, anyone can get going with technologies very quickly and without need for local resources.
Running Apache Kafka on Katacoda is quite easy to do with Diego Alvarez’ GitHub repo.
Here are the quick steps:
1. open Katacoda’s Minikube Kubernetes scenario at: Katacode Launch Single Node Cluster
Minikube has been installed and configured in the environment. Minikube is a virtualized Kubernetes cluster instance – that you can also easily run locally on your laptop – that is typically used to study and try out Kubernetes in a small scale environment.
2. Walk your way through this scenario:
- verify minikube is installed and working correctly
- get information on the Kubernetes Cluster that is running under control of Minikube
- optionally run a deployment on Kubernetes (based on a public Docker Container image)
- run and inspect the Kubernetes Dashboard – a GUI that offers great overview of what is happening on the K8S cluster Note: stop at step 4; do not press continue.
3. Get hold of the Kubernetes YAML files that will spin up the Kafka Cluster for us.
git clone https://github.com/d1egoaz/minikube-kafka-cluster
4. Go into the newly created directory:
cd minikube-kafka-cluster
and execute these four statements to create the Kubernetes resources that run the Kafka cluster.
kubectl apply -f 00-namespace/ kubectl apply -f 01-zookeeper/ kubectl apply -f 02-kafka/ kubectl apply -f 03-yahoo-kafka-manager/
It will take a minute or two before the Kafka Cluster is running. The kafka-ca1 namespace may look pretty bad for a while in the dashboard, with Pods trying to get started. Sit back, wait for a while and things will improve.
5. Running Kafka Manager
When all Pods are green in the dashboard, it is time for next moves.
Execute:
kubectl get svc --namespace kafka-ca1
This should list the services in kafka-ca1 namespace, one of which is kafka-manager. We are looking for the port at which this service is exposed.
Now click on the plus icon and open Select Port to View on Host 1 ; enter the port number found for the kafka-manager service.
This should open a new browser window, showing the Kafka Manager. Open the Cluster menu and click on Add Cluster. Add new cluster, and use the following data for Cluster Zookeeper Hosts: zookeeper-service:2181. Use the name kafka for the Cluster (although frankly any name will do) and press Save.
Note: this is not the creation of new Kafka cluster but instead the registration with Kafka Manager of an existing cluster.
Now you can create a Topic on the Kafka Cluster. As a next step you can deploy Pods that access the Kafka Cluster. They can connect to KAFKA_HOST= kafka.kafka-ca1 and KAFKA_PORT=9092.
