Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 1)

Marc Lameriks

I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE.
[https://kafka.apache.org/]

In my previous article, I described the steps I took, to set up a demo environment with an Oracle Database 21c XE, within an Oracle VirtualBox appliance, with the help of Vagrant.

In this article the focus is on installing Apache Kafka, trying out a quick start example and the problems I encountered, in my demo environment, when using an Open-Source Web GUI for Apache Kafka.

Apache Kafka

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
[https://kafka.apache.org/]

Kafka combines three key capabilities for event streaming:

  • Publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
  • Store streams of events durably and reliably for as long as you want.
  • Process streams of events as they occur or retrospectively.

[https://kafka.apache.org/documentation/#intro_platform]

I won’t dive deep into describing the functionality of Kafka, but I will mention how Kafka works in a nutshell:

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.

Servers: Kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions. Some of these servers form the storage layer, called the brokers. Other servers run Kafka Connect to continuously import and export data as event streams to integrate Kafka with your existing systems such as relational databases as well as other Kafka clusters. To let you implement mission-critical use cases, a Kafka cluster is highly scalable and fault-tolerant: if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.

Clients: They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-tolerant manner even in the case of network problems or machine failures. Kafka ships with some such clients included, which are augmented by dozens of clients provided by the Kafka community: clients are available for Java and Scala including the higher-level Kafka Streams library, for Go, Python, C/C++, and many other programming languages as well as REST APIs.
[https://kafka.apache.org/documentation/#intro_nutshell]

I will also briefly explain some main concepts and terminology:

  • An event records the fact that “something happened” in the world or in your business. It is also called record or message in the documentation. When you read or write data to Kafka, you do this in the form of events. Conceptually, an event has a key, value, timestamp, and optional metadata headers.
  • Producers are those client applications that publish (write) events to Kafka, and consumers are those that subscribe to (read and process) these events.
  • Events are organized and durably stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.
  • Topics are partitioned, meaning a topic is spread over a number of “buckets” located on different Kafka brokers.

[https://kafka.apache.org/documentation/#intro_concepts_and_terms]

Confluent Developer, Apache Kafka® Quick Start

Because my demo environment already had Docker installed, I wanted to install Kafka via a Docker image.

Of course, there are several images to choose from.

I went for Confluent Community Docker Image for Apache Kafka.
[https://hub.docker.com/r/confluentinc/cp-kafka/]

I wanted to start simple, so I followed the Apache Kafka® Quick Start (the guide demonstrating how to quickly get started with Apache Kafka) from the Confluent Developer website. In this Quick Start we will connect to a broker, create a topic, produce some messages, and consume them.
[https://developer.confluent.io/quickstart/kafka-docker/]

As you can read in the Quick Start, for the setup of a Kafka broker, a Docker Compose file is used.

Installing Docker Compose

I did not have Docker Compose running on my VM, so I had to install it.

A prerequisite for installing Docker Compose is Docker Engine for your OS.
[https://docs.docker.com/compose/install/#prerequisites]

I did have Docker Engine already running on my VM.

Remark:
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
[https://docs.docker.com/compose/]

From the subdirectory named env on my Windows laptop, I navigated to the scripts directory were I created a file docker-compose.sh with the following content (following the instructions in https://docs.docker.com/compose/install/):

#!/bin/bash
echo "**** Begin installing Docker Compose"

#Download the current stable release of Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

#Apply executable permissions to the binary
sudo chmod +x /usr/local/bin/docker-compose

#Test the installation
docker-compose --version

echo "**** End installing Docker Compose"

I changed the content of the Vagrantfile:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.define "ubuntu_docker" do |ubuntu_docker|
  
    config.vm.network "forwarded_port",
      guest: 1521,
      host:  1521,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5500,
      host:  5500,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Docker"
        vb.memory = "8192"
        vb.cpus = "1"
      
      args = []
      config.vm.provision "docker shell script", type: "shell",
          path: "scripts/docker.sh",
          args: args
      
      args = []
      config.vm.provision "docker-compose shell script", type: "shell",
          path: "scripts/docker-compose.sh",
          args: args
      
      # args = []
      # config.vm.provision "oracle-database-xe-21c shell script", type: "shell",
      #     path: "scripts/oracle-database-xe-21c.sh",
      #     args: args
      # 
    end
    
  end

end

Remark:
Because in this series of articles the focus is on Apache Kafka, I commented out the lines of code to install Oracle Database 21c XE (21.3.0).

In order to test this new set up, I used vagrant destroy and vagrant up.

With the following output:

ubuntu_docker: **** Begin installing Docker Compose
ubuntu_docker:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ubuntu_docker:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:100   664  100   664    0     0   6707      0 --:--:-- --:--:-- --:--:--  6707
 41 12.1M   41 5186k    0     0  6364k      0  0:00:01 --:--:--  0:100 12.1M  100 12.1M    0     0  7847k      0  0:00:01  0:00:01 --:--:-- 9430k
ubuntu_docker: docker-compose version 1.29.2, build 5becea4c
ubuntu_docker: **** End installing Docker Compose

Setting up and starting a Kafka broker

As I mentioned above, for the setup of a Kafka broker, a Docker Compose file is used.
[https://developer.confluent.io/quickstart/kafka-docker/]

So, on my Windows laptop, I added to the env directory a subdirectory called docker-compose, were I created a file docker-compose.yml with the following content (copied from the Quick Start website):

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

As you can see, this Docker Compose file installs two services, zookeeper (Apache ZooKeeper) and broker (a Kafka broker).

For information about Apache ZooKeeper please see: https://zookeeper.apache.org/doc/current/index.html

From the subdirectory named env on my Windows laptop, I navigated to the scripts directory were I created a file kafka.sh with the following content (following the instructions in https://developer.confluent.io/quickstart/kafka-docker/):

#!/bin/bash
echo "**** Begin installing Kafka"

cd /vagrant/docker-compose

docker-compose up -d

echo "**** End installing Kafka"

Remark about shared folder:

Mounting shared folders...
    ubuntu_docker: /vagrant => C:/My/AMIS/env

Remember, via the shared folder, the copied files are also available from within the Oracle VirtualBox appliance.

Next, I changed the content of the Vagrantfile:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.define "ubuntu_docker" do |ubuntu_docker|
  
    config.vm.network "forwarded_port",
      guest: 1521,
      host:  1521,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5500,
      host:  5500,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Docker"
        vb.memory = "8192"
        vb.cpus = "1"
      
      args = []
      config.vm.provision "docker shell script", type: "shell",
          path: "scripts/docker.sh",
          args: args
      
      args = []
      config.vm.provision "docker-compose shell script", type: "shell",
          path: "scripts/docker-compose.sh",
          args: args
      
      # args = []
      # config.vm.provision "oracle-database-xe-21c shell script", type: "shell",
      #     path: "scripts/oracle-database-xe-21c.sh",
      #     args: args
      # 
      args = []
      config.vm.provision "kafka shell script", type: "shell",
          path: "scripts/kafka.sh",
          args: args
      
    end
    
  end

end

In order to test this new set up, I used vagrant destroy and vagrant up.

With the following output:

ubuntu_docker: **** Begin installing Kafka
ubuntu_docker: Creating network "docker-compose_default" with the default driver
ubuntu_docker: Pulling zookeeper (confluentinc/cp-zookeeper:6.2.0)...
ubuntu_docker: 6.2.0: Pulling from confluentinc/cp-zookeeper
ubuntu_docker: Digest: sha256:9a69c03fd1757c3154e4f64450d0d27a6decb0dc3a1e401e8fc38e5cea881847
ubuntu_docker: Status: Downloaded newer image for confluentinc/cp-zookeeper:6.2.0
ubuntu_docker: Pulling broker (confluentinc/cp-kafka:6.2.0)...
ubuntu_docker: 6.2.0: Pulling from confluentinc/cp-kafka
ubuntu_docker: Digest: sha256:97f572d93c6b2d388c5dadd644a90990ec29e42e5652c550c84d1a9be9d6dcbd
ubuntu_docker: Status: Downloaded newer image for confluentinc/cp-kafka:6.2.0
ubuntu_docker: Creating zookeeper ...
ubuntu_docker: Creating zookeeper ... done
ubuntu_docker: Creating broker    ...
ubuntu_docker: Creating broker    ... done
ubuntu_docker: CONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS                          PORTS                                       NAMES
ubuntu_docker: 0be292c1abbe   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   2 seconds ago        Up Less than a second           0.0.0.0:9092->9092/tcp, :::9092->9092/tcp   broker
ubuntu_docker: 56c589565120   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   3 seconds ago        Up 1 second                     2181/tcp, 2888/tcp, 3888/tcp                zookeeper
ubuntu_docker: edd09e8d6321   hello-world                       "/hello"                 About a minute ago   Exited (0) About a minute ago                                               strange_shockley
ubuntu_docker: **** End installing Kafka

As you can see, the Docker containers named broker and zookeeper are up and running.
I closed the Windows Command Prompt.

Creating a topic and writing and reading messages

With the Kafka broker in place, I then continued with the instructions from the Quick Start.
[https://developer.confluent.io/quickstart/kafka-docker/]

I used vagrant ssh to connect into the running VM. Next, in order to create a topic, I used the following command on the Linux Command Prompt:

docker exec broker \
kafka-topics --bootstrap-server broker:9092 \
             --create \
             --topic quickstart

With the following output:

Created topic quickstart.

Then, in order to write messages to the topic, I used the following command on the Linux Command Prompt:

docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server broker:9092 \
                       --topic quickstart

Next, I typed in some lines of text. Each line is a new message.

this is my first kafka message
hello world!
this is my third kafka message. I’m on a roll :-D

Then, to exit the producer, I typed Ctrl-D to return to my Linux Command Prompt.

Next, in order to read messages from the topic, I used the following command on the Linux Command Prompt:

docker exec --interactive --tty broker \
kafka-console-consumer --bootstrap-server broker:9092 \
                       --topic quickstart \
                       --from-beginning

With the following output:

this is my first kafka message
hello world!
this is my third kafka message. I’m on a roll :-D

Then, to stop the consumer, I typed Ctrl-C to return to my Linux Command Prompt.

So, Kafka was working on my demo environment 😊.

I closed the Windows Command Prompt.

UI for Apache Kafka

What I also would like to have on my demo environment is some kind of Open-Source Web GUI for Apache Kafka.

After some research on the internet, I opted for UI for Apache Kafka.

UI for Apache Kafka is a free, open-source web UI to monitor and manage Apache Kafka clusters.

UI for Apache Kafka is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters – Brokers, Topics, Partitions, Production, and Consumption.

Set up UI for Apache Kafka with just a couple of easy commands to visualize your Kafka data in a comprehensible way. You can run the tool locally or in the cloud.
[https://github.com/provectus/kafka-ui/blob/master/README.md]

From the subdirectory named env on my Windows laptop, I navigated to the scripts directory were I created a file ui-for-apache-kafka.sh with the following content (following the instructions in https://github.com/provectus/kafka-ui/blob/master/README.md#running-from-docker-image):

#!/bin/bash
echo "**** Begin installing UI for Apache Kafka"

docker pull provectuslabs/kafka-ui

docker run -p 8080:8080 \
    -e KAFKA_CLUSTERS_0_NAME=local \
    -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 \
    -d provectuslabs/kafka-ui:latest

docker container ls -a

echo "**** End installing UI for Apache Kafka"

Remark:
I also added a command to get the list of all the Docker containers.

Because the web UI is made available at http://localhost:8080 and I wanted to use a Web Browser on my Windows laptop, I used the forwarded_port configuration option, to forward port 8080 on my host (Windows) to port 8080 on my guest (Ubuntu).

Vagrant forwarded ports allow you to access a port on your host machine and have all data forwarded to a port on the guest machine, over either TCP or UDP.
[https://www.vagrantup.com/docs/networking/forwarded_ports.html]

Next, I changed the content of the Vagrantfile:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.define "ubuntu_docker" do |ubuntu_docker|
  
    config.vm.network "forwarded_port",
      guest: 1521,
      host:  1521,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5500,
      host:  5500,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 8080,
      host:  8080,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Docker"
        vb.memory = "8192"
        vb.cpus = "1"
      
      args = []
      config.vm.provision "docker shell script", type: "shell",
          path: "scripts/docker.sh",
          args: args
      
      args = []
      config.vm.provision "docker-compose shell script", type: "shell",
          path: "scripts/docker-compose.sh",
          args: args
      
      # args = []
      # config.vm.provision "oracle-database-xe-21c shell script", type: "shell",
      #     path: "scripts/oracle-database-xe-21c.sh",
      #     args: args
      # 
      args = []
      config.vm.provision "kafka shell script", type: "shell",
          path: "scripts/kafka.sh",
          args: args
      
      args = []
      config.vm.provision "ui-for-apache-kafka shell script", type: "shell",
          path: "scripts/ui-for-apache-kafka.sh",
          args: args
      
    end
    
  end

end

In order to test this new set up, I used vagrant destroy and vagrant up.

With the following output:

ubuntu_docker: **** Begin installing UI for Apache Kafka
ubuntu_docker: Using default tag: latest
ubuntu_docker: latest: Pulling from provectuslabs/kafka-ui
…
ubuntu_docker: c631eae86696: Pull complete
ubuntu_docker: Digest: sha256:69d612ddf1ce38e9c5f59c47a8430247aa40da1822c32c6857cad48570bad791
ubuntu_docker: Status: Downloaded newer image for provectuslabs/kafka-ui:latest
ubuntu_docker: docker.io/provectuslabs/kafka-ui:latest
ubuntu_docker: 7eb53c0bb75aa5dfa8e1e3fcf6b0e33e83c466a489dd0744b40d707d06ebeecc
ubuntu_docker: CONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS                          PORTS                                       NAMES
ubuntu_docker: 7eb53c0bb75a   provectuslabs/kafka-ui:latest     "/bin/sh -c 'java $J…"   2 seconds ago        Up Less than a second           0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   competent_khorana
ubuntu_docker: b7c477be5799   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   30 seconds ago       Up 29 seconds                   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp   broker
ubuntu_docker: afe900c6f5ec   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   32 seconds ago       Up 30 seconds                   2181/tcp, 2888/tcp, 3888/tcp                zookeeper
ubuntu_docker: 5e4a03590e6e   hello-world                       "/hello"                 About a minute ago   Exited (0) About a minute ago                                               focused_ardinghelli
ubuntu_docker: **** End installing UI for Apache Kafka

On my Windows laptop, in a Web Browser, I entered the URL:

http://localhost:8080

The web UI started.

But unfortunately, the Kafka broker could not be found.

I had a closer look at the following part in my file ui-for-apache-kafka.sh:

-e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 \

Then, I looked at the README.md in the part about Environment Variables.
[https://github.com/provectus/kafka-ui#readme]


[https://github.com/provectus/kafka-ui#env_variables]

In the example from the Quick Start, there also was a bootstrap server mentioned.

docker exec broker \
kafka-topics --bootstrap-server broker:9092 \
             --create \
             --topic quickstart

I noticed the differences in hosts. Of course, using kafka as host wasn’t going to work. It should be broker.
And also, there was an Environment Variable for the Zookeeper service address.

So, I tried the following settings:
[in bold, I highlighted the changes]

docker run -p 8080:8080 \
    -e KAFKA_CLUSTERS_0_NAME=local \
    -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:9092 \
    -e KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181 \
    -d provectuslabs/kafka-ui:latest

But still, I did not get a correct result.

Just to be sure I also tried another Web GUI for Apache Kafka.

This time I opted for Kafdrop.

Kafdrop – Kafka Web UI

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.
[https://github.com/obsidiandynamics/kafdrop]

You can run the Kafdrop JAR directly, via Docker, or in Kubernetes.
For running it with Docker, the images are hosted at hub.docker.com/r/obsidiandynamics/kafdrop.

And to launch the container in the background, the following command can be used:

docker run -d --rm -p 9000:9000 \
    -e KAFKA_BROKERCONNECT=<host:port,host:port> \
    -e JVM_OPTS="-Xms32M -Xmx64M" \
    -e SERVER_SERVLET_CONTEXTPATH="/" \
    obsidiandynamics/kafdrop

[https://github.com/obsidiandynamics/kafdrop#running-with-docker]

Using this, I still had to work out the correct value of the Environment Variable KAFKA_BROKERCONNECT for my demo environment.

Lucky for me, there also was a docker-compose.yaml file that bundles a Kafka/ZooKeeper instance with Kafdrop.
[https://github.com/obsidiandynamics/kafdrop#docker-compose]

So, I had a look at it:

version: "2"
services:
  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:29092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - "kafka"
  kafka:
    image: obsidiandynamics/kafka
    restart: "no"
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

[https://raw.githubusercontent.com/obsidiandynamics/kafdrop/master/docker-compose/kafka-kafdrop/docker-compose.yaml]

Based on this example, I changed the content of file docker-compose.yml, I used before:
[in bold, I highlighted the changes]

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "broker:29092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - broker

Remark:
Of course, using kafka wasn’t going to work. So, I changed it to broker.

Because the web UI is made available at http://localhost:9000 and I wanted to use a Web Browser on my Windows laptop, I used the forwarded_port configuration option, to forward port 9000 on my host (Windows) to port 9000 on my guest (Ubuntu).

Next, I changed the content of the Vagrantfile:
[in bold, I highlighted the changes]

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.define "ubuntu_docker" do |ubuntu_docker|
  
    config.vm.network "forwarded_port",
      guest: 1521,
      host:  1521,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 5500,
      host:  5500,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 8080,
      host:  8080,
      auto_correct: true
      
    config.vm.network "forwarded_port",
      guest: 9000,
      host:  9000,
      auto_correct: true
      
    config.vm.provider "virtualbox" do |vb|
        vb.name = "Ubuntu Docker"
        vb.memory = "8192"
        vb.cpus = "1"
      
      args = []
      config.vm.provision "docker shell script", type: "shell",
          path: "scripts/docker.sh",
          args: args
      
      args = []
      config.vm.provision "docker-compose shell script", type: "shell",
          path: "scripts/docker-compose.sh",
          args: args
      
      # args = []
      # config.vm.provision "oracle-database-xe-21c shell script", type: "shell",
      #     path: "scripts/oracle-database-xe-21c.sh",
      #     args: args
      # 
      args = []
      config.vm.provision "kafka shell script", type: "shell",
          path: "scripts/kafka.sh",
          args: args
      
      args = []
      config.vm.provision "ui-for-apache-kafka shell script", type: "shell",
          path: "scripts/ui-for-apache-kafka.sh",
          args: args
      
    end
    
  end

end

In order to test this new set up, I used vagrant destroy and vagrant up.

With the following output:

ubuntu_docker: **** Begin installing Kafka
ubuntu_docker: Creating network "docker-compose_default" with the default driver
ubuntu_docker: Pulling zookeeper (confluentinc/cp-zookeeper:6.2.0)...
ubuntu_docker: 6.2.0: Pulling from confluentinc/cp-zookeeper
ubuntu_docker: Digest: sha256:9a69c03fd1757c3154e4f64450d0d27a6decb0dc3a1e401e8fc38e5cea881847
ubuntu_docker: Status: Downloaded newer image for confluentinc/cp-zookeeper:6.2.0
ubuntu_docker: Pulling broker (confluentinc/cp-kafka:6.2.0)...
ubuntu_docker: 6.2.0: Pulling from confluentinc/cp-kafka
ubuntu_docker: Digest: sha256:97f572d93c6b2d388c5dadd644a90990ec29e42e5652c550c84d1a9be9d6dcbd
ubuntu_docker: Status: Downloaded newer image for confluentinc/cp-kafka:6.2.0
ubuntu_docker: Pulling kafdrop (obsidiandynamics/kafdrop:)...
ubuntu_docker: latest: Pulling from obsidiandynamics/kafdrop
ubuntu_docker: Digest: sha256:b7ba8577ce395b1975b0ed98bb53cb6b13e7d32d5442188da1ce41c0838d1ce9
ubuntu_docker: Status: Downloaded newer image for obsidiandynamics/kafdrop:latest
ubuntu_docker: Creating zookeeper ...
ubuntu_docker: Creating zookeeper ... done
ubuntu_docker: Creating broker    ...
ubuntu_docker: Creating broker    ... done
ubuntu_docker: Creating docker-compose_kafdrop_1 ...
ubuntu_docker: Creating docker-compose_kafdrop_1 ... done
ubuntu_docker: CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS                     PORTS                                       NAMES
ubuntu_docker: 46cba830184c   obsidiandynamics/kafdrop          "/kafdrop.sh"            1 second ago    Up Less than a second      0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   docker-compose_kafdrop_1
ubuntu_docker: 625d446a5eb0   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   2 seconds ago   Up 1 second                0.0.0.0:9092->9092/tcp, :::9092->9092/tcp   broker
ubuntu_docker: a8980809b060   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   5 seconds ago   Up 2 seconds               2181/tcp, 2888/tcp, 3888/tcp                zookeeper
ubuntu_docker: d9fe3cfdb718   hello-world                       "/hello"                 6 minutes ago   Exited (0) 6 minutes ago                                               loving_rhodes
ubuntu_docker: **** End installing Kafka

On my Windows laptop, in a Web Browser, I entered the URL:

http://localhost:9000

The web UI started.

And the broker was recognized.

I closed the Windows Command Prompt.

Now, I wanted to create a topic and write messages to the topic, just to see if these were picked up by the web UI. So, I repeated some of the steps from the Quick Start.

I used vagrant ssh to connect into the running VM. Next, in order to create a topic, I used the following command on the Linux Command Prompt:

docker exec broker \
kafka-topics --bootstrap-server broker:9092 \
             --create \
             --topic quickstart

Then, in order to write messages to the topic, I used the following command on the Linux Command Prompt:

docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server broker:9092 \
                       --topic quickstart

Next, I typed in some lines of text. Each line is a new message.

this is my first kafka message
hello world!
this is my third kafka message. I’m on a roll :-D

Then, to exit the producer, I typed Ctrl-D to return to my Linux Command Prompt.

On my Windows laptop, I refreshed the Web Browser.

The quickstart topic became visible.

Then I clicked on the View Messages button.

Next, I clicked on the Search and View Messages button.

The 3 messages were found.

So, I finally got the result I was looking for.

Now it was time to understand why the other web UI (UI for Apache Kafka) wasn’t working.

More about the way forward, you can read in my next article. But I can already say that I got it working. And of course, in hindsight it all makes sense.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Next Post

Dapr and Kafka–easy binding

Dapr is a runtime framework for distributed applications. Dapr also provides a personal assistant to any application and environment that offers standardized access to different facilities (state, secrets, configuration, pub/sub, actors) and technologies. Simply by using Dapr we can leverage a large number of Dapr components that through the Dapr […]
%d bloggers like this: