Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 2)

Marc Lameriks

I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE.
[https://kafka.apache.org/]

In a series of articles, I will take you with me on my path to set everything up.

In my previous article, I described the steps I took, to set up a demo environment with Apache Kafka, within an Oracle VirtualBox appliance, with the help of Vagrant.

Remember, the problems I encountered, in my demo environment, when using an Open-Source Web GUI for Apache Kafka. I described that I used the docker run command to create the container for “UI for Apache Kafka”.

On my Windows laptop, in a Web Browser (http://localhost:8080), the web UI started, but unfortunately, the Kafka broker could not be found.

On the other hand, when I tried another Web GUI for Apache Kafka named “Kafdrop” (http://localhost:9000), where the container was created via a docker-compose.yaml file that bundles a Kafka/ZooKeeper instance with Kafdrop, everything worked as expected.

In this article the focus is on getting “UI for Apache Kafka” to work on my environment.

Investigating the problem

Let’s have a look again, at the file docker-compose.yml, that bundles a Kafka/ZooKeeper instance with Kafdrop:

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "broker:29092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - broker

It became clear that for the connection to the Kafka broker the following was used:

broker:29092

So, for the Docker run command for installing UI for Apache Kafka, I now tried the following settings:
[in bold, I highlighted the changes]

docker run -p 8080:8080 \
    -e KAFKA_CLUSTERS_0_NAME=local \
    -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092 \
    -e KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181 \
    -d provectuslabs/kafka-ui:latest

But still, I did not get a correct result.

At this moment, these were the Docker containers:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1b353f1de961 provectuslabs/kafka-ui:latest "/bin/sh -c 'java $J…" 1 second ago Up Less than a second 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp stoic_vaughan
46cba830184c obsidiandynamics/kafdrop "/kafdrop.sh" About a minute ago Up About a minute 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp docker-compose_kafdrop_1
625d446a5eb0 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" About a minute ago Up About a minute 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp broker
a8980809b060 confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" About a minute ago Up About a minute 2181/tcp, 2888/tcp, 3888/tcp zookeeper
d9fe3cfdb718 hello-world "/hello" 7 minutes ago Exited (0) 7 minutes ago loving_rhodes

Remark:
As you can see, the kafka-ui container got the name stoic_vaughan.
If you do not assign a container name with the –name option, then the daemon generates a random string name for you
[https://docs.docker.com/engine/reference/run/]

In the file docker-compose.yml used in the Quick Start, there was a link that was the start for me to understand what was going on.

# To learn about configuring Kafka for access across networks see
# https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

[https://developer.confluent.io/quickstart/kafka-docker/]

So, I read its content:

As it turned out, the article was from Robin Moffatt, who I actually heard speaking about Kafka at a Devoxx event.

When a client wants to send or receive a message from Apache Kafka®, there are two types of connection that must succeed:

  • The initial connection to a broker (the bootstrap). This returns metadata to the client, including a list of all the brokers in the cluster and their connection endpoints.
  • The client then connects to one (or more) of the brokers returned in the first step as required. If the broker has not been configured correctly, the connections will fail.

What sometimes happens is that people focus on only step 1 above, and get caught out by step 2. The broker details returned in step 1 are defined by the advertised.listeners setting of the broker(s) and must be resolvable and accessible from the client machine.
[https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/]

The article describes connecting a client to Kafka in various permutations of deployment topology.

And this is also mentioned:
What often goes wrong is that the broker is misconfigured and returns an address (the advertised.listener) on which the client cannot correctly connect to the broker.

I also read, amongst others, the following article from my colleague Maarten Smeets:
“Docker host and bridged networking. Running library/httpd on different ports”.
[https://technology.amis.nl/platform/docker/docker-host-and-bridged-networking-running-library-httpd-on-different-ports/]

So, let’s look at what I learned from this.

Docker networking:
When you start Docker, a default bridge network (also called bridge) is created automatically, and newly-started containers connect to it unless otherwise specified.

Containers on the default bridge network can only access each other by IP addresses, unless you use the –link option, which is considered legacy.

You can also create user-defined custom bridge networks.
On a user-defined bridge network, containers can resolve each other by name or alias.

During a container’s lifetime, you can connect or disconnect it from user-defined networks on the fly. To remove a container from the default bridge network, you need to stop the container and recreate it with different network options.
[https://docs.docker.com/network/bridge/]

Docker Compose networking:
By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

Your app’s network is given a name based on the “project name”, which is based on the name of the directory it lives in.
[https://docs.docker.com/compose/networking/]

Then, in order to list networks, I used the following command on the Linux Command Prompt:

docker network ls

With the following output:

NETWORK ID     NAME                     DRIVER    SCOPE
89add6982ae0   bridge                   bridge    local
b3fae2457c48   docker-compose_default   bridge    local
fe6979946f7d   host                     host      local
3b54b993d1c5   none                     null      local

Remember, on my Windows laptop, I added file docker-compose.yml, used in the Quick Start, in a subdirectory called docker-compose:

C:\My\AMIS\env\docker-compose\docker-compose.yml

And it’s available via shared folders on Linux as:

/vagrant/docker-compose\docker-compose.yml

So, by default Compose sets up a single network, in my case called: docker-compose_default

Then, in order to display detailed information on one or more networks (in my case the network named bridge) , I used the following command on the Linux Command Prompt:

docker network inspect bridge

With the following output:

[
    {
        "Name": "bridge",
        "Id": "89add6982ae07e4979cd696594ea25729b73f82a6abf263e93b81107726c0f61",
        "Created": "2022-01-03T18:35:54.952566708Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1b353f1de961c61a79a306154c9ba9f7b0982c5fc07e7b7f017a319b071f8692": {
                "Name": "stoic_vaughan",
                "EndpointID": "e23bce3d292fd76c212847bc6ad0a3165c8d4f06c01ae5918b4f967f5f38ac7f",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Next, in order to display detailed information on one or more networks (in my case the network named docker-compose_default) , I used the following command on the Linux Command Prompt:

docker network inspect docker-compose_default

With the following output:

[
    {
        "Name": "docker-compose_default",
        "Id": "b3fae2457c48407bfdd855627fa017899db62063464a4f2382ec1b48c20fe001",
        "Created": "2022-01-05T17:41:31.976044066Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "53a4b2cf9273e566c909c698ed62adf1885f43472d3dfead8e5dd21641868dae": {
                "Name": "docker-compose_kafdrop_1",
                "EndpointID": "bf01ffbf0f45cbf96aef34fd6cb71ea3e1d6ffa35bef53323d4ad0e2a0accc46",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "ad1c34b14bc8d06ca7840f263649b36e30177cf561fb84342321f948eabafa3b": {
                "Name": "broker",
                "EndpointID": "c038acbdd599f1ffc3665386aa7bc6f684ba2ba12d95098badbacd9d8df8e692",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "b64dece2636059905c32504ed2324455b3a721a3ea220931af53af4d3af25ff5": {
                "Name": "zookeeper",
                "EndpointID": "5b827fc0ef3566ced2f51650279af050c11a44ce601a991716261fac48fecc0c",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "docker-compose",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

To summarize this, I created this overview:

So, because the kafka-ui container was running on a different network than the kafka broker container, the connection failed.

You can connect a running container to a network, with the following command:

docker network connect [OPTIONS] NETWORK CONTAINER

Connects a container to a network. You can connect a container by name or by ID. Once connected, the container can communicate with other containers in the same network.
[https://docs.docker.com/engine/reference/commandline/network_connect/]

But I opted for connecting a container to a network when it starts.
[https://docs.docker.com/engine/reference/run/#network-settings]

So, I changed the content of file ui-for-apache-kafka.sh:
[in bold, I highlighted the changes]

#!/bin/bash
echo "**** Begin installing UI for Apache Kafka"

docker pull provectuslabs/kafka-ui

docker run --name kafka-ui \
    --network docker-compose_default \
    -p 8080:8080 \
    -e KAFKA_CLUSTERS_0_NAME=local \
    -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092 \
    -d provectuslabs/kafka-ui:latest 

docker container ls -a

echo "**** End installing UI for Apache Kafka"

Remark:
I also assigned a container name with the –name option.

In order to test this new set up, I used vagrant destroy and vagrant up.

With the following output:

ubuntu_docker: **** Begin installing UI for Apache Kafka
ubuntu_docker: Using default tag: latest
ubuntu_docker: latest: Pulling from provectuslabs/kafka-ui
...
ubuntu_docker: Status: Downloaded newer image for provectuslabs/kafka-ui:latest
ubuntu_docker: docker.io/provectuslabs/kafka-ui:latest
ubuntu_docker: 3d99bb38d660e261966e9d3c9a01b9f2927ba95f0c83abe2f0630635cf5e4a80
ubuntu_docker: CONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS                     PORTS                                       NAMES
ubuntu_docker: 3d99bb38d660   provectuslabs/kafka-ui:latest     "/bin/sh -c 'java $J…"   5 seconds ago        Up Less than a second      0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   kafka-ui
ubuntu_docker: 08ca78c86c60   obsidiandynamics/kafdrop          "/kafdrop.sh"            About a minute ago   Up About a minute          0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   docker-compose_kafdrop_1
ubuntu_docker: 51ce7a7fd518   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   About a minute ago   Up About a minute          0.0.0.0:9092->9092/tcp, :::9092->9092/tcp   broker
ubuntu_docker: 4df9df3c5c9a   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   About a minute ago   Up About a minute          2181/tcp, 2888/tcp, 3888/tcp                zookeeper
ubuntu_docker: 8350952ef978   hello-world                       "/hello"                 5 minutes ago        Exited (0) 5 minutes ago                                               nervous_carver
ubuntu_docker: **** End installing UI for Apache Kafka

I closed the Windows Command Prompt.

Creating a topic and writing and reading messages

Now, I wanted to create a topic and write messages to the topic, just to see if these were picked up by the web UI. So, I repeated some of the steps from the Quick Start.
[https://developer.confluent.io/quickstart/kafka-docker/]

UI for Apache Kafka

I used vagrant ssh to connect into the running VM. Next, I used the appropriate commands (I described in my previous article) on the Linux Command Prompt.

On my Windows laptop, in a Web Browser, I entered the URL:

http://localhost:8080

The web UI started.

This time, the Kafka broker could be found.

In the left menu, I clicked on Topics.

Next, I clicked on quickstart.

Then, I clicked on tab Messages.

The 3 messages were found.

So, I finally got the result I was looking for.

Then, in order to list networks, I used the following command on the Linux Command Prompt:

docker network ls

With the following output:

NETWORK ID     NAME                     DRIVER    SCOPE
a669d30f7ed4   bridge                   bridge    local
f356f74a1780   docker-compose_default   bridge    local
ae86e253ccb2   host                     host      local
a1c22fae8af1   none                     null      local

Then, in order to display detailed information on one or more networks (in my case the network named bridge) , I used the following command on the Linux Command Prompt:

docker network inspect bridge

With the following output:

[
    {
        "Name": "bridge",
        "Id": "a669d30f7ed4c98758c967c71505f0f25b0df0ad1d58ef0e427d626845f2cbe4",
        "Created": "2022-01-07T06:40:29.803554769Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

As you can see, there are no containers in this network.

Next, in order to display detailed information on one or more networks (in my case the network named docker-compose_default) , I used the following command on the Linux Command Prompt:

docker network inspect docker-compose_default

With the following output:

[
    {
        "Name": "docker-compose_default",
        "Id": "f356f74a17808d8c6dd40621dcac24b124007106a99af67430bb120ecf8a8f8d",
        "Created": "2022-01-07T06:41:07.231820169Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "08ca78c86c6059da3b900ee6d9d79a302acd806f1cbb534b04c0f8399605747d": {
                "Name": "docker-compose_kafdrop_1",
                "EndpointID": "9d5457de4eade244bc688addc51e25f45defb9eb6bbe777744297af02adf3d8e",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            },
            "3d99bb38d660e261966e9d3c9a01b9f2927ba95f0c83abe2f0630635cf5e4a80": {
                "Name": "kafka-ui",
                "EndpointID": "d6f9ffbf88b70a6bb2bedea4cc8902cf97216685c640aeb312543e1cc4613a7d",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": ""
            },
            "4df9df3c5c9ae5c507c2fc2abfbddb69ac712342cd8042cf24468f212f73c865": {
                "Name": "zookeeper",
                "EndpointID": "fdffbe7e25330f448392e9e05560217d8754a39ce32540b078a8b94b711bac6f",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "51ce7a7fd5180f4c804bb16df6b6aa1821250fe3228fd90c3fa824728b24254a": {
                "Name": "broker",
                "EndpointID": "0d371fabbfc317494fcf306cf80f1523c077a03d30cf1eb8eebd9c9bed92f6e5",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "docker-compose",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

To summarize this, I created this overview:

As you can see, the kafka-ui container is now part of the same network as the other containers.

So, besides “Kafdrop”, now I also got “UI for Apache Kafka” to work on my demo environment. Although I now had two Web GUI’s for Apache Kafka on my demo environment, I decided to leave it like this for now.
In the next part of this series of articles the focus is on installing and using Apache Kafka Connect.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Next Post

Update Go on Linux

It turned out to be a fairly small deal. For reference sake a brief write up of updating the Go runtime on my Ubuntu environment. Current version: go version Find latest releases of Go at https://go.dev/dl/ There is a new one. I copy the link and then run: wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz […]
%d bloggers like this: