I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE.
[https://kafka.apache.org/]
In a series of articles, I will take you with me on my path to set everything up.
In my previous article, I described the steps I took, to set up a demo environment with Apache Kafka, within an Oracle VirtualBox appliance, with the help of Vagrant.
Remember, the problems I encountered, in my demo environment, when using an Open-Source Web GUI for Apache Kafka. I described that I used the docker run command to create the container for “UI for Apache Kafka”.
On my Windows laptop, in a Web Browser (http://localhost:8080), the web UI started, but unfortunately, the Kafka broker could not be found.
On the other hand, when I tried another Web GUI for Apache Kafka named “Kafdrop” (http://localhost:9000), where the container was created via a docker-compose.yaml file that bundles a Kafka/ZooKeeper instance with Kafdrop, everything worked as expected.
In this article the focus is on getting “UI for Apache Kafka” to work on my environment.
Investigating the problem
Let’s have a look again, at the file docker-compose.yml, that bundles a Kafka/ZooKeeper instance with Kafdrop:
--- version: '3' services: zookeeper: image: confluentinc/cp-zookeeper:6.2.0 container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-kafka:6.2.0 container_name: broker ports: # To learn about configuring Kafka for access across networks see # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 kafdrop: image: obsidiandynamics/kafdrop restart: "no" ports: - "9000:9000" environment: KAFKA_BROKERCONNECT: "broker:29092" JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify" depends_on: - broker
It became clear that for the connection to the Kafka broker the following was used:
So, for the Docker run command for installing UI for Apache Kafka, I now tried the following settings:
[in bold, I highlighted the changes]
docker run -p 8080:8080 \ -e KAFKA_CLUSTERS_0_NAME=local \ -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092 \ -e KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181 \ -d provectuslabs/kafka-ui:latest
But still, I did not get a correct result.
At this moment, these were the Docker containers:
| CONTAINER ID | IMAGE | COMMAND | CREATED | STATUS | PORTS | NAMES |
| 1b353f1de961 | provectuslabs/kafka-ui:latest | "/bin/sh -c 'java $J…" | 1 second ago | Up Less than a second | 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp | stoic_vaughan |
| 46cba830184c | obsidiandynamics/kafdrop | "/kafdrop.sh" | About a minute ago | Up About a minute | 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp | docker-compose_kafdrop_1 |
| 625d446a5eb0 | confluentinc/cp-kafka:6.2.0 | "/etc/confluent/dock…" | About a minute ago | Up About a minute | 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp | broker |
| a8980809b060 | confluentinc/cp-zookeeper:6.2.0 | "/etc/confluent/dock…" | About a minute ago | Up About a minute | 2181/tcp, 2888/tcp, 3888/tcp | zookeeper |
| d9fe3cfdb718 | hello-world | "/hello" | 7 minutes ago | Exited (0) 7 minutes ago | loving_rhodes |
Remark:
As you can see, the kafka-ui container got the name stoic_vaughan.
If you do not assign a container name with the –name option, then the daemon generates a random string name for you
[https://docs.docker.com/engine/reference/run/]
In the file docker-compose.yml used in the Quick Start, there was a link that was the start for me to understand what was going on.
# To learn about configuring Kafka for access across networks see # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
[https://developer.confluent.io/quickstart/kafka-docker/]
So, I read its content:
As it turned out, the article was from Robin Moffatt, who I actually heard speaking about Kafka at a Devoxx event.
When a client wants to send or receive a message from Apache Kafka®, there are two types of connection that must succeed:
- The initial connection to a broker (the bootstrap). This returns metadata to the client, including a list of all the brokers in the cluster and their connection endpoints.
- The client then connects to one (or more) of the brokers returned in the first step as required. If the broker has not been configured correctly, the connections will fail.
What sometimes happens is that people focus on only step 1 above, and get caught out by step 2. The broker details returned in step 1 are defined by the advertised.listeners setting of the broker(s) and must be resolvable and accessible from the client machine.
[https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/]
The article describes connecting a client to Kafka in various permutations of deployment topology.
And this is also mentioned:
What often goes wrong is that the broker is misconfigured and returns an address (the advertised.listener) on which the client cannot correctly connect to the broker.
I also read, amongst others, the following article from my colleague Maarten Smeets:
“Docker host and bridged networking. Running library/httpd on different ports”.
[https://technology.amis.nl/platform/docker/docker-host-and-bridged-networking-running-library-httpd-on-different-ports/]
So, let’s look at what I learned from this.
Docker networking:
When you start Docker, a default bridge network (also called bridge) is created automatically, and newly-started containers connect to it unless otherwise specified.
Containers on the default bridge network can only access each other by IP addresses, unless you use the –link option, which is considered legacy.
You can also create user-defined custom bridge networks.
On a user-defined bridge network, containers can resolve each other by name or alias.
During a container’s lifetime, you can connect or disconnect it from user-defined networks on the fly. To remove a container from the default bridge network, you need to stop the container and recreate it with different network options.
[https://docs.docker.com/network/bridge/]
Docker Compose networking:
By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
Your app’s network is given a name based on the “project name”, which is based on the name of the directory it lives in.
[https://docs.docker.com/compose/networking/]
Then, in order to list networks, I used the following command on the Linux Command Prompt:
docker network ls
With the following output:
NETWORK ID NAME DRIVER SCOPE 89add6982ae0 bridge bridge local b3fae2457c48 docker-compose_default bridge local fe6979946f7d host host local 3b54b993d1c5 none null local
Remember, on my Windows laptop, I added file docker-compose.yml, used in the Quick Start, in a subdirectory called docker-compose:
C:\My\AMIS\env\docker-compose\docker-compose.yml
And it’s available via shared folders on Linux as:
/vagrant/docker-compose\docker-compose.yml
So, by default Compose sets up a single network, in my case called: docker-compose_default
Then, in order to display detailed information on one or more networks (in my case the network named bridge) , I used the following command on the Linux Command Prompt:
docker network inspect bridge
With the following output:
[
{
"Name": "bridge",
"Id": "89add6982ae07e4979cd696594ea25729b73f82a6abf263e93b81107726c0f61",
"Created": "2022-01-03T18:35:54.952566708Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"1b353f1de961c61a79a306154c9ba9f7b0982c5fc07e7b7f017a319b071f8692": {
"Name": "stoic_vaughan",
"EndpointID": "e23bce3d292fd76c212847bc6ad0a3165c8d4f06c01ae5918b4f967f5f38ac7f",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
Next, in order to display detailed information on one or more networks (in my case the network named docker-compose_default) , I used the following command on the Linux Command Prompt:
docker network inspect docker-compose_default
With the following output:
[
{
"Name": "docker-compose_default",
"Id": "b3fae2457c48407bfdd855627fa017899db62063464a4f2382ec1b48c20fe001",
"Created": "2022-01-05T17:41:31.976044066Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.19.0.0/16",
"Gateway": "172.19.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"53a4b2cf9273e566c909c698ed62adf1885f43472d3dfead8e5dd21641868dae": {
"Name": "docker-compose_kafdrop_1",
"EndpointID": "bf01ffbf0f45cbf96aef34fd6cb71ea3e1d6ffa35bef53323d4ad0e2a0accc46",
"MacAddress": "02:42:ac:13:00:04",
"IPv4Address": "172.19.0.4/16",
"IPv6Address": ""
},
"ad1c34b14bc8d06ca7840f263649b36e30177cf561fb84342321f948eabafa3b": {
"Name": "broker",
"EndpointID": "c038acbdd599f1ffc3665386aa7bc6f684ba2ba12d95098badbacd9d8df8e692",
"MacAddress": "02:42:ac:13:00:03",
"IPv4Address": "172.19.0.3/16",
"IPv6Address": ""
},
"b64dece2636059905c32504ed2324455b3a721a3ea220931af53af4d3af25ff5": {
"Name": "zookeeper",
"EndpointID": "5b827fc0ef3566ced2f51650279af050c11a44ce601a991716261fac48fecc0c",
"MacAddress": "02:42:ac:13:00:02",
"IPv4Address": "172.19.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "docker-compose",
"com.docker.compose.version": "1.29.2"
}
}
]
To summarize this, I created this overview:
So, because the kafka-ui container was running on a different network than the kafka broker container, the connection failed.
You can connect a running container to a network, with the following command:
docker network connect [OPTIONS] NETWORK CONTAINER
Connects a container to a network. You can connect a container by name or by ID. Once connected, the container can communicate with other containers in the same network.
[https://docs.docker.com/engine/reference/commandline/network_connect/]
But I opted for connecting a container to a network when it starts.
[https://docs.docker.com/engine/reference/run/#network-settings]
So, I changed the content of file ui-for-apache-kafka.sh:
[in bold, I highlighted the changes]
#!/bin/bash echo "**** Begin installing UI for Apache Kafka" docker pull provectuslabs/kafka-ui docker run --name kafka-ui \ --network docker-compose_default \ -p 8080:8080 \ -e KAFKA_CLUSTERS_0_NAME=local \ -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092 \ -d provectuslabs/kafka-ui:latest docker container ls -a echo "**** End installing UI for Apache Kafka"
Remark:
I also assigned a container name with the –name option.
In order to test this new set up, I used vagrant destroy and vagrant up.
With the following output:
ubuntu_docker: **** Begin installing UI for Apache Kafka ubuntu_docker: Using default tag: latest ubuntu_docker: latest: Pulling from provectuslabs/kafka-ui ... ubuntu_docker: Status: Downloaded newer image for provectuslabs/kafka-ui:latest ubuntu_docker: docker.io/provectuslabs/kafka-ui:latest ubuntu_docker: 3d99bb38d660e261966e9d3c9a01b9f2927ba95f0c83abe2f0630635cf5e4a80 ubuntu_docker: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ubuntu_docker: 3d99bb38d660 provectuslabs/kafka-ui:latest "/bin/sh -c 'java $J…" 5 seconds ago Up Less than a second 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp kafka-ui ubuntu_docker: 08ca78c86c60 obsidiandynamics/kafdrop "/kafdrop.sh" About a minute ago Up About a minute 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp docker-compose_kafdrop_1 ubuntu_docker: 51ce7a7fd518 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" About a minute ago Up About a minute 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp broker ubuntu_docker: 4df9df3c5c9a confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" About a minute ago Up About a minute 2181/tcp, 2888/tcp, 3888/tcp zookeeper ubuntu_docker: 8350952ef978 hello-world "/hello" 5 minutes ago Exited (0) 5 minutes ago nervous_carver ubuntu_docker: **** End installing UI for Apache Kafka
I closed the Windows Command Prompt.
Creating a topic and writing and reading messages
Now, I wanted to create a topic and write messages to the topic, just to see if these were picked up by the web UI. So, I repeated some of the steps from the Quick Start.
[https://developer.confluent.io/quickstart/kafka-docker/]
UI for Apache Kafka
I used vagrant ssh to connect into the running VM. Next, I used the appropriate commands (I described in my previous article) on the Linux Command Prompt.
On my Windows laptop, in a Web Browser, I entered the URL:
The web UI started.
This time, the Kafka broker could be found.
In the left menu, I clicked on Topics.
Next, I clicked on quickstart.
Then, I clicked on tab Messages.
The 3 messages were found.
So, I finally got the result I was looking for.
Then, in order to list networks, I used the following command on the Linux Command Prompt:
docker network ls
With the following output:
NETWORK ID NAME DRIVER SCOPE a669d30f7ed4 bridge bridge local f356f74a1780 docker-compose_default bridge local ae86e253ccb2 host host local a1c22fae8af1 none null local
Then, in order to display detailed information on one or more networks (in my case the network named bridge) , I used the following command on the Linux Command Prompt:
docker network inspect bridge
With the following output:
[
{
"Name": "bridge",
"Id": "a669d30f7ed4c98758c967c71505f0f25b0df0ad1d58ef0e427d626845f2cbe4",
"Created": "2022-01-07T06:40:29.803554769Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
As you can see, there are no containers in this network.
Next, in order to display detailed information on one or more networks (in my case the network named docker-compose_default) , I used the following command on the Linux Command Prompt:
docker network inspect docker-compose_default
With the following output:
[
{
"Name": "docker-compose_default",
"Id": "f356f74a17808d8c6dd40621dcac24b124007106a99af67430bb120ecf8a8f8d",
"Created": "2022-01-07T06:41:07.231820169Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"08ca78c86c6059da3b900ee6d9d79a302acd806f1cbb534b04c0f8399605747d": {
"Name": "docker-compose_kafdrop_1",
"EndpointID": "9d5457de4eade244bc688addc51e25f45defb9eb6bbe777744297af02adf3d8e",
"MacAddress": "02:42:ac:12:00:04",
"IPv4Address": "172.18.0.4/16",
"IPv6Address": ""
},
"3d99bb38d660e261966e9d3c9a01b9f2927ba95f0c83abe2f0630635cf5e4a80": {
"Name": "kafka-ui",
"EndpointID": "d6f9ffbf88b70a6bb2bedea4cc8902cf97216685c640aeb312543e1cc4613a7d",
"MacAddress": "02:42:ac:12:00:05",
"IPv4Address": "172.18.0.5/16",
"IPv6Address": ""
},
"4df9df3c5c9ae5c507c2fc2abfbddb69ac712342cd8042cf24468f212f73c865": {
"Name": "zookeeper",
"EndpointID": "fdffbe7e25330f448392e9e05560217d8754a39ce32540b078a8b94b711bac6f",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
},
"51ce7a7fd5180f4c804bb16df6b6aa1821250fe3228fd90c3fa824728b24254a": {
"Name": "broker",
"EndpointID": "0d371fabbfc317494fcf306cf80f1523c077a03d30cf1eb8eebd9c9bed92f6e5",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "docker-compose",
"com.docker.compose.version": "1.29.2"
}
}
]
To summarize this, I created this overview:
As you can see, the kafka-ui container is now part of the same network as the other containers.
So, besides “Kafdrop”, now I also got “UI for Apache Kafka” to work on my demo environment. Although I now had two Web GUI’s for Apache Kafka on my demo environment, I decided to leave it like this for now.
In the next part of this series of articles the focus is on installing and using Apache Kafka Connect.
