AVRO schema are often used to serialize JSON data into a compact binary format in order to for example transport it efficiently over Kafka. When you want to validate your JSON against an AVRO schema in Java, you will encounter some challenges. The JSON which is required to allow validation […]
Kafka
Generate random JSON data from an AVRO schema using Java
Recently I was designing an AVRO schema and wanted to test how data would look like which conformed to this schema. I developed some Java code to generate sample data. This of course also has uses in more elaborate tests which require generation of random events. Because AVRO is not […]
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 2)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE. [https://kafka.apache.org/] In a series of articles, I will take you with me on my path to set […]
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 1)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE. [https://kafka.apache.org/] In my previous article, I described the steps I took, to set up a demo environment […]
A quick and easy Apache NiFi development environment
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared […]
A Free Apache Kafka Cloud Service – and how to quickly get started with it
Last week I presented on Apache Kafka – twice. Once to a group of over 100 students, once to 30+ colleagues. In both instances, I invited attendees to partake in a workshop with hands-on labs to get acquainted with Apache Kafka. I had prepared a Docker Compose based Kafka platform […]
Filesystem events to Elasticsearch / Kibana through Kafka Connect / Kafka
Filesystem events are useful to monitor. They can indicate a security breach. They can also help understanding how a complex system works by looking at the files it reads and writes. When monitoring events, you can expect a lot of data to be generated quickly. The events might be interesting […]
Running Kafka, KSQL and the Confluent Open Source Platform 4.x using Docker Compose on a Windows machine
For conducting some experiments and preparing several demonstrations I needed a locally running Kafka Cluster (of a recent release) in combination with a KSQL server instance. Additional components from the Core Kafka Project and the Confluent Open Source Platform (release 4.1) would be convenient to have. I needed everything to […]
Simple CQRS – Tweets to Apache Kafka to Elastic Search Index using a little Node code
Put simply – CQRS (Command Query Responsibility Segregation) is an architecture pattern that recognizes the fact that it may be wise to separate the database that processes data manipulations from the engines that handle queries. When data retrieval requires special formats, scale, availability, TCO, location, search options and response times, […]
Get going with KSQL on Kubernetes
This article describes how to quickly get going with KSQL on Kubernetes. KSQL is Confuent’s ‘continuous streaming query language’. It allows us to write SQL-like queries that operate on Kafka Topics. Queries that join, filter and aggregate – for each event that gets produced and over time windows. Results are […]
15 Minutes to get a Kafka Cluster running on Kubernetes – and start producing and consuming from a Node application
For workshop I will present on microservices and communication patterns I need attendees to have their own local Kafka Cluster. I have found a way to have them up and running in virtually no time at all. Thanks to the combination of: Kubernetes Minikube The Yolean/kubernetes-kafka GitHub Repo with Kubernetes […]
Recording, slides and code resources for Oracle Code session: Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
On YouTube, you will find the 44 min recording of my talk on Oracle Code San Francisco (October 2017): https://www.youtube.com/watch?v=khP3yHmbtpM The slides are on SlideShare: Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push (OracleCode San Francisco 2017) from Lucas Jellema All code sources are […]
Kafka Streams and NodeJS – Consuming and periodically reporting in Node.JS on the results from a Kafka Streams streaming analytics application
In several previous articles on Apache Kafka, Kafka Streams and Node.JS for interacting with Apache Kafka, I have described how to create a Node.JS application that publishes messages to a Kafka Topic (based on entries in a CSV file), how to create a simple Kafka Streams Java application that processes […]
Oracle Service Bus: Produce messages to a Kafka topic
Oracle Service Bus is a powerful tool to provide features like transformation, throttling, virtualization of messages coming from different sources. There is a (recently opensourced!) Kafka transport available for Oracle Service Bus (see here). Oracle Service Bus can thus be used to do all kinds of interesting things to messages […]