Kafka is not dead, it’s just in a different packaging
‘Kafka is dead’ is a statement that is circulating more and more often. Critics point to a lack of innovation, the complexity of the product and the emergence of …
Tag: kafka
Platys – generate a customized container powered Data Platform environment
Platys is a tool created by Guido Schmutz, architect and data specialist at Accenture and frequent teacher at various universities. In order to be able to quickly create an …
No Install, free polyglot Apache Kafka environment in ephemeral development environment with Node, Java, Python and Go
Click on a button- or this link. Wait about four to five minutes. You will have access to a free cloud based environment with a three-broker-node Apache Kafka Cluster, …
Generate random JSON data from an AVRO schema using Java
Recently I was designing an AVRO schema and wanted to test how data would look like which conformed to this schema. I developed some Java code to generate sample …
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 2)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted …
Dapr and Kafka–easy binding
Dapr is a runtime framework for distributed applications. Dapr also provides a personal assistant to any application and environment that offers standardized access to different facilities (state, secrets, configuration, …
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 1)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted …
Apache NiFi: Reading COVID data from a REST API and producing it to a Kafka topic
Apache NiFi can be used to accelerate big data projects by allowing easy integration between various data sources. Using Apache NiFi it is easy to track what happened to …
A quick and easy Apache NiFi development environment
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. …
DIY Kafka Topic Watcher tool – Node, Express, Server Sent Events and Apache Kafka
This article can be read in at least two different ways: as a somewhat lengthy introduction of a handy tool that you can easily run to inspect messages published …
A Free Apache Kafka Cloud Service – and how to quickly get started with it
Last week I presented on Apache Kafka – twice. Once to a group of over 100 students, once to 30+ colleagues. In both instances, I invited attendees to partake …
Oracle Cloud Streaming Service – Scalable, Reliable, Kafka-like Event service on OCI
Oracle Cloud Infrastructure offers the Streaming service. This service is easily described as a Kafka-like fully managed event platform for high volume streams of data that can be processed …
Running Apache Kafka on Minikube
I frequently want to demonstrate the working of Apache Kafka and/or provide colleagues and other audiences with an environment for a workshop that involves Apache Kafka. I have been …
Filesystem events to Elasticsearch / Kibana through Kafka Connect / Kafka
Filesystem events are useful to monitor. They can indicate a security breach. They can also help understanding how a complex system works by looking at the files it reads …
CQRS through Twitter – more fun than meaning
As part of my 50 Shades of Data presentation, I talk about CQRS – Command Query Responsibility Segregation. The idea that you can have one or more read only …
Querying and Publishing Kafka Events from Oracle Database SQL and PL/SQL
One of the session at CodeOne 2018 discussed an upcoming feature for Oracle Database – supported in Release 12.2 and up – that would allow developers to consume Kafka …
Running Kafka, KSQL and the Confluent Open Source Platform 4.x using Docker Compose on a Windows machine
For conducting some experiments and preparing several demonstrations I needed a locally running Kafka Cluster (of a recent release) in combination with a KSQL server instance. Additional components from …
Simple CQRS – Tweets to Apache Kafka to Elastic Search Index using a little Node code
Put simply – CQRS (Command Query Responsibility Segregation) is an architecture pattern that recognizes the fact that it may be wise to separate the database that processes data manipulations …
Get going with KSQL on Kubernetes
This article describes how to quickly get going with KSQL on Kubernetes. KSQL is Confuent’s ‘continuous streaming query language’. It allows us to write SQL-like queries that operate on …
15 Minutes to get a Kafka Cluster running on Kubernetes – and start producing and consuming from a Node application
For workshop I will present on microservices and communication patterns I need attendees to have their own local Kafka Cluster. I have found a way to have them up …
Microservices and Updating Data Bound Context on Oracle Cloud with Application Container and Event Hub (plus DBaaS and MongoDB)–Part Two
This article describes – in two installments – how events are used to communicate a change in a data record owned by the Customer microservice to consumers such as …
Microservices and Updating Data Bound Context on Oracle Cloud with Application Container and Event Hub (plus DBaaS and MongoDB)–Part One
This article describes – in two installments – how events are used to communicate a change in a data record owned by the Customer microservice to consumers such as …
Prepare and link bind Oracle Database Cloud, Application Container Cloud, Application Container Cache and Event Hub
A fairly common combination of Oracle Public Cloud services that I use together – for example for the implementation of microservices – is DBaaS, Application Container Cloud, Application Cache …
Recording, slides and code resources for Oracle Code session: Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
On YouTube, you will find the 44 min recording of my talk on Oracle Code San Francisco (October 2017): https://www.youtube.com/watch?v=khP3yHmbtpM The slides are on SlideShare: Real Time UI with …
Setting up Oracle Event Hub (Apache Kafka) Cloud Service and Pub & Sub from local Node Kafka client
Oracle offers an Event Bus Cloud Service – an enterprise grade Apache Kafka instance – with large numbers of partitions and topics, (retained) messages and distributed nodes. Setting up …
Materials for Workshop Microservices Choreography with Kubernetes, Docker, Kafka and Node.JS
Thursday June 1st – yet another community event at AMIS. This one dedicated to Microservices. What are microservices, why do we think they are interesting? How are they different? …
Apache Kafka on the Oracle Cloud: My First experiences with Oracle Event Hub Cloud Service
Oracle recently made their ‘Kafka on Cloud’ service available: the Event Hub cloud service – offered as part of the Big Data Compute Cloud Service. In this article, I …
Node.js application writing to MongoDB – Kafka Streams findings read from Kafka Topic written to MongoDB from Node
MongoDB is a popular, light weight, highly scalable, very fast and easy to use NoSQL document database. Written in C++, working with JSON documents (stored in binary format BSON), …
Node.js application using SSE (Server Sent Events) to push updates (read from Kafka Topic) to simple HTML client application
This article describes a simple Node.js application that uses Server Sent Events (SSE) technology to push updates to a simple HTML client, served through the Express framework. The updates …
Kafka Streams and NodeJS – Consuming and periodically reporting in Node.JS on the results from a Kafka Streams streaming analytics application
In several previous articles on Apache Kafka, Kafka Streams and Node.JS for interacting with Apache Kafka, I have described how to create a Node.JS application that publishes messages to …
Apache Kafka Streams – Running Top-N Aggregation grouped by Dimension – from and to Kafka Topic
This article explains how to implement a streaming analytics application using Kafka Streams that performs a running Top N analysis on a Kafka Topic and produces the results to …
Getting Started with Kafka Streams – building a streaming analytics Java application against a Kafka Topic
Kafka Streams is a light weight Java library for creating advanced streaming applications on top of Apache Kafka Topics. Kafka Streams provides easy to use constructs that allow quick …
Workshop Apache Kafka – presentation and hands on labs for getting started
The AMIS SIG session on Apache Kafka (9th February 2017) took 25 participants by the hand on a tour of Apache Kafka. Through presentations, demonstrations and a hands-on workshop, …
NodeJS – Publish messages to Apache Kafka Topic with random delays to generate sample events based on records in CSV file
In a recent article I described how to implement a simple Node.JS program that reads and processes records from a delimiter separated file. That is stepping stone on the …
Oracle Service Bus: Produce messages to a Kafka topic
Oracle Service Bus is a powerful tool to provide features like transformation, throttling, virtualization of messages coming from different sources. There is a (recently opensourced!) Kafka transport available for …
Oracle Service Bus: A quickstart for the Kafka transport
As mentioned on the following blog post by Lucas Jellema, Kafka is going to play a part in several Oracle products. For some usecases it might eventually even replace …
Talk of the Town at Oracle OpenWorld 2016: Machine Learning & Predictive Analytics
Talk of the town during Oracle OpenWorld 2016 definitely included the term Machine Learning. Machine learning was mentioned in every other session it seemed. Sometimes fully justified – and …
Oracle Public Cloud and Kafka – Events powering the cloud – The Oracle PaaS Cloud Event BusHub
Events are quite relevant in modern computer architecture. At various levels of the stack. Events can represent various things – from IoT based measurements and logistical updates to web …