‘Kafka is dead’ is a statement that is circulating more and more often. Critics point to a lack of innovation, the complexity of the product and the emergence of alternatives: But is that justified? According to experts, Kafka continues to play an essential role, especially as part of larger platforms and cloud solutions. We spoke to Patrick Stevens (CTO at AMIS Conclusion) and Maarten Smeets (Software Architect at AMIS Conclusion) about their vision of Kafka. Why do they think this technology is still indispensable in modern IT architectures?
What is Apache Kafka?
Apache Kafka is an open-source platform developed by LinkedIn in 2011 and made available through the Apache Software Foundation. Patrick: “Simply put, Kafka is a very large and fast post office. Messages are collected, sorted and prepared for systems that use this data.” Maarten adds: “Kafka is at its core a powerful, but simple product. It transports data quickly and reliably but has no knowledge of the content and offers few management options out of the box.”
Thanks to its simplicity, speed and robustness, Kafka has become a global standard for real-time data processing. Its open-source nature makes integration with larger data platforms and cloud solutions easy, making the product particularly relevant for organizations that rely on real-time data.
Kafka is not dead, but it lacks a number of functionalities
A common argument against Kafka is that it has hardly been renewed since 2015. But Patrick sees this differently. He says: “Kafka’s basic functionality is robust and reliable. The fact that it does not need to be renewed proves just how robust the product is.” Where Kafka lacks certain functionalities in its basic form, it is often integrated into larger platforms that complement these shortcomings. Patrick explains: “Many organizations choose to use Kafka within platforms that add extra tools and services. This simplifies management and provides additional functionalities that are essential in complex IT landscapes.” Maarten adds: “Think of management interfaces, adapters for integrations, message validation and security features. These are often offered by platforms such as Confluent or Axual, but also large cloud providers such as Oracle, Microsoft and AWS have built their own shell around products that have Kafka under the hood or are inspired by it.”
Cloud providers and Kafka-like solutions
Large cloud providers offer their own event streaming services that support Kafka. Patrick explains, “Services such as Azure Event Hub, AWS Managed Streaming for Apache Kafka (MSK), and Google Cloud Managed Service for Apache Kafka support Kafka APIs and build on the principles that Kafka introduced.” Maarten emphasizes the advantage: “These services make it easier to work with familiar tools and methodologies. At the same time, you benefit from the scalability, ease of use, and managed infrastructure that cloud providers offer. These integrated solutions allow you to harness the power of Kafka without the complexity of self-management.”
Why Kafka as part of a platform?
By using Kafka as part of a more comprehensive platform or a cloud-based service, organizations benefit from Kafka’s powerful event streaming capabilities, complemented by additional tools that simplify management and integration. Patrick says: “You don’t have to develop or integrate all the missing parts yourself. These platforms offer a complete package that is immediately suitable for enterprise use.” Maarten adds: “This not only allows for faster implementation, but also reduces operational complexity. Moreover, these platforms ensure that you can meet strict requirements around security and compliance.”
In practice: Kafka in combination with cloud solutions
A good example of the power of Kafka in combination with cloud solutions is the use of Kafka-like services in complex, data-rich environments. Kafka is ideal for applications where real-time data processing is essential, such as in Internet of Things (IoT), real-time financial data, and logistics processes. This type of solution is mainly used by large organizations with high data volumes and complex integrations.
An illustrative example from AMIS Conclusion’s customer portfolio is the project at Eneco. Maarten explains: “At Eneco, we use Kafka to share real-time data from all wind and solar farms uniformly with the systems for control and energy trading. Kafka offers simplicity, flexibility and can process huge amounts of data with low latency. In this case, it is more than 30 million messages per day.” Many organizations also opt for managed services such as Azure Event Hubs or AWS Managed Streaming for Apache Kafka (MSK). Maarten continues: “These services offer Kafka compatibility, so that existing applications and knowledge can be reused. At the same time, cloud providers take care of the management and infrastructure. This approach avoids complex, loose connections between all kinds of applications and offers a scalable solution with simplicity, flexibility and high performance in complex IT landscapes.”
Is Kafka the right choice for every organization?
Although Kafka is a strong technology, it is not always the best choice. Patrick explains: “For small businesses or relatively simple integrations, Kafka can be too complex and heavy. In such cases, it is better to opt for a simpler alternative, such as a Pub/Sub model.” However, for large organizations that work with large amounts of data and many complicated systems, Kafka is an essential tool.” Maarten emphasizes: “When it comes to complex integrations and high demands in terms of scalability and reliability, Kafka is really indispensable.”
Kafka’s influence on modern IT architectures
Does the statement that Kafka is dead have any truth to it? Patrick is adamant: “Absolutely not. Kafka’s principles and APIs are deeply woven into the architecture of modern event-driven systems. Even if organizations don’t use the original Kafka software, they often work with tools and services that are inspired by or compatible with Kafka.” Maarten agrees and emphasizes the fundamental nature of Kafka: “It’s not about Kafka having to provide everything itself, but that it forms a foundation on which many other solutions are built.”
Kafka lives, and how
Kafka may no longer be the only product deployed for event streaming, but it’s still at the heart of many modern data architectures. By integrating it into platforms and leveraging cloud-based services with Kafka compatibility, organizations can transcend the limitations of the commodity. This results in complete, managed solutions that are easy to implement. Patrick and Maarten summarize it aptly: “Kafka is not dead. However, it is increasingly hidden in packaged services from cloud providers. It remains an essential technology with a lasting impact on the way we interact with data. That is precisely why it is more relevant than ever.”
