Jolt is a Java library which can be used to transform JSON to JSON. A Jolt transformation specification itself is also a JSON file. You can use it in products such as Apache NiFi and Apache Camel. In this blog post I’ll describe my first experiences with Jolt transformations. For […]
Maarten Smeets
Maarten is a software architect and Oracle ACE. Over the past years he has worked for numerous customers in the Netherlands in developer, analyst and architect roles on topics like software delivery, performance, security and other integration related challenges. Maarten is passionate about his job and likes to share his knowledge through publications, frequent blogging and presentations.
Apache NiFi: JSON to SOAP
Apache NiFi is a powerful open source integration product. A challenge you might encounter when integrating systems is that one system can produce JSON messages and the other has a SOAP API available. In this blog post I’ll show how you can use NiFi to convert JSON input to a […]
Apache NiFi: Automating tasks using NiPyAPI
Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps […]
Apache NiFi: Avoid these common pitfalls
Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It has a powerful UI which can be used for both development and operations. In addition, the NiFi Registry is available to make promoting software from one environment to the next, easy. In order […]
Vagrant and Hyper-V: Don’t do it!
I’ve used Vagrant since 2015 in combination with Virtualbox for creating development machines. Recently however I’m experiencing more issues with Virtualbox. For example CPUs getting stuck when assigning multiple CPUs to a VM and issues with auto adjusting the guest resolution when resizing the VM window. These annoyances drove me […]
Merge AVRO schema and generate random data or Java classes
Previously I wrote about generating random data which conforms to an AVRO schema (here). In a recent use-case, I encountered the situation where there were several separate schema files containing different AVRO types. The message used types from those different files. For the generation of random data, I first needed […]
Java: How to validate JSON against an AVRO schema
AVRO schema are often used to serialize JSON data into a compact binary format in order to for example transport it efficiently over Kafka. When you want to validate your JSON against an AVRO schema in Java, you will encounter some challenges. The JSON which is required to allow validation […]
Generate random JSON data from an AVRO schema using Java
Recently I was designing an AVRO schema and wanted to test how data would look like which conformed to this schema. I developed some Java code to generate sample data. This of course also has uses in more elaborate tests which require generation of random events. Because AVRO is not […]
Apache NiFi: Forwarding HTTP headers
Apache NiFi can be used to expose various flavors of webservices. Using NiFi in such a way provides benefits like quick development using a GUI and of course data provenance. You know who called you with which data and where the data went. The NiFi is very scalable, delivery can […]
Apache NiFi: Reading COVID data from a REST API and producing it to a Kafka topic
Apache NiFi can be used to accelerate big data projects by allowing easy integration between various data sources. Using Apache NiFi it is easy to track what happened to your data (data provenance) and to provide features like guaranteed ordered delivery and error handling. In this example I’m going to […]
A quick and easy Apache NiFi development environment
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared […]
Product selection done quickly
Your customer has a specific challenge and wants to have it solved by means of software. They ask you as a software/solution architect to advise them on this topic and they want to be able to choose a solution based on your advise by the end of next week (we’re […]
Measure the Quality of your Source Code!
Quality is something which is hard to define. Different areas of expertise use their own definitions of what quality is. Without an objective standard which carries weight, anyone can claim to provide a quality product or service according to some standard. This makes it difficult to compare products and to […]
OBS Studio: How to record online presentations (Zoom, Teams, Webex, Meet, …)
Sometimes you want to record online presentations to maybe re-watch them later or share them with colleagues. It is not always possible to easily do this from within the tool which is used for the presentation. An easy alternative is to record such sessions using OBS Studio, a free and […]
Blog milestone: 200 blog posts, 1M page views!
This is my 200th blog post. Also I reached 1M page-views recently. In those 200 blog posts, I used approximately 325K words. An average non-fiction book is somewhere between 70K to 80K words. So that would amount to somewhere around 4 books. Why did I write so many blog posts, […]
Jenkins: Obtaining and displaying credentials
Jenkins is a solid CI/CD platform which has proven itself over the years. Many organizations use it to build, test and deploy their applications. In Jenkins it is possible to define credentials or to use an external credential store. You can then use these inside your pipelines and jobs. Direct […]
SonarCloud: OWASP Dependency-Check reports
SonarCloud is a hosted SonarQube SaaS solution which helps you with code quality management. It is free to use for open source projects. You cannot install 3rd party plugins in SonarCloud however. This puts some limitations on the kind of data you can put in SonarCloud. For Java this is […]
GitHub Actions and SonarCloud
GitHub Actions allow you to do most CI/CD tasks for free, directly from your GitHub repository. One of the challenges however is that there is no build-in facility like for example SonarQube to manage code quality. Luckily, SonarSource provides SonarCloud; a SonarQube SaaS offering which is free for public projects! […]
GitHub Actions: A first impression
I’m a regular user of GitHub. Recently I discovered GitHub also has a build-in CI/CD workflow solution called GitHub Actions. Curious about how this would work I decided to try it out. I had previously build a Jenkins Pipeline to perform several static and dynamic application security tests on a Java project […]
OWASP ZAP: A quick introduction to a versatile open source DAST tool
OWASP ZAP (Zed Attack Proxy) is an open source dynamic application security testing (DAST) tool. It is available here and has a website with documentation here. I recently encountered it when looking for open source security test tools to embed in a CI/CD pipeline (here). I was surprised by how versatile this tool […]
Java Security: Open Source tools for use in CI/CD pipelines
It is often expected of a DevOps team to also take security into consideration when delivering software. Often however, this does not get the attention it deserves. In this blog post I’ll describe some easy to use, CI/CD pipeline friendly, open source tools you can use to perform several checks […]
Java Agent: Rewrite Java code at runtime using Javassist
You might want to change the behavior of a Java application at runtime without having to alter the original sources and recompile them. This can be done by using a Java Agent. Java Agents are used by several products. Dynatrace uses for example a Java Agent to collect data from inside […]
Python: A Google Translate service using Playwright
There are a lot of use-cases in which you might want to automate web-browser actions. For example to automate tedious repetitive tasks or to perform automated tests of front-end applications. There are several tools available to do this such as Selenium, Cypress and Puppeteer. Several blog posts (such as here) […]
Jenkins Pipeline: SonarQube and the OWASP Dependency-Check
The OWASP Foundation plays an important role in helping to improve security of software worldwide. They have created a popular and well-known awareness document called the ‘OWASP Top 10‘. This document lists the following risk: using components with known vulnerabilities. Software nowadays can be quite complex consisting of many direct and […]
Surprisingly easy: Anchore Engine for container vulnerability scanning in a Jenkins pipeline running on Kubernetes
Anchore Engine is a popular open source tool for container image inspection and vulnerability scanning. It is easily integrated in a Kubernetes environment as an admission controller or in a Jenkins build pipeline using a plugin. A while ago I took a look at Anchore Engine and created a small introductory presentation and Katacoda scenario for […]
Jenkins: Building Java and deploying to Kubernetes
Kubernetes is a popular platform to run and manage containerized applications. A CI/CD solution is often needed but not always provided. You might need to set this up for yourself. In this blog post I’ll provide a minimal end-to-end solution for Java applications. This starts with a commit in source […]
OpenEBS: cStor storage engine on KVM
OpenEBS provides a Kubernetes native distributed storage solution which is friendly on developers and administrators. It is completely open source and part of the CNCF. Previously I wrote about installing and using OpenEBS, Jiva storage engine, on the Charmed Kubernetes distribution of Canonical. The Jiva storage class uses storage inside managed […]
Production-like Kubernetes on your laptop. Kubespray on KVM
There are various options to install a production-like Kubernetes distribution on your laptop. Previously I tried out using the Canonical stack (Juju, MAAS, Charmed Kubernetes) for this. This worked nicely but it gave me the feeling that it was a bit Canonical specific and it felt a bit heavy on resources at […]
Anchore Engine: Container image vulnerability scanning
Applications nowadays, are usually deployed inside containers. A container consists of libraries and tools which allow the application to run inside. Since there can be exploitable vulnerabilities, it is not only important to take security into account for your application, but also for the container it runs in. There are […]
OBS Studio + Snap Camera: Putting yourself in your presentation live for free!
When giving online presentations, it helps for personal marketing if people can see you on the screen. Various tools provide features which help you achieve that, for example Microsoft Teams. Sometimes though you do not have that available or want to be able to do more than what the tool […]
OpenEBS: Create persistent storage in your Charmed Kubernetes cluster quick and easy!
As a developer I wanted to experiment with Kubernetes environments which approximate production deployments. In order to do that I wanted a distributed storage solution and chose OpenEBS. Mainly because it was easy to get started and quick to get up and running. In this blog post I’ll describe how […]
Charmed Kubernetes on KVM using MAAS and Juju
Coming to this solution was a journey. I was looking for a Kubernetes installation which was easy to deploy and cleanup on my own laptop (I didn’t want to have to pay for a hosted solution). I did want a solution which was more or less production-like because I wanted […]
Quick and easy: A multi-node Kubernetes cluster on CentOS 7 + QEMU/KVM (libvirt)
Kubernetes is a popular container orchestration platform. As a developer understanding the environment in which your application is going to run is important since this can help you use available services of the platform and fix issues. There are several options to run Kubernetes locally to get some experience with […]
Spring: Blocking vs non-blocking: R2DBC vs JDBC and WebFlux vs Web MVC
Spring Framework version 5, released in Sept 2017, introduced Spring WebFlux. A fully reactive stack. In Dec 2019 Spring Data R2DBC was released, an incubator to integrate relational databases using a reactive driver. In this blog post I’ll show that at high concurrency, WebFlux and R2DBC perform better. They have […]
The size of Docker images containing OpenJDK 11.0.6
When running Java applications in containers, you need to be careful with your resources. If you’re not careful with layering your images (for example using Google’s Jib), you can quickly get into disk-space issues, especially when your base image and/or application gets updated regularly. One of the ways you can save […]
Performance of relational database drivers. R2DBC vs JDBC
R2DBC provides non-blocking reactive APIs to relational database programmers in Java. It is an open specification, similar to JDBC. JDBC however uses a thread per connection while R2DBC can handle more connections using less threads (and thus potentially use less memory). This could also mean threads are available to do other […]
Secure browsing using a local SOCKS proxy server (on desktop or mobile) and an always free OCI compute instance as SSH server
Oracle provides several services as ‘always free’. In contrast to Azure and Amazon, these include compute instances which remain ‘forever’ free to use. Although there are some limitations on CPU, disk, network resources, these instances are ideal to use as a remote SSH server and with a little effort a […]
HTTP benchmarking using wrk. Parsing output to CSV or JSON using Python
wrk is a modern HTTP benchmarking tool. Using a simple CLI interface you can put simple load on HTTP services and determine latency, response times and the number of successfully processed requests. It has a LuaJIT scripting interface which provides extensibility. A distinguishing feature of wrk compared to for example ab […]
pgAdmin in Docker: Provisioning connections and passwords
pgAdmin is a popular open source and feature rich administration and development platform for PostgreSQL. When provisioning Postgres database environments using containers, it is not unusual to also provision a pgAdmin container. The pgAdmin image provided on Docker Hub does not contain any server connection details. When your pgAdmin container […]
Apache Camel + Spring Boot: Different components to expose HTTP endpoints
Apache Camel is an open source integration framework that allows you to integrate technologically diverse systems using a large library of components. A common use-case is to service HTTP based endpoints. Those of course come in several flavors and there is quite a choice in components to use. In this blog post […]