In our daily work we are dealing with data from many sources. Data in CSV files, from Cloud APIs, in mail servers, configuration files, Terraform plans, in logging systems, source code repositories and many more. Different formats, access methods, tools. And retrieving data from one such source can be challenging […]
Linux background process and redirecting the standard input, output and error stream
Often, I use a Linux shell script that executes several commands. Sometimes you don’t want to wait for a command to end and you want the next command to immediately run after the previous one is started. In my previous article I mentioned such a situation, with regard to preparing […]
Apache NiFi: Importing and exporting parameters
When you import a new process group or upgrade an existing one, missing parameters contexts and parameters will automatically be added. The new parameters will be filled with values from the environment where the process group was committed to the Registry (except sensitive parameter values). This is usually a development […]
Creating a minimal container image for a Go application
How hard can it be – to create the smallest possible container image to run a Go application? It is not hard to create a container image that contains an binary executable file – and that is what a Go application turns into during the build process. However, that does […]
Apache NiFi: Having fun with Jolt transformations
Jolt is a Java library which can be used to transform JSON to JSON. A Jolt transformation specification itself is also a JSON file. You can use it in products such as Apache NiFi and Apache Camel. In this blog post I’ll describe my first experiences with Jolt transformations. For […]
Apache NiFi: JSON to SOAP
Apache NiFi is a powerful open source integration product. A challenge you might encounter when integrating systems is that one system can produce JSON messages and the other has a SOAP API available. In this blog post I’ll show how you can use NiFi to convert JSON input to a […]
Sorting an Angular Material table – how to use MatSort and sortingDataAccessors
Angular Material provides the MatSort directive as an easy way the include interactive sorting in the tables of your application. In this blogpost I will show the best way to add sorting to your Material table. In addition, I will show how to use custom sortingDataAccessors to sort on nested properties and case insensitively, and how to combine different sorting requirements.
Apache NiFi: Automating tasks using NiPyAPI
Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps […]
WebSinker–generic event detection for unhooked systems to trigger WebHook
Quite a few systems – business applications large and small – do not have support for WebHooks. That is: they do not offer the ability to register a URL as HTTP endpoint (WebHook) that the system will send requests to that inform the endpoint about relevant events. However, there are […]
AMIS DataSafe, the answer to Black Swan scenarios
How to be prepared for a black swan scenario?
Brief: What is Oracle Wallet?
An Oracle Wallet is an encrypted file that contains database credentials – username and password – for connecting to an Oracle Database. This avoids writing hard code user passwords in a shell script or in an application database configuration file. A wallet can also contain certificates – for example for […]
Apache NiFi: Avoid these common pitfalls
Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It has a powerful UI which can be used for both development and operations. In addition, the NiFi Registry is available to make promoting software from one environment to the next, easy. In order […]
Adding Grafana to my VM with Minikube and Elasticsearch
For a demo, I needed an environment with Grafana. Lucky for me, I had the configuration for such an environment using Vagrant and Oracle VirtualBox. In the past, I already set up such a demo environment, available within an Oracle VirtualBox appliance, as I described in a series of articles. […]
Marrying Fn Functions to Dapr.io–leverage the power of the proxy
Project Fn provides a framework for creating and running serverless functions. It is the foundation for the Functions service on Oracle Cloud Infrastructure. Dapr.io is an open source project that provides a powerful personal assistant for any application and a distributed application runtime that especially shines with microservices. Some aspects […]
Query Past Generations in Oracle Database SQL – PI Day Special
Today is PI Day. March 14 or: 3-14 in the US. Of course it can be March 14th 2015 – 3-14-15 – only once but for some reason March 14 is baptized PI Day and is not unlike April Fool’s day the international day for nerdi stuff. You may want […]
How to scale and smarten up a real-life energy grid with Azure IoT
In this episode of the IoT Show, we discuss the IoT smart grid project that AMIS Conclusion has created for a customer in The Netherlands. We talk about a real-life large-scale energy grid and all the challenges we face implementing this on an Enterprise scale. Robbrecht van Amerongen will talk […]
Vagrant and Hyper-V: Don’t do it!
I’ve used Vagrant since 2015 in combination with Virtualbox for creating development machines. Recently however I’m experiencing more issues with Virtualbox. For example CPUs getting stuck when assigning multiple CPUs to a VM and issues with auto adjusting the guest resolution when resizing the VM window. These annoyances drove me […]
Trouble shooting while upgrading a VM with Minikube, Elasticsearch and Kibana (using the –vm-driver=none option) –on my Windows laptop using Vagrant and Oracle VirtualBox
For a demo, I needed an environment including Elasticsearch and Kibana (Elastic Stack). Lucky for me, I had the configuration for such an environment using Vagrant and Oracle VirtualBox. In the past, I already set up such a demo environment, available within an Oracle VirtualBox appliance, as I described in […]
Merge AVRO schema and generate random data or Java classes
Previously I wrote about generating random data which conforms to an AVRO schema (here). In a recent use-case, I encountered the situation where there were several separate schema files containing different AVRO types. The message used types from those different files. For the generation of random data, I first needed […]
Java: How to validate JSON against an AVRO schema
AVRO schema are often used to serialize JSON data into a compact binary format in order to for example transport it efficiently over Kafka. When you want to validate your JSON against an AVRO schema in Java, you will encounter some challenges. The JSON which is required to allow validation […]
Generate random JSON data from an AVRO schema using Java
Recently I was designing an AVRO schema and wanted to test how data would look like which conformed to this schema. I developed some Java code to generate sample data. This of course also has uses in more elaborate tests which require generation of random events. Because AVRO is not […]
Update Go on Linux
It turned out to be a fairly small deal. For reference sake a brief write up of updating the Go runtime on my Ubuntu environment. Current version: go version Find latest releases of Go at https://go.dev/dl/ There is a new one. I copy the link and then run: wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz […]
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 2)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE. [https://kafka.apache.org/] In a series of articles, I will take you with me on my path to set […]
Connecting Go application to Oracle Database–On Prem and Autonomous , with and without Oracle Client libraries
I have been struggling a bit with this one: how to connect from a Go application to an Oracle Database – both a locally running database as well as an Autonomous Database Instance running on Oracle Cloud. Ideally, my application does not require Oracle Client Libraries to be installed in […]
Apache NiFi: Forwarding HTTP headers
Apache NiFi can be used to expose various flavors of webservices. Using NiFi in such a way provides benefits like quick development using a GUI and of course data provenance. You know who called you with which data and where the data went. The NiFi is very scalable, delivery can […]
Dapr and Kafka–easy binding
Dapr is a runtime framework for distributed applications. Dapr also provides a personal assistant to any application and environment that offers standardized access to different facilities (state, secrets, configuration, pub/sub, actors) and technologies. Simply by using Dapr we can leverage a large number of Dapr components that through the Dapr […]
Apache Kafka, setting up a demo environment using Vagrant and Oracle VirtualBox (part 1)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE. [https://kafka.apache.org/] In my previous article, I described the steps I took, to set up a demo environment […]
Quick Start Compute Instance on OCI (for software development)
I have worked quite a bit in Oracle Cloud, starting back in 2015. I have used many different OCI services – such as Integration, Streaming, Functions, OKE, API Gateway, Object Storage, DBaaS and several more. However, for some reason I have almost complete avoided the almost simplest service of all: […]
Oracle Cloud Go SDK and Instance Principal based Authentication
In a recent article, I described the use of the OCI SDK for Go(Lang) – in my example to interact with the ObjectStorage service. In the article, I used authentication based on tenancy and user ocid, fingerprint and private key. However, when the Go application is running in an OCI […]
Did I really just commit and push a file with a password in it?–Remove all traces of pushed Git commit
Surely I would never be so stupid as to commit a file that contains sensitive information – say my Oracle Cloud account’s private key. And even if I did, I would not push that change to a public git repo on GitHub, would I? It turns out that I did […]
Oracle Database 21c XE, setting up a demo environment using Vagrant and Oracle VirtualBox (part 4)
I wanted to set up a demo environment with Apache Kafka (an open-source distributed event streaming platform) and an Oracle Database, all within containers. For my purpose I opted for Oracle Database XE. [https://kafka.apache.org/] As I did before, on my Windows laptop, I wanted to use a demo environment within […]
Submitting my Custom Dapr State Component as a formal Open Source Contribution–work in progress
I have used many open source products in my professional life. I have not studied the sources of many of these products and projects. My main contributions to the open source community consisted of presentations (many dozens) and articles (many 100s) in which I have spread the word. I have […]
Composing DAPR Custom State Store Component for Oracle Cloud Object Storage
Dapr is a runtime framework for distributed applications as well as a personal assistant with many generic qualities relevant to any application. Through Dapr, applications can easily benefit from configuration and secret management, from interacting with dozens of external technologies in both inbound and outbound direction, from having messages received […]
Apache NiFi: Reading COVID data from a REST API and producing it to a Kafka topic
Apache NiFi can be used to accelerate big data projects by allowing easy integration between various data sources. Using Apache NiFi it is easy to track what happened to your data (data provenance) and to provide features like guaranteed ordered delivery and error handling. In this example I’m going to […]
Personal link shortener in AWS
I like to do presentations about a lot of topics. Most of these presentations are recorded and the link to that recording is then shared to people who couldn’t attend the presentation live. Up to now I used bitly.com to shorten my URLs. Last time that I used bitly I […]
Interacting with OCI Object Storage using Go SDK (first steps)
I am still fairly new at working with Go. I have the ambition of working with and perhaps even on several open source projects that all use Go as their implementation language. It seems like the number of open source projects that use Go is increasing [rapidly]. It seems that […]
A quick and easy Apache NiFi development environment
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared […]
Extending Local Dapr Development Environment for Custom Components
In a previous article I described how I prepared my local development environment from working with the source code of the Dapr project – locally editing, building and running customization on Dapr. This is a prerequisite for creating custom Dapr components – as is my true intention. In this article […]
Customizing Dapr–preparing my environment
In other articles I will write about the value of Dapr – as a personal assistant to any application that helps out with state, configuration and secrets, receiving inbound messages, publishing messages and invoking services and that supports microservice architectures through a network of closely collaborating personal assistants to all […]
Dapr on OKE–Distributed Application Runtime on OCI Kubernetes Engine
One of the most exciting frameworks I have ran into into recent years is Dapr (dapr.io) – distributed application runtime. Dapr is a cross technology runtime framework that supports applications in many and The Dapr docs describe how to install Dapr locally and on a Kubernetes cluster. The documentation explicitly […]