Apache NiFi: JSON to SOAP

Apache NiFi is a powerful open source integration product. A challenge you might encounter when integrating systems is that one system can produce JSON messages and the other has a SOAP API available. In this blog post I’ll show how you can use NiFi to convert JSON input to a SOAP service call. This involves abstracting an AVRO schema for the JSON, converting it to XML and transforming the XML to a SOAP message.

In this example I’m using several publicly available websites. You should of course be careful with this. Do not copy/paste sensitive XML or JSON on these sites!

Input

I’ve used the following API to develop this example: https://api.covid19api.com/summary. It will give me a JSON like:

Abstract an AVRO schema from a JSON sample

I’ve used the following site to generate an AVRO schema from the JSON sample: http://www.dataedu.ca/avro. This site has been created by Iraj Hedayati.

This created an AVRO schema like:

If you’re going to fetch data from a Kafka topic which already uses an AVRO schema, you don’t have to create your own AVRO schema since then it is already provided. The schema is required by the NiFi ConvertRecord processor, XMLRecordSetWriter controller service, to be able to generate XML from JSON.

Generate an XSD schema from XML

The ConvertRecord processor in Apache NiFi generates an XML file like;

I used the following site (here) to generate an XSD for this XML. Why would I need to create an XSD? Having an XSD makes it easier to create an XSLT file. The resulting XSD looked like:

Create a transformation from XML to SOAP

Next I manually created a target XSD based on a general SOAP definition. I took a sample from here. I generated an XSD for the SOAP message (similar as done before) and manually merged it with the previously generated schema definition. This target XSD looked like:

In order to create a transformation from the source to the target, I’ve used Altova MapForce. There are of course other solutions available such as Oxygen XML Developer. When creating XSLTs, I recommend to use a tool which has a GUI and which is also a bit intelligent because it will save you work.

The resulting XSLT looked like:

NiFi

When the AVRO schema and XSLT file are ready, the NiFi work is relatively straightforward.

Some things to notice in the configuration:

For ConvertRecord you have to specify a Record Reader and a Record Writer

In the JsonTreeReader, a controller service, I have specified the previously generated AVRO schema;

The XMLRecordSetWriter does not require any specific configuration.

For TransformXML I’ve used a SimpleKeyValueLookupService to store my XSLT file;

Validating the result

In order to validate my request, in absence of an actual SOAP service I used the following in my docker-compose.yml file (see the base file here);

  echo:
    image: mendhak/http-https-echo:23
    environment:
      - HTTP_PORT=8888
      - HTTPS_PORT=9999
    ports:
        - "8888:8888"
        - "9999:9999"

This is an echo service which lets me view the request which is done in the docker logs. This way I can see the request and reply in NiFi and the request in the logging. The endpoint to call from NiFi in this case is http://echo:8888

Of course this is not truly SOAP yet. You need to add a SOAPAction HTTP header (amongst other things). This is easy by adding a dynamic attribute on the InvokeHTTP processor.

This header will then be set when calling the SOAP service. The below screenshot shows the reply from the echo service. This indicates the header has been set.