SOA Suite 12c: Native Format Translation in Service Bus Pipeline – XML to and from JSON, CSV and fixed position formats

Data offered as internet resource in a structured XML or non-XML format (such as JSON, CSV or fixed position) – cannot be retrieved by the File Adapter, but it can be read by the Service Bus. Using a combination of an HTTP transport style Business Service, a Service Callout and a Native XSD mapping, it turns out to be fairly easy to implement what is best described as a remote file adapter. This article describes the implementation of a findCountries operation that reads a comma separated values file with countries, accessed with a simple HTTP GET request to the website OurAirports.com. It leverages the new Native NXSD translation feature.

The Service Bus deals with XML based data. That is really all it is good at. Data that is not XML based is not structured in the eyes of the Service Bus. That is okay, as long as you do not need to interpret or process that data as part of the service implementation. Sometimes however, such non-XML based data should be processed by the Service Bus, and should therefore be treated as structured. Service Bus in SOA Suite 12c has a valuable addition that provides the way to address this challenge: the NXSD schema editor and the associated Native Translation activity in Service Bus Pipelines.

The facility that you probably know from the File and FTP adapter that allows you to sample a file in some non-XML format to define a mapping from that format to XML (for inbound) or from XML to that format (for outbound) has been extended and generalized. Outside the context of the File and FTP adapter wizard, we can create an NXSD schema (also for JSON and MFL) for structured, non-XML format data. We can use a file that contains a sample of the data and run the same wizard as for the File Adapter. At runtime, the data we want to have converted does not live in a file. It lives in a Service Bus variable – retrieved for example from the request message or perhaps from the response from a business service. In the pipeline, we can use the new NXSD Translation activity. This activity takes a variable with the data to be translated and an NXSD schema and it produces the resulting XML data – stored in another pipeline variable. Or, vice versa, it takes XML and the NXSD schema and produces the non-XML string (in for example JSON or CSV format).

In this article, I will demonstrate how this facility can be used to implement something I have called “The Remote File Adapter”. We will read an internet resource – a file somewhere on the internet – in CSV format (could also have been a data stream in JSON format for example). The data from this remote resource is subsequently processed in the same way a local file would have been.

Create Business Service to retrieve file contents

Create a new Service Bus Application with Service Bus Project, for example the CountryService project.

Create a business service OurAirportsCountriesBS, with transport set to http.

On the second page of the wizard, pick the option Messaging with both Request and Response set to Text as the content type.

On the third page of the wizard, set the Endpoint URI for this business service to the URL at which the comma separated countries file can be accessed: http://www.ourairports.com/data/countries.csv.

Press Finish to close the wizard and complete the creation of the business service.

Create the Native XML Schema definition

The SOA Suite has the capability to come up with an XML representation of data in all kinds for formats that are not XML. Native XML Schema Definitions – indicated with NXSD – are created that describe both the XML structure that best represents the native data format and provide mapping instructions on how to convert from the native format to that XML structure and vice versa. This capability is used in various adapters – as we already have seen with the File Adapter and the REST Adapter – and can be used in Service Pipelines as well (and also in Mediator and BPEL components in SOA Composites).

In our current implementation, we need to convert the countries.csv file into an XML structure that we can filter for the countries that should be returned. To that end, first create a sample file of the data in the CSV file. You can do this by downloading the file to your local file system or creating a new file in the CountryService project, opening the countries.csv file and copying a few records to the new project file.

From the New Gallery, select General | XML | NXSD Schema.

This will bring up the Native Format Builder Wizard that introspects a sample of the data in native format and with your guidance creates the native schema that represents it. This wizard is the same that you use when specifying the mapping for the File Adapter. Only this time, the resulting NXSD schema can be used outside the scope of the File Adapter. At any moment when in a SB Pipeline (or in SOA Composite Mediator or BPEL process) we are dealing with data that is structured – but not in XML format so not accessible to our tooling, for example JSON, Fixed Position format or CSV – we can use the native mapping to convert from the native structure to XML. The reverse operation is also available with the NXSD mapping: from XML to the native format specified.

Create an NXSD schema for the sample file.

Specify the name of the NXSD file to be created:

Specify format type:

Specify the sample file, just like you would in the File Adapter:

Define the names for the Root and Record level XML elements – to produce nicer looking XML mainly.

Set the names of the XML elements derived from the fields in the CSV file:

Click Finish.

Create a WSDL definition that describes the external service interface for the Pipeline and the Proxy Service

A simple service interface describes an operation to retrieve a list of country elements:

Create the pipeline to retrieve remote data and transform from the native format

Right click the new WSDL document. Select the option Service Bus | Generate Pipeline from the context menu:

The Pipeline wizard appears. The pipeline service interface is described by the WSDL document:

Click OK to create the pipeline:

Open the CountryPipeline. Add a Pipeline Pair.

Add a Service Callout activity – to call out to the business services that retrieves the CSV document from the URL. Have the callout activity load it into a pipeline variable called countriesCSV.

Configure the Service Callout to invoke the business service OurAirportsCountriesBS. Check the radio button marked Configure Payload Document. Set the Body field to body in the request document – indicating that the request should use the content of the $body variable as its payload (which is irrelevant since the callout does an HTTP GET request that carries no payload). Set the Response Body to countriesCSV. This means that the body from the response returned by this callout should be used to populate the custom pipeline variable countriesCSV. Subsequent activities in the pipeline can access the contents of this variable using the familiar $countriesCSV notation.

Add an NXSD Translation activity to the Response Pipeline. This activity is used in general to use a Native XSD to map data in native format into the corresponding XML representation, or vice versa. In this case, it maps the contents of the $countriesCSV variable as XML to the $body variable, using the Native XSD that was created earlier based on the sample countries file.

Configure the NXSD Translation as is shown in the figure: Apply Translation should be Native to XML. The input is variable $countriesCSV, populated by the Service Callout. The NXSD Schema to use for this mapping is the Countries element nxsd_OurAirportsCountries.xsd. The result of the translation should be set as the content of the $body variable.

The CountryService overview is shown in the next figure. The Service Callout has introduced a wire from the pipeline to the business service OurAirportsCountriesBS.

At this point, you can test the CountryService’s getCountries operation by running the CountryService. The search criteria you pass in the request message will be completely ignored and the response message will not be according to the service’s interface definition, but you should get an XML representation of all countries provided by OurAirports.com.