SOA Suite File Adapter Pre and Post processing using Valves and Pipelines

A quick note on the notion of valves and pipelines that can be configured in File (and FTP) Adapter Services and References (inbound and outbound) to perform file pre- and post processing on the files before they enter the composite application proper as XML or after they have left the composite application, turned from XML to their native format and before they are written out to file.

Valves can easily be created – in a way that reminds me of Servlet Filters – and the pipeline that can be configured with a chain of valves is also quite similar to a filter chain. A valve is custom Java Class that implements one or two specific interfaces. This class is packaged in a JAR file that is added to the classpath of the SOA Suite: the valve becomes part of the generic SOA Suite infrastructure, to be used potentially by multiple composite applications – not necessarily by just a single composite. Note however that use of a valve is configured in the File Adapter binding in every composite application that wants to leverage it.

Valves can be used for several operations. Some examples on the inbound end are:

• filter files: only let through files or lines that are relevant
• split files: let through structured data and write to grid or temp storage the associated attachments
• pre-transform: convert binary (Word, PDF) to plain text or JSON to XML or CSV
• security/validation: perform check on contents and decide whether to let through or not
• throttle: slow down processing or wait for a toke to become available
• decrypt, correct/enrich or unzip files
• archive files in a more specific way than the file system copy currently supported by the file adapter

For outbound post-processing, a similar list could be composed.

Currently valves are not supported by JMS or Database Adapter – only for file and ftp. I have an inkling that this may change in the fairly near future, though this is only guesswork.

Note: it feels to me that some of the things valves can do regarding file and ftp adapter bindings, we can do through custom web service policies for web service and http binding – message pre- and post processing for inbound and outbound bindings respectively.

Valves are documented pretty well in the Technology Adapters documentation, along with several examples and a step by step description of configuring them: http://download.oracle.com/docs/cd/E21764_01/integration.1111/e10231/adptr_file.htm#sthref306. Appendix B of that manual has a number of samples of Adapter Valves: http://download.oracle.com/docs/cd/E21764_01/integration.1111/e10231/adptr_valve.htm#CHDDIICH

For my own reference – and who knows whom else may find this useful – I will describe the simplest example of implementing and leveraging a valve for the file adapter.

The first part is creating and deploying the custom valve, the second is configuring a pipeline with the valve in a regular composite application. Both parts are very straightforward.

Part one: creating and deploying the custom valve

1. create a JDeveloper application – using the generic template for example – with a single project

2. add the jar-file bpm-infra.jar to the libraries in the project; this file is located in the SOA Suite 11g run time environment, to be found under $MW_HOME/AS11gR1SOA/soa/modules/oracle.soa.fabric_11.1.1/bpm-infra.jar

In the Application Navigator, right-click the project. Select Project Properties. The Project Properties dialog is displayed.Click Libraries and Classpath. Click Add Jar/Directory. The Add Archive or Directory dialog is displayed. Browse to select the bpm-infra.jar file. Click OK. The bpm-infra.jar file is listed under Classpath Entries.

3. Create the Java Class that is the custom valve. This class should implement the interface Valve or StagedValve or – to make life easier – you can simply extend from AbstractValve.

public class MyFirstValve extends AbstractValve {
   ... additional overrides without any logic inside
    public InputStreamContext execute(InputStreamContext inputStreamContext) throws IOException,
                                                                                    PipelineException {
        System.out.println("The valve is executing the inputstream");
        // Get the input stream that is passed to the Valve
        InputStream originalInputStream = inputStreamContext.getInputStream();

        //Read a property key from the adapter binding property
        String mySpecialProperty =
            (String)getPipeline().getPipelineContext().getProperty("mySpecialProperty");
        System.out.println("The valye of the special property read from the File Adapter binding is "+mySpecialProperty);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        byte[] buf = new byte[4096];
        int len = 0;
        while ((len = originalInputStream.read(buf)) > 0) {
            System.out.println("snippet from stream: " + new String(buf));
            bos.write(buf, 0, len);
        }
        bos.close();
        ByteArrayInputStream bin = new ByteArrayInputStream(bos.toByteArray());
        inputStreamContext.setInputStream(bin);
        System.out.println("done processing the stream in the valve");
        return inputStreamContext;
    }

The Valve that I have created below is extremely simple: it received the contents of the file as read by the file adapter and potentially pre-processed by other valves earlier in the pipeline. It copies the content to an outputstream and returns an inputstream based on that outputstream. Meanwhile it writes the contents of the file to the console. However: it does not change anything, it does not validate anything: it is pretty useless as it stands. However, it would be very easy to start adding real functionality to it, as you can hopefully tell from looking at it.

4. Create a JAR deployment profile in the project

5. Deploy the project (well, that one class really) to a JAR file

6. Copy the JAR-file to the classpath of the SOA Suite run time environment, for example to the directory $MIDDLEWARE_HOME/user_projects/domains/soa_domain/lib:

In order for the SOA Suite to recognize the valve class, you need to (re)start the server after adding the JAR-file.

This completes part one. The valve was created and deployed and sits not ready and waiting for composite applications that want to make use of it.

Part two: configuring a pipeline with the valve in a regular composite application

1. Create a new Composite Application (or take an existing one). Add an inbound File Adapter binding to it if it does not already contain one.

2. Open the JCA configuration file for the File Adapter binding. Add a single line to it, that specifies a valve-pipeline to be engaged when executing the adapter. This line could read for example:

<property name="PipelineFile" value="mypipeline.xml"/>

The reference mypipeline.xml indicates a file called mypipeline.xml that is located in the same directory as the jca file itself.

3. Create the file mypipeline.xml with the valve(s) that are clustered together in a pipeline, for use in File and FTP adapter bindings.

<?xml version="1.0"?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
<valves>
        <valve>nl.amis.valve.MyFirstValve</valve>
</valves>
</pipeline>

The file contains one or more valve elements – in the order in which they should be invoked. A valve element contains a fully qualified Java class name – that refers to a class that implements the Valve interface and is available on the classpath of the SOA Suite run time environment.

4. optionally: the valve can read properties that are set on the specific File Adapter binding that it is invoked for. Before deploying the composite application, or during deployment (using a configuration plan) or even after deployment (through the Enterprise Manager Fusion Middleware Control), these properties can be defined and modified, thus impacting the behavior of the Valve – if that is the effect the property has inside the valve.

In the composite.xml file, these properties are set like this:

    <binding.jca config="PickupFile_file.jca">
      <property name="mySpecialProperty" source="" type="xs:string" many="false"
                override="may">someValueForMyProperty</property>
    </binding.jca>

5. Deploy the Composite application and activate, for example by copying a file to the directory that an inbound file adapter is polling from.

The console window now proves that the Valve was invoked and did its job – little as it was:

Note that not only is the content of the file that was processed reported (and realize that it could have been manipulated by the valve), the output also demonstrates how the valve identified the value of the property that was set on the file adapter binding. This could have guided the valve in its behavior for the particular circumstances of this composite application.