Guaranteed delivery not obvious for ESB.

Recently I have made an ESB system based on dequeueing an advanced queue and conditionally forward of the content to different files on the file system. In the development process I ran into some issues which for me raised the question "Are messages processed by the Oracle ESB delivered to the end-point with perfect guaranty?" Or do messages get lost along the way?

In this blog a system is build that tries to give an answer.


Figure 1 shows my little ESB system. There are two incoming adapters reading/dequeueing an Oracle advanced queue in my local XE database. Why I needed two adapters for dequeueing the same Q is explained later. Both adapters are connected to a router wich is writing the contents of the message to a specific file. For logging purposes an extra log file is created for every message that is processed by both routers.


Figure 1: ESB system overview


The queue is a single-consumer Q in a local XE database. Messages are enqueued with a correlation ID (numbers 1..4). The payload is an Oracle object-type, not an XML-payload. The ESB is handling this very well. The AQ-adapter(s) in the ESB dequeues the messages based on the correlation ID. Unexpectedly, I could not just create one AQ-adapter that would read all messages and let the routing service figure out where to route the messages, based on the correlation ID. In my humble opinion that’s the purpose of routing. However, this is actually done by the AQ-adapters. So for now I’ve build two AQ-adapters: one for correlation ID = 2 and one for correlation ID = 3. This means however, that messages with no correlation ID or with correlation ID other than 2 or 3 are never dequeued by the ESB system!

Messages that are picked up (dequeued) are then written to files.
In my first attempt, I defined one file for each dequeued message, name pattern: corr#_%yyMMddHHmmssSS%.txt (# = 2 or 3) and
one logfile which writes (logs) each and every message (name pattern: %yyMMddHHmmssSS%.log) and put all files in the same directory.

This was a very bad attempt because I found that not all incoming messages either weren’t written to file or were delivered too fast so that the name patterns was not specific enough: files would be overwritten by other files: Unguaranteed delivery?!

By querying the queue table I can count the messages per correlationID as is seen in figure 2. The number of incoming messages in the ESB instance page showed the same amount of messages (figure 3). On the file system there were log files and txt files missing.

Figure 2: query Q table

Figure 3: ESB instances view

How to fix/workaround this?
1) By placing the log files in another direcotory.
2) By using %SEQ% instead of %yyMMddHHmmssSS% in the name pattern.

%SEQ% apparently results in a unique identifier per outboud file adapter service. Meaning that if you have 2 file adapters both generating files like %SEQ%.txt simultaneously, both will have files named 1.txt; 2.txt and so forth. So do not let all your outbound file adapters write to the same output directory.

In my case I created name patterns like corr#_%SEQ%.txt. Resulting in the files shown in figure 4.

Figure 4: screenshot of file system

So, in the end I got was able to create a small ESB system that was guaranteed delivering messages from a Q to different files on the file system. But this little test nevertheless shows you must be carefull how you design your ESB systems.

Besides that I think it would be nice if the correlation ID and other header and meta information can be read by the router and transformations some how, so the system can do with just one AQ-adapter and routing can be done more finegrained.

Also, in my attempts I was unable to get the header or correlation ID from the incoming messages. A bug or not yet discovered feature? In the outbound file adapter the impression is made that you can write header information into the file, but you can’t. Only payload can be routed, transformed and so on.