The other night I had an interesting dinner date: seated around me were an Oracle ACE Director (Mark) and two Oracle employees, experts on Oracle Coherence (Ewan) and Oracle Data Integrator (Craig) respectively. The dinner was the closing event for all presenters and ACE Directors attending the EMEA SOA Partner Community Forum, put together by Jürgen Kress. A great event, a fine dinner and even better dinner conversation. Apart from discussing the Bejing 2008 and London 2021 (that is not a typo, the gentlemen were fairly pessimistic about the construction schedules for most Olympic venues), Dutch and British politics, American peculiarities, the wonderful Dutch countryside etc. we inevitably found ourselves back on the topics most on our minds: grids and clustering, message and data transport and the likes.
We discussed the SOA Grid – strongly advocated by Dave Chappell – and how Coherence would provide the grid infrastructure for that. Then I had the opportunity to follow up on the question I had raised earlier in the general session of how Coherence is not just a Data Grid but a Compute Grid as well. I was interested to learn whether the parallel processing power of the nodes in a Coherence Cluster can be used – not just for storing, retrieving and manipulating data – but for more general tasks as well. Ewan took out paper and pencil and started a small lecture – interrupted from time to time by the waiter to tried to introduce the various courses of the surprise menu we were having.
As it happens, setting up a compute grid facility is not all that hard. The Coherence grid manages data in Caches. A Cache is similar to a Map – but of course it lives in a clustered environment. The Coherence’ NamedCache can have MapListener attached that can handle
events like Inserts, Updates and Deletes. A special type of MapListener is the Backing Map Listener. It fires synchronously – blocking the thread that loaded the data on the grid that caused the listener to fire.
What the BML can do when it realizes a special data element was loaded – for example a Task Request with callback to report the outcome – it can leverage the Java 5 Executor facility to have a task scheduled and exectued asynchronously in another thread.
Our biggest challenge would seem to partition the task at hand in portions that can be processed individually by the nodes in the cluster and a way to reassemble the results. Certainly an interesting exercise for one of our upcoming knowledge events. Note that the resources at the end of this article provide a lot of help getting started with this.
JMS
A queue (JMS or otherwise) can be a useful mechanism to insulate various parties in our architecture from one another. The main prerequisites for a JMS implementation seem to be: accept messages rapidly, hold them reliably, be scalable and make messages rapidly retrievable. It seems almost like a definition of a data grid! Therefore, it seems more than feasible to base a JMS implementation on a grid like Oracle Coherence. An interesting thought that may lead to a very fast and scalable implementation that could provide a strong backbone for ESB and SOA infrastructures in general. More on this in later contributions.
Oracle Data Integrator
One of my favorite Oracle products of recent times has been Oracle Data Integrator (ODI). Given the fact that I have not done a lot with it so far, that is a strange thing to say. However, reading about its capabilities and discussing the product with colleagues and peers with more in depth experience continues to pique my curiosity and feed my interest. Having Craig – senior top notch principal specialist on ODI – sitting across the table allowed for another round of ODI drooling. We discussed the trickle feed pattern – the bridge in our conversation being provided by the JMS discussion – as ODI leverages the queue to turn bulk volumes of changes, for example captured from the source database when a mass update took place, into manageable batches that are for example passed to a queue for trickly processing by the target reading from the queue.
It is still somewhat surprising to me that whatever ODI does in terms of transformation is translated back to a SQL engine. There is (still) no native XML transformation based on XSLT for example. In the end, ODI is a very fast, pretty smart SQL processor (or rather SQL engine operator). Having ESBs with very strong XML transformation capabilities to complement ODI in that area, I am not devastated.
Note that ODI 11g will use JDeveloper 11g as its design time environment (this tidbit did not originate with Craig but with two ODI folks I talked to at JavaOne 2008, earlier this year). When the question was asked in a general session at this event, Craig sidestepped the question as he was not sure whether that was information he could divulge.. It is sometimes pretty hard to be at Oracle, never quite knowing what you are allowed to tell people while bubbling with information and enthusiasm that you really would like to share..
Resources
Brian Oliver’s Pimp my Data Grid
Ashish’s thoughts: When you let the grid do your job
Example scenario of implementing a parallel job processor with Oracle Coherence
OTN Forum Thread: Coherence Grid as Service Event Medium
