The most provoking presentation I sat in on during the Database Systems 2006 conference was probably the one by Rick van der Lans, on 10 golden rules for SOA design. Rick is a fairly wellknown speaker in The Netherlands (and abroad his bio states and I ran into one of his presentation in Finland to underline that point, see http://www.tieturi.fi/seminaarit/ohjelmistokehitys2006/documents/soa_2006_esite_web2.pdf), on wide range of topics like Business Intelligence, Database Design and more recently Service Oriented Architectures. He is a very entertaining presenter, knows how to engage the audience and has a number nice presentation tricks up his sleeve.
I was a bit skeptical initially: it is hard to believe anyone has in depth knowledge on that wide range of topics he presents on as well as participate on real-world projects. However, although not all my hesitations have disappeared, his presentation was certainly worthwhile and triggered some very good discussions. Clearly he has a lot of experience to build his story on and he knows very well how to read the developments in the industry.
His best tip, which he re-iterated several times, is: make sure you buy your SOA infrastructure – like ESB, Adapters, BAM tools, BPEL Container, WebService Management and Governance tools – from a vendor who will survive the shake-0ut that in his opinion is imminent. There are over 30 vendors active in the SOA space and soon a shake out must follow that will have only a few survive. He strongly recommended the audience to only deal with vendors that would be among the survivors.
Unfortunately, he conceded, he was not able to tell us who would survive. A quick poll among the audience – combined with his earlier findings, suggested that IBM, Microsoft and Oracle were among the likely survivors (BEA was mentioned as well). Rick suggested that where IBM has never ever been successful on the upper or outer-layer of the application stack, they might be primarily successful at the core layers of the SOA stack, with for example MQ Series or other ESB infrastructure components. Microsoft on the other with a strong background on the client side might on the other hand be a less likely candidate for the hard core, the backbone infrastructure – Database, BUS and MOM end of the SOA spectrum.
SOA implementations more often than not FAIL
Rick stated that in his experience the majority of SOA implementations to date have failed. Clearly, doing SOA is not trivial. He proceeded to outline a list of ten issues, principles, tips to bear in mind.
1. Multi Layered Services
Rick stressed the multi-layered nature of the service set up: there are multiple layers of services, from coarse grained to very fine and from very business oriented to more technology driven. And everywhere, at all layers between the services is the BUS, the ESB infrastructure that allows communication between services.
He used the terms Basic (or Raw) services for the lowest level of services that do nothing more than expose legacy functionality through well defined interfaces in terms of standardized protocols. On top of these basic, fine grained services are the composite services that combine basic services into better and more economically reusable composite services. On top of those, we build the Business Process Services – these latter often as BPEL process definitions running in BPEL containers.
At this point Rick introduced the “SOAP Line”. Everything from the basic services upwards is above the SOAP Line, the legacy functionality is below it. He did not clearly define the SOAP Line. The impression I got was that above the SOAP Line, all services are in fact SOAP based WebServices. If that is the case, I would have to disagree with Rick on this. The overhead associated with SOAP invocations, including the serialization and deserialization of XML content, is often much more than is acceptable. Given the fact that part of the core competencies of most ESBs and BPEL Containers is their ability to bridge technologies through adapters for protocols like JCA, JMS, EJB, plain Java, JDBC/SQL and PL/SQL and others, it may not be necessary or even desirable to publish basic and perhaps even composite services (only) through SOAP interfaces. Of course a requirement for any service at any level is a clear specification, typically through a WSDL document, and an interface in one of a limited number of more or less standardized protocols (say the ones commonly suported by Bus infrastructures.
Questions regarding the multi layered service architecture to be answered include: where do you do clean(s)ing of data? Where does the composition take place? For example: can a basic service make calls to multiple legacy interfaces? Whose responsibility is it to extract data in the proper format. For example: if a legacy system provides data in its own proprietary format and the business service needs a specific (standardized?) busines format – at which level should the transformation between these formats take place? Or: the Business Service is invoked with a lengthy XML message that somewhere contains the customer identification. As part of the business process, some information is required about the customer. Information that is available in a legacy system and is exposed through a basic service. At which level should the customer identification – the only part required and understood by the legacy functionality – be extracted from the XML Message. Again, I had to disagree with Rick’s suggestion that the basic service should accept the XML message and do the extraction before invoking the legacy functionality. It would be my assumption that in order to further the reusability of basic services, their interfaces should be simple and specific. I would not want to clutter up the basic services with functionality to extract customer ids from several different types of XML messages – more a task for the composite or even the business level. But a good discussion nevertheless.
2. Bottom up or Top Down
How should the services be designed? Bottom Up – from basic working outward to business – or Top Down – from Business working inward to Basic or in a more mixed manner? Rick stated that Bottom Up the typical developer’s or architect’s approach is. From the level of legacy functionality or basic services it is virtually impossible to define meaningful business services.
The other way round, working from the business level all the way down to basic services, is equally unrealistic. Working Top Down it would be very difficult to go beyond the level Composite Services – or even get to the best possible design of the composite services. So then what?
The fact that a mixed and iterative approach is best should not come as a surprise. Business is leading – as the primary gain from SOA is at the business level in terms of flexibility (agile business process designand redesign, short time to market, integration across the enterprise and beyond the enterprise and reduction in costs). You do not sell a SOA to the upper management! You sell them Business Flexibility!
Following the first cut definition of Business Services should be an analysis of legacy systems, a rough design of basic services followed by the mapping between business services on the one hand and basic services on the other in terms of the composite services. Following the rough analysis could be a more refined iteration. It is also important to address a realistic – small!- scope in one iteration. Do not try to inventarize all services in the beginning. I heard a story about an organization that did this, analyzed some 30.000 services and ran out of budget before even one of them could be implemented.
3. Design Services Outside In
First design the service definition, its interface, through a WSDL. It is like Test Driven Development or Design by Contract. The same suggestion in even stronger terms was voiced by Guy Crets (see: Database Systems 2006 – Part 1)
Rick suggested that a Java Developer typically works from the inside out: writing code first and specifying the interface later. I would have to disagree on that point as especially with Java development quite often the interfaces – an integral construct in the Java language and an important part of OO Application Design – are specified first and the implementation follows.
4. Impact on IT organization, impact on Business (End User) organization
The impact of a SOA on the organization will be substantial. First the IT organization: For example: the role of Bus and BPEL administrator needs to be fulfilled. And Service Level Agreements for these infrastructures as well as for the individual services need to be defined. We no longer develop applications. We develop services that are composed into business process implementations at the business service level. By a business analyst perhaps? And who will take care of generic services that are not tied to specific business processes?
However, the impact on the Business organization can be even further reaching. Since we no longer create departmental applications, but instead processes, it is quite likely that some or even many of these processes transgress the boundaries between departments. Who then will be the (business) owner of such a process and business service? Should the departmental structure of organizations be revised?
5. Analyze Legacy
Obviously, when creating the basic services we need to have clear insight in the legacy applications and the functionality they offer. However, there is more to be aware of. For example: a legacy application has had a single point of entry for many years: the user interface. When we open up the legacy application through services, we create new access points, by-passing the user interface. It may very well be the case that functionality that has never been called upon before, now suddenly is invoked and turns out to be incorrect in some way. For example validations and enforcement of business rules, previously done in the user interface now relies on the implementation that was never used before.
6. Minimize the number of vendors
Even though standardizations and interoperability are assumed and suggested, it may not be as simple as one would hope to build the SOA infrastruture using components from many different vendors. Rick strongly recommended to try to work with as few vendors as you possibly can, picking of course only the vendors who will survive the shake-out.
7. BPEL is not BPM (business process modelling)
BPEL (Business Process Execution Language) is fairly technical. Do not mistake the attractive process-diagrams that BPEL Design tools can present for rigorous and conceptual Business Proces Modelling. With BPM you specify what should take place in the business processes, using BPEL we can specify how the what should be done. He mentioned Engage – a Dutch company that among others Oracle is speaking with – as an example of a BPM tool. Rick did indicate by the way that it would be wise to restrict oneself in doing BPM to the concepts momentarily available in BPEL – the language and the implementation you use.
After rule 7, time became limited and Rick had to rush through his last three rules:
8. Testing tools from day one!
9. Governance
You need a shadow-layer on top of the services, a Web Service Management infrastructure, that takes care of governance: security (authentication, authorization, encryption/decryption), registration and monitoring, load balancing, decoupling etc.
10. Do you need such a SOA thing?
Before you start developing a SOA – ask yourself what it is you actually need. Maybe you do not need a SOA in the first place! Remember ( one of the themes of my own presentation at this conference, see Database Systems 2006 – Part 1) SOA is only a means, not an end in itself!
