Introducing Oracle Real-Time Integration Business Insight – complement to SOA Suite & Service Bus

The launch of Oracle Real-Time Integration Business Insight – a name that suggests Larry’s direct involvement in naming the product – marks the moment at which the integration product stack of Service Bus and SOA Suite makes a big step into the business domain. The implementation of simple and complex integrations and business processes with SB and SOA Suite is a technical affair. Steeped in WSDLs and XSDs and Pipelines, Mediators and BPEL processes, to the business this is all one big black box. And what goes on inside that black box is quite important, because it drives the business. But to the business it is quite inaccessible.

Not any longer.

With Real-Time Integration Business Insight – or just Insight from here one – Oracle releases a means for the business to easily look into the black box. To gain understanding in business terms of the activity in that black box.

Insight was designed with the following objectives in mind:

•Learn what is going on inside the black box –in functional terms and from a business perspective; learn what happens, how often, when & what problems occur, where are bottlenecks

•Provide operational information in real time for immediate action – Monitor (live) – and provide tactical insight – Report (later) – for deeper understanding, trend-analysis and future planning. Allow drill down, inspect & analyze, operate

•Decide in an iterative fashion what to watch for- define and fine tune at runtime, through a browser based UI, with no need for deployment cycles

•Be non invasive – no code changes required, no involvement of the development team

•Avoid any potential to disrupt the execution of services and processes – breaking the chain by observing it too closely

•Have very low overhead – the execution of integration flows should not suffer from being monitored

Insight provides two important capabilities: capturing metrics from the integration components buzzing and whizzing inside the SOA Suite (and recording them into a database) and exposing these metrics in business oriented dashboards and overviews. Additionally, the captured information is exposed through OBI EE reports and in BAM.

The main entry point into Insight for business users – after a model of the information to capture and present has been created an activated – is shown below:

A core concept in Insight is “Milestones”. A milestone is a meaningful state for [any instance of] an end-to-end business flow. At some point in the flow, it can be said that the instance has reached a stage, or enters a state, and thereby has reached or passed a milestone. It is not a technical concept at all – it is a business label that starts to apply to an instance under business conditions.

Once the business has identified the milestones that are applicable for a specific type of flow, the milestones can be mapped to Service Bus projects and SOA Composites. A somewhat tech-savvy user will use the browser based tooling to associate the milestones with entering or exiting Pipelines or Business Services in Service Bus or with entering into or responding from a SOA Composite or a component within a composite (Mediator, Task, Business Rule, BPEL) or even a specific activity inside a BPEL process. .

At runtime, Insight will for example show the numbers of instances – that satisfy the filters – that have reached certain milestones. This screenshot shows that three instances have reached a terminal milestone, 3 more (1 and 2) have exited their flow in an error state and 3 more are still running, after reaching the Approved milestone. From this overview, we can quickly drill down to find out the details about the instances for each milestone.

It is important to realize that Insight does not care about many of the implementation details of integration flows that would bother us with setting up our own BAM reporting or with trying to gain insight through the Enterprise Manager FMW Control. For example: insight will track business flows across Service Bus and SOA Composite and across different message flows and disconnected ECIDs. Sections of execution that are unconnected in the message flow trace can be monitored as part of the same end to end business flow in Insight.

The details in the overview below are drawn from four different, unconnected underlying implementation components – one Service Bus project and three SOA composites – that do not share an ECID and show up as unconnected message flow traces in EM FMW Control.

In addition to the numbers of of instances per milestone, we also learn something about the time taken to get from one milestone to the next.

We can drill down to an individual instance and inspect its full life cycle. The timeline below shows all the stages of a one specific instance, and the values of business indicators – to be discussed below – along the way.

From this timeline – that spans five different unconnected underlying implementation components – we can click on a particular milestone – typically especially an error milestone – to open an email client and send an email to an administrator to inspect and fix pending problems for the underlying instance in SOA Suite. This email contains a deep link that allows the administrator to navigate into Enterprise Manager FMW and the correct message flow trace in no time at all.

Some out of the box charts are available with visualizations of the highlights for the business flow being monitored. Here we see the numbers of instances per milestone against time, the number of still active instances, the distribution of errored out instances over the terminal condition and the average processing time (end to end) of completed instances.

After the milestones have been identified, top down in a functional way, from the business perspective, we can also define the business indicators: which attributes of each end to end business flow is meaningful to us? What values about these flows should be exposed in dashboards and reports? To be used to classify and filter and group by [dimensions], or to show as measures in graphs and to calculate with and aggregate over. The business owner indicates what the things are he or she would like to know for each instance of a business flow and specifies the names for these indicators.

It is up to his or her tech savvy assistant to map these indicators to the XML messages flowing through the underlying implementation – using XPath definitions to extract the value for each indicator. Here is a screenshot of how the mapping of business indicators take place. First the indicator is associated with a component – Service Bus pipeline or business service or SOA Composite Service, Reference or Component – and then the XPath expression if defined to compose the indicator value from the message entering or leaving that component. For indicators associated with BPEL components, their value can also be derived from BPEL variables – global or local.

With the business indicators set up, Insight allows custom dashboards to be defined. These are charts that present aggregated overviews of recent and current activity using the values of business indicators extracted from the running instances. Here is a very simple example of a pie chart that publishes a sum of a numerical business indicator across all monitored instances.

After creating the initial model and using it for some time to collect metrics and monitor instances of business flows, the Insight model can easily be fine tuned and extended. The model is independent of the deployed integration components and can be changed without impacting the execution of the integration flows at all. Models could even be tweaked in the production environment – with the activation step as ‘deployment process’. Alternatively, a model could be evolved in a different environment and through export and import be promoted to the production stage.

Redeployment of the Service Bus project or the SOA Composites does not affect the Insight model – unless of course the mappings for milestones and business indicators are no longer valid in the updated implementation.

Note that we can create multiple models on top of the same components. For example when a specific composite plays a role in various business flows, or we want to create multiple windows on the same business flow – it is perfectly fine to do so.

The collected data is stored in a database. It is available in BAM (Business Activity Monitor) to further create reports, dashboards and alerts against. The same data is also exploited in OBI EE through several out of the box reports and any custom report you would like to add. Through straight SQL, you should be able to leverage the collected metrics in any other way you see fit.

I have the feeling that Insight can also play a role in automated white box testing. Since insight captures various functional states and business indicator values along the execution of an end to end business flow – and stores the in a database – it should be doable to run test scenarios and check how these scenarios play out in the Service Bus and SOA implementation components – do they follow the intended path, do they produce the expected values and end in the intended state. I have not actually tried this out yet, but it seems like an interesting option at the very least.

Introductory Video on Insight

Conclusion

With Real-Time Integration Business Insight, Oracle hands us a facility to bring monitoring of the execution of integration flows in the hands of the business. From a business perspective- using a top down, functional approach – we can define what we are interested in. Regardless of how the implementation happens to be done, without involving development teams and having to go through lengthy deployment cycles. Defining what to capture is easy, and iteratively further evolving our interest is easy too. Low risk, low impact, high value. The metrics collected by Insight at run time can be applied in various ways – in near real-time or in a somewhat more tactical time frame. Using various means and formats.

For me, Insight brings the formerly black box of integration to a place where functional meaning can easily be grasped. Now it is up to the business to define their interest. That by the way can still be a serious challenge. The potential of Insight for rapid prototyping of models and dashboards will prove very useful in sitting with the business and structuring the insight they need and should get.