This article gives an overview of some of the most eye catching stories from Oracle OpenWorld 2015 around integration. It discusses on premises products such as SOA Suite, BPM Suite and MFT as well as cloud services like ICS, SOA CS and soon to be API Platform CS and MFT CS.
If you want to be a SaaS vendor – an ambition that Oracle clearly has – then you need a good solution to deal with integration challenges, if only to prevent SaaS from becoming to mean Silo-as-a-Service. Being able to create business transactions that span applications as well as data look ups from other applications and synchronization of data across applications is an essential part of the PaaS platform that a SaaS vendor should offer to its customers. The applications involved in the integration scenarios can all be cloud based – although potentially in various clouds from different vendor – and can also include applications running on premises.
The Integration Cloud Service offered by Oracle address this crucial challenge of making applications talk to each other in an structured, coordinated manner that can productively be configured and easily be managed. The underlying technology for ICS is taken from Oracle SOA Suite and Service Bus. That fact vouches for the capabilities, stability and maturity of ICS – and should otherwise be encapsulated as ICS is positioned as a solution for not-very-technical users. ICS provides a rich, largely declarative and functional user experience – even though occasionally some XML and XPath may shine through.
ICS went live in the early Summer of 2015. It provides pre-built integrations, that are a little reminiscent of elements in AIA (Application Integration Architecture) out of the box, such as Sales Cloud to Service Cloud and CPQ Cloud to Sales Cloud. Oracle intends to provide many more of these integrations – stating that “Nobody knows Oracle applications better than Oracle itself”. Oracle partners are invited to also implement such ready-to-run integrations and share them through the Cloud Marketplace.
With ICS, anyone can implement tailored integrations to connect any combination of applications, either in the cloud or on premises. These integrations can be synchronous – waiting for the response – or asynchronous, fire-and-forget or even event-driven (for example a Convert Opportunity to Quote / Order business event published from Sales Cloud). A new addition to ICS is support for scheduled integration – where integration are executed according to time based schedules, for example for daily synchronizations. These integrations can be based on service calls as well as (s)FTP based file exchanges.
Other upcoming additions to ICS are support for content based filtering & routing and even orchestration – including branching, looping and chaining of call outs. One keynote (by Thomas Kurian) linked this orchestration to BPEL:
It is my sincere hope that even if under the covers BPEL may be used that fact is not exposed and the ICS developer just deals with orchestration in functional terms.
The new UI canvas editor allows creation and inspection of these complex flows.
Cloud Adapters provide generic connectivity to applications – such as SalesForce.com, Oracle Sales Cloud, Siebel, Microsoft Dynamics CRM and SAP – and platform services, technologies & protocols – such as Oracle Database, JMS, XMPP, REST, Oracle Messaging Cloud, Apache Spark and Oracle Storage Cloud. These adapters know how to interact with the application or technology at hand; they understand the protocol, the APIs, the events available for subscription (for example with Oracle Service Cloud) and the generic message structure. The design time part of the adapters have the ICS developer configure the adapter for a specific interaction. At run time, the integration flow leverages the adapter with the specific configuration to request or push the relevant data in the correct format over the required protocol executing the underlying handshake. The complexity of interacting with SAP, SalesForce or Workday is largely encapsulated in the adapter.
The library of Cloud Adapters has grown considerably in 2015 and Oracle’s ambitions for further development of adapters are impressive – over 60 potentially adapters have been identified, from Twitter and Facebook to Workday and SuccessFactors. Additionally, partners are wholeheartedly invited to also create Cloud Adapters using the SDK, and publishing them through the Cloud Marketplace. Note: there is no reason why ICS should not be used to integrate two non-Oracle SaaS applications; it might very well prove the best way to integrate SalesForce.com with SAP.
At least some of the integrations executed by ICS are likely to involve an on premises application on one end. To prevent organizations from having to open ports into their enterprise network or deploy core assets in the DMZ or otherwise compromise on security, network topology or application architecture in order to allow connectivity from ICS, Oracle intends to release an agent that runs on the internal network. All interactions with ICS are conducted through and initiated by the agent through a secure network tunnel; there is no unsolicited inbound traffic from ICS. This agent can be installed directly from the browser; it is akin to a stripped down version of SOA Suite – with all cloud adapters – that carries out configurations and integration flows according to instructions it collects from ICS. Interactions with on premises applications, databases and other components can be configured through the ICS design time and are then transferred to the on premises agent for local execution. This agent is expected to become available in the Spring of 2016.
Alternatively, integration patterns between cloud and on premises can use a local SOA Suite or Service Bus instance, and for example interact through the Oracle Messaging Cloud Service for HTTP based asynchronous interactions or use a web proxy for synchronous exchanges with the cloud. API Gateway could also be engaged in these approaches.
SOA CS and other integration cloud services
The Integration Cloud Service offers a browser based design time for declarative configuration of integration flows using out-of-the-box components, predefined mappings and minimal coding. For many scenarios, that might be enough to get the required level of integration, especially as the library of cloud adapters is further extended.
There will be many occasions though where the integrations that need to be implemented are more complex than ICS can handle. Or situations where existing integrations in on premises SOA Suite and Service Bus need to be moved into the cloud, at least for development and test. Oracle opened up the SOA Cloud Service in the Fall of 2015 – offering the equivalent of SOA Suite 12c in the public cloud.
The design time is JDeveloper – just as with on premises SOA Suite; the deployment however is not to a local, on premises environment but instead to a cloud based instance managed by Oracle. The Cloud Adapters that are available with ICS are also in SOA Suite in premises and the SOA Cloud Service. The Oracle Developer Cloud Service can be enlisted to perform the build – using the public Oracle Maven artefacts and sources in Git or Subversion – and deployment to the SOA CS instance, using a Hudson job.
Both Service Bus and SOA (SCA) engine are available in SOA CS, stand alone or combined. Near the end of 2015, support for Managed File Transfer, ESS (Enterprise Scheduler Service) and B2B will be added to SOA CS.
Note that until the Spring of 2016, when 12.2.1 is provided, the SOA CS will run only SOA Suite 12.1.3. With 12.2.1 will also come adapter support for Oracle ERP, Messaging Cloud Service as well as support for elastic up and down scaling.
At this moment, there is no BPM CS and there are unfortunately no plans announced for providing it. However, BPM Suite 12c is certified to run on JCS (using existing or new BPM Suite on premises licenses).
Several other services are available in Oracle Public Cloud that offer integration [related] functionality. Mobile Cloud Service exposes (REST) APIs targeted at mobile apps – although other consumers can benefit from these services as well. MCS will typically interact with ICS, SOA CS or JCS to access the services backing the APIs. Oracle Data Integrator 12c has been certified to run on Java Cloud Service.
A brand new GoldenGate Cloud Service has been announced. This service will offer the same functionality from the cloud for replicating data [events] across heterogeneous systems that is available with the on premises product. No operational overhead is incurred on the enterprise IT department by moving the workload of GoldenGate itself to the cloud. When one or both of the systems involved are cloud based, there could even be a latency benefit with running GoldenGate in the cloud. Workloads however can be moved back and forth from cloud to on premises. Given the fact the GoldenGate is frequently used temporarily for migration and upgrade scenarios that take just a few months to complete where long term licenses frequently seem a bit of an overkill, usage based pricing from the GoldenGate CS during a limited period may be very attractive for many organizations.
The new IoT (Internet of Things) Cloud Service gathers streaming data from the intelligent edge (edge devices that preprocess messages received from the ‘things’ (such as sensors, appliances, cameras etc.) and analyzes them in real time. The results from this analysis can lead to onward messages to services in ICS and SOA CS. Note that events in ICS and SOA CS may also provide input to IoT CS.
The user interface in IoT CS for creating the streaming event networks and dashboards is Stream Explorer – also available on premises as the user friendly real time event analyzer on top of Oracle Event Processor.
SOA Suite
Even though much of the focus during the conference was on integration in the cloud, there was also some very well deserved attention for the on premises SOA Suite product, for which Oracle claims over 7000 customers. The brand new 12.2.1 on premises release of Fusion Middleware includes a new major release for SOA Suite. This release has a number of areas with relevant changes.
For developers, the end-to-end REST and JSON support is a major new feature. Unlike before, REST services are just a wrapper around SOAP services and JSON messages do not have to be converted to and from XML and can be manipulated using JavaScript instead of XSLT, XQuery or Java. The REST Branch in Service Bus is an example of the end to end REST features. Support for JavaScript is widespread: in Service Bus and BPEL Actions as well as through E4X in data mappings and JavaScript instead of XPath in expressions and conditions.
The debugger has been further enhanced compared to the 12.1.3 release. For example, the SOA debugger provides remote debugging capability for XSLT maps that have been deployed in the application server. The debugger can also be used with XSLT maps in Oracle Service Bus projects.
SOA Suite 12.2.1 runs on WebLogic Server 12.2.1 which also has several interesting new features that SOA Suite users can benefit from. One is the new (Alta UI) look & feel for Enterprise Manager Fusion Middleware Control. WebLogic Server 12.2.1 is certified for use with JDK 8 (Update 40 or later). Also the now full support for Java EE 7 in WLS opens up some options with SOA Suite, for example with the JMS 2.0 API. WLS 12.2.1 is certified to run in a Docker Container. Whether this also applies to the SOA Suite run time is not entirely clear. From version 12.2.1, WebLogic Server supports parallel deployment which should improve SOA and Service Bus startup time.
Under the label Oracle Integration Continuous Availability features – which is an extra cost option for SOA Suite – release 12.2.1 offers a new set of operational features including availability, performance, scalability, diagnostic and other operational aspects. Operations around SOA Suite can be streamlined in several ways with these new features.
Composite Instance Patching supports deployment of an updated version of a SOA Composite definition and having already running instances pick up the updated definition (similar to what is already possible for long running BPM process instances). There are limitations on what changes can be included in such a patch; these include XSLT/XQuery mapping definitions, adapter configuration, BPEL assign and invoke activities.
In-Memory SOA provides optimized performance, scalability, and reduced database growth by offering the choice per SOA Composite to not write records for instances to the database, only write them upon instance or at least defer writing them to the database in batches in a background transaction. A Coherence memory grid is used to keep track of running transactions.
Three other big features in the Continuous Availability option are
· Circuit Breaker – improved resiliency when downstream services go down by shutting down the upstream service until the problem is resolved.
· Integration Workload Statistics – a performance diagnostic tool similar to AWR for Oracle Database
· Automatic Service Migration – Supports Faster fail-over by leveraging WebLogic’s Automatic Service Migration (ASM) feature to automatically failover singleton services like JTA, JMS to an already running cluster node.
The Cloud Adapters that were discussed for the ICS and SOA CS services are also available for on premises SOA Suite. This expanding set of adapters will soon after OOW 2015 include adapters for Oracle Sales Cloud, HCM Cloud and Eloqua. Custom adapters developed using the SDK can be deployed with both on premises and cloud based integration products.
A big next feature down the road for SOA Suite is multitenancy. Similar to multitenancy in Oracle Database (pluggable databases) and WebLogic (partitions), this feature will allow multiple tenants to run in perfect isolation in the same SOA Suite instance. This effectively means achieving higher density or getting a higher workload on the same set of CPUs and licenses.
Along with the 12.2.1 version of SOA Suite, there were also updates for MFT (Managed File Transfer) and B2B (Business-to-Business). The new release for MFT has valuable built in support for WebCenter Content and Document Cloud Service as end point, as well as ODI and Storage CS.
Additionally, support was added for jobs to be run against remote end points when events occur. Other new features include APIs for creation of transfers and resubmit of bulk actions, priority settings on file transfers, support for PGP cipher selection and some life cycle management around deployment plans.
The next biggest thing for MFT is the Oracle Managed File Transfer Cloud Service, in the end of 2015 based on 12.1.3 and early Spring 2016 based on 12.2.1.
Similarly, B2B will go to the cloud as part of both SOA CS and ICS. Release 12.2.1 has a number of new features, including endpoint cloning, FIPS compliance, PGP support, MLLP Synch-Request-Reply, endpoint throttling, performance improvements and a new Alta UI based user experience.
Beyond release 12.2.1, B2B leverages and extends some other new features in SOA Suite to offer improved monitoring, analytics, and end-2-end transaction/flow tracing when used in conjunction with other SOA components. A self service partner portal is envisioned, focused on easing the B2B operator task load; this portal will begin as a way for partners to self-register, define endpoints and select document formats.
The BPM Suite 12.2.1 release was part of FMW 12cR2 published a few days for the OOW 2015 conference. This release is primarily a quality consolidation that brings together the bundle patch releases issues for 12.1.3. In addition, it provides integration with WebCenter Portal and Content 12.2.1. This release includes user interface (UI) components that can be embedded in WebCenter Portal: Task List, where Portal users can access tasks assigned to them by BPM, and Application List, where Portal users can launch BPM applications. These BPM UIs can be imported as a Portal Resource Catalog, then embedded into any Portal page.
BPM Suite 12.2.1 also includes a full set of REST APIs that enable applications built with any technology to leverage BPM Suite as an engine for workflow management, service integration, and process orchestration. This release specifically provides APIs for fine grained task control and ACM case control.
Analytics
Analytics is an area around SOA Suite where a lot is happening. The Business Activity Monitoring (BAM) product was further enhanced in the 12.2.1 release. The integration team is working on a new product called Real-Time Integration Business Insight and StreamExplorer keeps getting enriched for real time [event] stream analysis.
In BAM 12.2.1, many stability concerns with the 12.1.3 release were addressed. Additionally, the look and feel as updated to the Alta UI look and feel, with richer and faster, more light weight data visualization components. Defining KPIs and Alerts is easier with the use of functions. With a simple checkbox, reports can be configured to be updated dynamically in real-time. Similarly, reports can use all existing data from the database or only use fresh data from a sliding time window. The new Opaque Query allows a developer to create a complex SQL query that business users can make use of in business views. Hierarchies for date-time and other nested relations (such as geographical, product categories, organizational structure) can be defined and used for aggregation and drill down interaction.
BAM dashboards can help provide insight in many types of data, including process analytics from running and [recently] completed BPEL and BPM process instances. With a little more effort, BAM can use Service Bus alerts, SOA Suite composite sensors and other JMS messages to report on other integration activity in Service Bus services and SOA Composites. This requires the deployed artefacts to have been configured for reporting analytics. Without this [at least a little intrusive] configuration, no BAM reporting will be available. The new product Real-Time Integration Business Insight (Integration Insight from hereon) overcomes this requirement. Integration Insight is capable of extracting real time analytics for running Service Bus and SOA Composite instances. These analytics can be mapped to a business model with milestones, business dimensions, measures and attributes. Integration Insight reports on the live analytics in BAM based dashboards defined in terms of the business model definitions.
Through a browser based user interface with the familiar Oracle Public Cloud experience, business users and integration architects can configure Integration Insight on top of already deployed integration applications and start reporting on them.
This capability of extracting analytics from non-instrumented Service Bus and SOA Composite applications is very powerful. It would be great if it could also be used for running BPM processes. Additionally, it seems highly desirable to be able to feed these analytics into Stream Explorer for real time processing. And of course these Integration Insight analytics should be available for custom BAM reports with the full power of BAM and supporting data objects at our disposal.
OOW 2014 saw the initial announcement of Stream Explorer, a simplified and business-friendly approach for streaming analytics based on Oracle Event Processor. The first release of Stream Explorer was published in the Spring of 2015. Version 2 of the product is imminent, with a number of important enhancements. Stream analytics based on Stream Explorer will be included as first class service in ICS, SOA CS and IOT CS.
The new functionality in V2 includes easy to understand visualization of real-time data (BAM) and better embedded charts are one of these. Formulas, functions and expressions can be defined to derive values for new attributes in explorations. More patterns are supported out of the box and custom patterns can be added as well. Business Rules can be integrated into explorations to filter events from streams based on logic the business users can easily maintain. Stream Explorer is made aware of geo locations and spatial relations; this can be used to implement geo-fencing – relating events to locations and their immediate surroundings. Stream Explorer is hooked up with the worlds of Big and Fast Data, starting out with integration with Apache Spark and Kafka. Leveraging this integration, support will be added to Stream Explorer for machine learning around Clustering, Classification, and Regression models.
More attention is given to the use of Stream Explorer not only for events from the physical world of devices and things on the IoT or the real time realm of fast data on social networks and web sites, but also for the analysis of integration flows and business processes executed in SOA Suite and BPM Suite itself. Process analytics published to BAM and especially the data extracted by the Integration Insight collector can easily be processed in Stream Explorer to help with real time monitoring and early warnings against imminent SLA violations or looming infrastructure problems. Stream Explorer can also analyze business logic in real time such as unanswered requests, overdue events, unfinished tasks and suspicious transactions. With thousands of customers using SOA Suite for business critical integration flows and business processes, there is a lot of potential for Stream Explorer in this area.
API Management
API is one of the hottest terms in the IT industry at this point. APIs are the reusable programmatic interfaces where systems meet and interact. APIs represent the services that systems expose to external consumers.
APIs are crucial to make integration scenarios work. SaaS applications should publish APIs to allow extensions and integrations – and should make proper documentation available about those APIs. The same applies to PaaS services that expose documented APIs for management activities. Among the Oracle Public Cloud Service with a vested interested in consuming APIs in addition to exposing them are the Mobile CS, ICS and SOA CS; on premises, important API stakeholders are Service Bus and SOA Suite.
The current API Manager (based on Service Bus) and API Manager Cloud Service facilitate the creation of APIs that expose the functionality of backend systems and services. These APIs are published for use by application developers and are managed and monitored at runtime. This product is a good start – but it is not much more than that. During the OOW 2015 conference, the integration team made a strong statement about APIs – their management and governance and the platform that Oracle wants to provide to that end that will be the best in the industry. It sounded like a clear stake in the ground: the Oracle Integration team means business when it comes to APIs.
The API Platform Cloud Service was announced for publication some time in 2016. A service with focus on the API consumer (the application developer) and with the objective to help streamline, control and accelerate the API development lifecycle and provide a one-stop shop for API access to SaaS and on-premises services and data. The API Platform is positioned to support the full life cycle and all aspects, from planning the API in the first place, designing and implementing it, publishing and marketing the API, handling operations and the actual usage including tracking, billing, rating and life cycle management.
Aspects for the API Platform mentioned by Oracle are that is should enable to easily build and customize APIs, usage plans, monetization approaches for different consumer types, and also to monitor and control API usage by consumers and to proactively engage consumers about new features, usage issues. Around security, the platform should ensure that only the right consumers have access to the right resources, it has to defend against potential attacks against your infrastructure. The API Platform should play a part in service optimization: dynamically improve capacity, response time and uptime of services and route requests to the optimal service. The platform should help with API Analysis: understand how your API is discovered, used and by whom and keep aware of important happenings in your API ecosystem.
The API Platform is designed to cover both cloud and on premises components as well as the bridge between the two – the API Gateway into the on premises world. The platform will be extensible, meaning that custom Gateway APIs, analytics systems, policies and UIs can be plugged in – alongside the out of the box capabilities.
To be honest, I am not sure exactly what this API Platform will evolve into. However, the objectives are good and the drive behind this is passionate. Oracle gets the potential of APIs and the necessity of having a good solution for dealing with them. The many APIs in the Oracle Public Cloud – SaaS and PaaS – are a great starting point: implement governance for those APIs, and the pudding has been proven alright!
