Demonstration of Oracle Stream Explorer for live device monitoring – collect, filter, aggregate, pattern match, enrich and publish

This article describes a use case for Oracle Stream Explorer – Oracle’s business user friendly interface on top of OEP – Oracle Event Processor. We assume a large number of devices – such as printers, copiers, sensors, detectors, coffee machines – spread across the globe – and the cloud.

All devices continuously report their status, by sending a message every other second that contains their device identifier, a code that can indicate the healthy status or an error and some additional details. The sheer number of devices combined with the continuous stream of reports they sent in set the challenges perimeters within which we have to implement fast and effective monitoring. Our specific challenge is: “whenever a device reports an error code three times within 10 seconds, we consider that device broken, and action should be taken” (that also means that we do not spring into action on the first or even second fault report from a device). Additionally: we only require a single action for a broken device – once the action is initiated, we do not have to start an action again for that same device – unless of course it is broken again at a much later point in time.

The concrete implementation described in this article looks as follows:

For the sake of a simple demonstration, we read device message reports from a csv file, instead of a live stream such as a JMS destination or an HTTP channel. Note that the Stream Explorer implementation would be exactly the same for these other stream types. Stream Explorer processes the device signals. For signals that satisfy the requirements of a broken device, the information is enriched from a database with device details – such as the physical location of the device – and finally an EDN event is composed and published. This event is consumed by a SOA Composite application in the SOA Suite 12c environment. This composite can virtually do anything, including assigning a task, starting a BPM process or sending an email.

The implementation described in this article is also demonstrated in a video on YouTube:

Implementing the Stream Explorer application

With Stream Explorer – everything starts from a Stream – a source of events or messages such as a JMS Queue or Topic, the SOA Suite Event Delivery Network, an HTTP channel or a CSV file such as in this case. Through one or more Explorations – that each can do filtering, aggregation, enrichment and pattern matching – finally conclusions can be published to a target. Targets can be JMS destinations, HTTP channels, a CSV file and the Event Delivery Network of SOA Suite.

In a number of steps, we will go from the CSV file with device signals to the EDN events that represent broken devices.

The first step will be an exploration that filters the non-ok signals from the stream:

The second step will find failing devices by counting the number of non-ok signals in a 10 second period and filtering on any device with a count greater than or equal to 3:

Next, to prevent any failing device from being reported more than once (in a certain period of time) we perform deduplication, using one of the special patterns shipped out of the box in Stream Explorer:

The remaining messages report a unique failing device and we need to enrich those messages with details about the device location, taken from a Reference defined for a database table:

The enriched messages are routed to a target: an EDN event that is published to the SOA Suite runtime whose address is configured:

The next sections show – just as the video does – how these explorations are created in Stream Explorer.

Open Stream Explorer and log in

Open the Catalog.

DeviceSignals Stream and NonOkSignals Exploration

Create a new Stream, of type CSV.

Press Next. Select the CSV file:

You may want to briefly inspect this file:

Back in the wizard, refine the Data Shape definition and give it a name:

Press Create to complete the Stream definition.

Next, the Exploration wizard is started. Set the name of the Exploration to NonOkSignals:

Press the Create button.

The Exploration editor is showing and the first messages read from the CSV file are produced in the exploration:

Define a filter for the exploration, to only produce messages for non ok error codes:

Publish the exploration and return to the catalog.

Exploration for Failing Devices

Create a new Exploration. Call it FaultyDevice[s] and use NonOkSignals as the source.

Press Create to navigate to the editor for the exploration.

Specify a summary: count DeviceId , group by DeviceId. Next, add a filter, to produce results only when for a DeviceId we have counted 3 or more NonOkSignals. Finally, specify a time window over which to calculate the count. Set the range of the window to 20 seconds and set the evaluation frequency to [once every] 3 [seconds].

Publish the exploration. Note that if you wait for some time, you will see devices being reported in the Live Output Stream section of this page:

Return to the Catalog.

Deduplicate Devices

Create a Pattern – an exploration based on a predefined pattern. Pick the Eliminate Duplicates pattern.

Select FaultyDevice as the source for this pattern exploration. Select DeviceId as the key to determine the uniqueness by (eliminate the second and subsequent events from the FaultyDevice stream). Set the window to 1 minute. This means that any subsequent events for a device are blocked during 1 minute after the initial event for the device. After that minute, the slate is cleared for that device.

 

Publish this exploration too.

Return to the catalog.

Enrich the FailingDeviceEvents and Pulish them as EDN Event

Create a Reference:

Note: before doing this, a Data Source to the schema that contains the database table should have been set up in OEP – either through the WLEvent Visualizer or directly in the OEP configuration file:

The table used as a reference is shown here:

Define the Reference – set its name and its type:

Then select the table that provides the data for this reference:

And press Create.

Now create a new exploration – our last one in this article:

Set its name and select Exploration4 – the name auto-assigned to the [Eliminate Duplicates] Pattern based exploration – as the source:

Add the Reference DeviceDetails as a second source for this exploration:

Then specify the correlation condition:

This completes the logic for the Exploration. We do need to add a target to it – to make the outcomes from this Exploration produce EDN events.

Click on Configure Target. The select EDN as the target type in the wizard:

Type in the connection details for the SOA Suite managed server. The EDN event definitions will be loaded.

Note: at this stage, there seems to be an issue when the EDL files import XSD definitions that themselves import XSD definitions. Additionally, it seems that EDN events with nested elements are not handled well by Stream Explorer.

In this case, my SOA Suite domain contains just a single EDN event definition that satisfies the conditions mentioned above. Select this definition:

Next, provide the mapping between properties in the current exploration and the elements in the EDN event:

And press Finish.

Publish the Exploration to make it active.

After some time, new failing devices will have be spotted and EDN events will have been published. These EDN events in this have trigger the SOA Composite DeviceMaintenance. Here we see an example of some instances as well as the details of one of these instances.

The device signals picked by Stream Explorer and processed through four explorations result in this SOA composite being invoked – only once – for every device that has found to be failing. The hard in terms of live observing a great number of devices simultaneously and continuously is taken on by Stream Explorer. And configuring these explorations turns out to be quite straightforward and rather declarative.

Resources

Zip with CSV file with device data and SQL scripts for the creation of the device details table: deviceMonitoringResources .