Demonstration of Oracle Stream Explorer for live device monitoring – collect, filter, aggregate, pattern match, enrich and publish

1

This article describes a use case for Oracle Stream Explorer – Oracle’s business user friendly interface on top of OEP – Oracle Event Processor. We assume a large number of devices – such as printers, copiers, sensors, detectors, coffee machines – spread across the globe – and the cloud.

image

All devices continuously report their status, by sending a message every other second that contains their device identifier, a code that can indicate the healthy status or an error and some additional details. The sheer number of devices combined with the continuous stream of reports they sent in set the challenges perimeters within which we have to implement fast and effective monitoring. Our specific challenge is: “whenever a device reports an error code three times within 10 seconds, we consider that device broken, and action should be taken” (that also means that we do not spring into action on the first or even second fault report from a device). Additionally: we only require a single action for a broken device – once the action is initiated, we do not have to start an action again for that same device – unless of course it is broken again at a much later point in time.

image

The concrete implementation described in this article looks as follows:

image

For the sake of a simple demonstration, we read device message reports from a csv file, instead of a live stream such as a JMS destination or an HTTP channel. Note that the Stream Explorer implementation would be exactly the same for these other stream types. Stream Explorer processes the device signals. For signals that satisfy the requirements of a broken device, the information is enriched from a database with device details – such as the physical location of the device – and finally an EDN event is composed and published. This event is consumed by a SOA Composite application in the SOA Suite 12c environment. This composite can virtually do anything, including assigning a task, starting a BPM process or sending an email.

The implementation described in this article is also demonstrated in a video on YouTube:

Demo of Stream Explorer in action for monitoring devices

Implementing the Stream Explorer application

With Stream Explorer – everything starts from a Stream – a source of events or messages such as a JMS Queue or Topic, the SOA Suite Event Delivery Network, an HTTP channel or a CSV file such as in this case. Through one or more Explorations – that each can do filtering, aggregation, enrichment and pattern matching – finally conclusions can be published to a target. Targets can be JMS destinations, HTTP channels, a CSV file and the Event Delivery Network of SOA Suite.

In a number of steps, we will go from the CSV file with device signals to the EDN events that represent broken devices.

image

The first step will be an exploration that filters the non-ok signals from the stream:

image

The second step will find failing devices by counting the number of non-ok signals in a 10 second period and filtering on any device with a count greater than or equal to 3:

image

Next, to prevent any failing device from being reported more than once (in a certain period of time) we perform deduplication, using one of the special patterns shipped out of the box in Stream Explorer:

image

The remaining messages report a unique failing device and we need to enrich those messages with details about the device location, taken from a Reference defined for a database table:

image

The enriched messages are routed to a target: an EDN event that is published to the SOA Suite runtime whose address is configured:

image

The next sections show – just as the video does – how these explorations are created in Stream Explorer.

Open Stream Explorer and log in

image

image

Open the Catalog.

DeviceSignals Stream and NonOkSignals Exploration

Create a new Stream, of type CSV.

image

Press Next. Select the CSV file:

SNAGHTML14b4e240

You may want to briefly inspect this file:

image

Back in the wizard, refine the Data Shape definition and give it a name:

image

Press Create to complete the Stream definition.

Next, the Exploration wizard is started. Set the name of the Exploration to NonOkSignals:

image

Press the Create button.

The Exploration editor is showing and the first messages read from the CSV file are produced in the exploration:
image

Define a filter for the exploration, to only produce messages for non ok error codes:

image

Publish the exploration and return to the catalog.

Exploration for Failing Devices

Create a new Exploration. Call it FaultyDevice[s] and use NonOkSignals as the source.

image

Press Create to navigate to the editor for the exploration.

Specify a summary: count DeviceId , group by DeviceId. Next, add a filter, to produce results only when for a DeviceId we have counted 3 or more NonOkSignals. Finally, specify a time window over which to calculate the count. Set the range of the window to 20 seconds and set the evaluation frequency to [once every] 3 [seconds].

image

Publish the exploration. Note that if you wait for some time, you will see devices being reported in the Live Output Stream section of this page:

image

Return to the Catalog.

Deduplicate Devices

Create a Pattern – an exploration based on a predefined pattern. Pick the Eliminate Duplicates pattern.

image

Select FaultyDevice as the source for this pattern exploration. Select DeviceId as the key to determine the uniqueness by (eliminate the second and subsequent events from the FaultyDevice stream). Set the window to 1 minute. This means that any subsequent events for a device are blocked during 1 minute after the initial event for the device. After that minute, the slate is cleared for that device.

 

Publish this exploration too.image

Return to the catalog.

Enrich the FailingDeviceEvents and Pulish them as EDN Event

Create a Reference:

image

Note: before doing this, a Data Source to the schema that contains the database table should have been set up in OEP – either through the WLEvent Visualizer or directly in the OEP configuration file:

image

The table used as a reference is shown here:

image

Define the Reference – set its name and its type:

image

Then select the table that provides the data for this reference:

image

And press Create.

Now create a new exploration – our last one in this article:

image

Set its name and select Exploration4 – the name auto-assigned to the [Eliminate Duplicates] Pattern based exploration – as the source:

image

Add the Reference DeviceDetails as a second source for this exploration:

image

Then specify the correlation condition:

image

This completes the logic for the Exploration. We do need to add a target to it – to make the outcomes from this Exploration produce EDN events.

Click on Configure Target. The select EDN as the target type in the wizard:

image

Type in the connection details for the SOA Suite managed server. The EDN event definitions will be loaded.

image

Note: at this stage, there seems to be an issue when the EDL files import XSD definitions that themselves import XSD definitions. Additionally, it seems that EDN events with nested elements are not handled well by Stream Explorer.

In this case, my SOA Suite domain contains just a single EDN event definition that satisfies the conditions mentioned above. Select this definition:

image

Next, provide the mapping between properties in the current exploration and the elements in the EDN event:

image

And press Finish.

Publish the Exploration to make it active.

After some time, new failing devices will have be spotted and EDN events will have been published. These EDN events in this have trigger the SOA Composite DeviceMaintenance. Here we see an example of some instances as well as the details of one of these instances.

image

The device signals picked by Stream Explorer and processed through four explorations result in this SOA composite being invoked – only once – for every device that has found to be failing. The hard in terms of live observing a great number of devices simultaneously and continuously is taken on by Stream Explorer. And configuring these explorations turns out to be quite straightforward and rather declarative.

Resources

Zip with CSV file with device data and SQL scripts for the creation of the device details table: deviceMonitoringResources .

About Author

Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Kubernetes & Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press book Oracle SOA Suite 12c Handbook. Frequent presenter on user groups and community events and conferences such as JavaOne, Oracle Code, CodeOne, NLJUG JFall and Oracle OpenWorld.

1 Comment