Live Monitoring of SOA Suite Service Execution with Stream Explorer – leveraging Custom OWSM Policy and JMS

This article demonstrates how live monitoring of SOA Suite service execution can be implemented using a custom OWSM policy that reports to a JMS queue and with a simple Stream Explorer exploration that aggregates these JMS messages:

The ingredients are:

• a SOA Suite 12c runtime environment
• a Stream Explorer installation

and two files available with this article:

• CustomPolicyAssertionArchive.jar (that contains the custom policy implementation)
• AMIS_Custom_Policies.zip (that contains the policy definition)

and a JSON configuration of the policy binding.

Using the ingredients we will walk through the following stages and steps:

Stage 1:

• Copy JAR file to the WLS_SOA_domain/lib directory (and restart the domain)
• Import the ZIP file into the EM FMW Control (to define the new policy)
• Attach the policy to a SOA Composite and configure the operations map property
• Invoke the SOA Composite and check the SOA domain log file (to find service execution reports logged in the file)

Stage 2:

• Configure JMS artifacts to provide the conduit for the service execution reports (JMS Server, Module, Connection Factory and Queue)
• Update the configuration of the policy binding with the JMS destination
• Invoke the SOA Composite and check the JMS Queue monitoring page in the WebLogic Administration Console (to find messages produced for web service calls)

Stage 3:

• Run Stream Explorer and create a Stream on top of the JMS Queue
• Create an Exploration on top of the Stream to report aggregated service execution metrics (per service and per operation over the last 30 minutes)
• Invoke several operations on the SOA Composite (several times) and see how the StreamExplorer exploration is updated to provide the latest insight

This provides the foundation for a wide range of applications of the Service Execution Reporter policy along with Stream Explorer. In future articles, we will see the type of focused monitoring this foundation enables us to perform.

 

Stage 1 – Basic application of Service Execution Reporter policy

This previous article describes how the Service Execution Reporter policy is developed. The policy is deployed to a JAR file that you can download here: CustomPolicyAssertionArchive (extract it from the ZIP file). The configuration of the policy is laid down in a ZIP file that you can download here: AMIS_Custom_Policies.

The JAR file has to be copied to the WLS_SOA_domain/lib directory. Using the target information in the EM FMW Control – see next figure – I find out about the exact the file location for the WebLogic domain that hosts the SOA Suite:

the lib directory under this domain home is where the jar file should be moved.

Subsequently, the domain has to be restarted in order to make the contents of the jar file available in the SOA Suite run time.

Import the ZIP file into the EM FMW Control (to define the new policy)

Start EM FMW Control.

navigate to Web Logic Domain – soa_domain | Web Services | WSM Policies.

Click on Import

Import the by clicking on Import

and selecting the right zip file:

The report back:

and the policy is listed:

 

Attach the policy to a SOA Composite and configure the operations map property

Open the SOA Composite, such as the FlightService composite shown below. Click on the Service Binding to which the policy is to be attached:

Open the Policies tab:

Click on the button Attach/Detach to open the dialog where policies can be attached to the service binding.

Select the amis/monitoring policy. Click on Attach to bind this policy to the service binding.

Click on OK to confirm the policy attachment.

Click on Override Policy Configuration to set the property values that apply specifically to this policy attachment:

 

The properties that are defined in the policy configuration file – SOASuiteServiceExecutionReporterPolicyFile.xml – are listed and the current values are shown. These values can now be overridden for this attachment of the policy to the FlightService.

The full value of the operationsMap property in this case is:

{
"getFlightDetailsRequest" : {
"operation" : "getFlightDetails",
"oneWay" : "false",
"request" : {
"doReport" : "true",
"payload" : [
{
"name" : "carrierCode",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsRequest/flig:FlightCode/com:CarrierCode",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
},
{
"name" : "flightNumber",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsRequest/flig:FlightCode/com:FlightNumber",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
},
"response" : {
"doReport" : "true",
"payload" : [
{
"name" : "flightStatus",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsResponse/flig:FlightStatus",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
}
},
"retrievePassengerListForFlightRequest" : {
"operation" : "retrievePassengerListForFlight",
"oneWay" : "false",
"request" : {
"doReport" : "true",
"payload" : [
{
"name" : "carrierCode",
"xpath" : "/soap:Envelope/soap:Body/flig:retrievePassengerListForFlightRequest/flig:FlightCode/com:CarrierCode",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
},
"response" : {
"doReport" : "true"
}
}
}

Obviously, you will have to provide the values that make sense for the services you want to the attach the policy to. Note: if you do not define the operationMap property for a particular policy binding, the service execution is reported. However, these reports obviously cannot report the operation name (only the message type) nor any values from the payload.

Click on Apply to confirm the property values.

At this point, the policy is primed for action for the FlightService.

Invoke the SOA Composite and check the SOA domain log file (to find service execution reports logged in the file)

By invoking the various FlightService operations, we can now see the policy in action.

The effect of this call is reported by the custom policy in the log-file:

A call to another operation results in a similar report:

in the log file:

The third operation – sendFlightStatusUpdate – is not configured at all in the operationsMap property. When this operation is invoked:

The report:

Stage 2 – Configuration of resources to route Service Execution Reports to JMS

The reports produced by the policy can be reported to a JMS destination in addition to the log file output. And we need that. So we first need to prepare a simple JMS Queue that we can next configure on the policy to have the JMS reporting going.

Open the WebLogic Administration Console. Open the Services | Messaging node in the Domain Structure Navigator. Create a new JMS Server:

Set the name. Then press Next. Select the managed server running the SOA Suite (the engine that runs the SOA Composite applications) as the target.

Press Finish.

Click on the Services | Messaging | JMS Modules node. Click on the New button to create a new JMS Module.

Set the name of the JMS module:

Click on Next.

Select the managed server running the SOA Suite as the target for the JMS Module:

and press Next.

Check the checkbox and press Finish.

Open the tab Subdeployments:

Click on New to create  a new subdeployment. Set the name:

And click on Next.

Select the JMS Server that was created earlier on as the target:

Click Finish:

Open the Configuration tab. Click on the new button to  create the Connection Factory:

Select the right radio button and click Next.

Set the name and the JNDI Name:

and click Next.

The target for the JMS Module is shown:

Click Finish. Create a new resource of type Queue:

Set the name and the JNDI Name:

Press Next.

Select the appropriate subdeployment and JMS Server (those that were created earlier):

Press Finish.

All four JMS artifacts are now created:

 

Update the configuration of the policy binding with the JMS destination

The policy was initially uploaded with a global configuration that includes the properties JMSDestination and JMSConnectionFactory set to empty strings. To configure the appropriate JMS artifact references, open the EM FMW Control and navigate to Web Logic Domain – soa_domain | Web Services | WSM Policies.

Locate the policy amis/monitoring. Click on Open link. Open the Assertion tab and click on Configuration.

Set the properties JMSDestination and JMSConnectionFactory  to “jms/ServiceExecutionReportingQueue” and “jms/ServiceExecutionReportingCF” respectively :

Click OK to apply these values.

 

Invoke the SOA Composite and check the JMS Queue monitoring page in the WebLogic Administration Console

From SOAP UI make one call to the service by the SOA composite that has the Service Execution Reporter attached.

Both the request en response message will pass through the policy and trigger both an entry in the log file as well as a message sent to the JMS queue. We can verify the latter in the WebLogic Admin Console by checking the Monitoring tab for the queue:

Drilling down provides a little more insight into the messages that were published to the queue:

Invoke the SOA Composite’s service from SoapUI a few more times and the message count on the Monitoring tab for the JMS queue will increase further.

Clearly we have established JMS publication of a MapMessage for each service execution of the FlightService (and any other service that has the ServiceExecutionReporter policy attached.

 

Stage 3 – Monitor Service Execution using Oracle Stream Explorer explorations

The final piece of today’s puzzle is the step from the JMS Queue with its MapMessags to the Stream Explorer exploration that provides a count of recent service executions.

Run Stream Explorer

and create a Stream on top of the JMS Queue. Click on Create New Item and Select Stream as the new Item Type to create.

Enter a name and a description and select the Stream’s source type as JMS:

Click Next.

Configure the JMS destination (the queue to use as the source) as shown next:

The URL is for the WebLogic managed server that hosts the JMS Queue; the admin username and password are used here to access the JMS Queue.

Click Next.

Specify the name for the ‘shape’ – the data structure in Stream Explorer to capture the events from the stream.

Select Manual Mapping and define the properties of the shape – corresponding with the properties written in the JMS Map – which are:

service, operation, ecid, stage, executionTimestamp – and whichever payload elements are configured for extraction in the operationsMap.

Press Create to create the Stream.

The wizard for creating the Exploration kicks in immediately after completing the Stream definition.

Specify name and description and optionally some tags.

Press Create. This takes you to the Exploration editor.

A lot is specified for the exploration:

• The Summary to calculate is a count of the number of events – grouped by service and operation.
• Filter only the events that have the stage set to request
• Calculate the Summary over the last one hour and update the count every 10 seconds

Invoke several operations on the SOA Composite (several times) and see how the StreamExplorer exploration is updated to provide the latest insight:

 

Here we see how first (bottom two entries) some calls were made to the operation retrievePassengerListForFlight – the last two within 10 seconds of each other because an entry with COUNT_of-service equal to 2 is missing. Subsequently, up to 7 calls were made to the getFlightDetails operation – not interrupted by calls to other operations in the FlightService. Noe that calls 5 and 6 were close together – within 10 seconds of each other.

Let’s attach the policy to another SOA composite – just for kicks:

Then invoke an operation on the ConversionService composite:

followed by a few calls to the FlightService – and see the result in the Stream Explorer report:

 

It should hopefully be clear now that we have an way to observe and analyze service execution behavior using Stream Explorer and leveraging the output from the custom Service Execution Reporter policy.