Live Monitoring of SOA Suite Service Execution with Stream Explorer – leveraging Custom OWSM Policy and JMS

4

This article demonstrates how live monitoring of SOA Suite service execution can be implemented using a custom OWSM policy that reports to a JMS queue and with a simple Stream Explorer exploration that aggregates these JMS messages:

image

The ingredients are:

  • a SOA Suite 12c runtime environment
  • a Stream Explorer installation

and two files available with this article:

  • CustomPolicyAssertionArchive.jar (that contains the custom policy implementation)
  • AMIS_Custom_Policies.zip (that contains the policy definition)

and a JSON configuration of the policy binding.

Using the ingredients we will walk through the following stages and steps:

Stage 1:

  • Copy JAR file to the WLS_SOA_domain/lib directory (and restart the domain)
  • Import the ZIP file into the EM FMW Control (to define the new policy)
  • Attach the policy to a SOA Composite and configure the operations map property
  • Invoke the SOA Composite and check the SOA domain log file (to find service execution reports logged in the file)

Stage 2:

  • Configure JMS artifacts to provide the conduit for the service execution reports (JMS Server, Module, Connection Factory and Queue)
  • Update the configuration of the policy binding with the JMS destination
  • Invoke the SOA Composite and check the JMS Queue monitoring page in the WebLogic Administration Console (to find messages produced for web service calls)

Stage 3:

  • Run Stream Explorer and create a Stream on top of the JMS Queue
  • Create an Exploration on top of the Stream to report aggregated service execution metrics (per service and per operation over the last 30 minutes)
  • Invoke several operations on the SOA Composite (several times) and see how the StreamExplorer exploration is updated to provide the latest insight

This provides the foundation for a wide range of applications of the Service Execution Reporter policy along with Stream Explorer. In future articles, we will see the type of focused monitoring this foundation enables us to perform.

 

Stage 1 – Basic application of Service Execution Reporter policy

This previous article describes how the Service Execution Reporter policy is developed. The policy is deployed to a JAR file that you can download here: CustomPolicyAssertionArchive (extract it from the ZIP file). The configuration of the policy is laid down in a ZIP file that you can download here: AMIS_Custom_Policies.

The JAR file has to be copied to the WLS_SOA_domain/lib directory. Using the target information in the EM FMW Control – see next figure – I find out about the exact the file location for the WebLogic domain that hosts the SOA Suite:

image

the lib directory under this domain home is where the jar file should be moved.

Subsequently, the domain has to be restarted in order to make the contents of the jar file available in the SOA Suite run time.

Import the ZIP file into the EM FMW Control (to define the new policy)

Start EM FMW Control.

image

navigate to Web Logic Domain – soa_domain | Web Services | WSM Policies.

Click on Import

image

Import the by clicking on Import

image

and selecting the right zip file:

image

The report back:

image

and the policy is listed:

image

 

Attach the policy to a SOA Composite and configure the operations map property

Open the SOA Composite, such as the FlightService composite shown below. Click on the Service Binding to which the policy is to be attached:

image

Open the Policies tab:

image

Click on the button Attach/Detach to open the dialog where policies can be attached to the service binding.

image

Select the amis/monitoring policy. Click on Attach to bind this policy to the service binding.

Click on OK to confirm the policy attachment.

Click on Override Policy Configuration to set the property values that apply specifically to this policy attachment:

image

 

The properties that are defined in the policy configuration file – SOASuiteServiceExecutionReporterPolicyFile.xml – are listed and the current values are shown. These values can now be overridden for this attachment of the policy to the FlightService.

image

The full value of the operationsMap property in this case is:

{
"getFlightDetailsRequest" : {
"operation" : "getFlightDetails",
"oneWay" : "false",
"request" : {
"doReport" : "true",
"payload" : [
{
"name" : "carrierCode",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsRequest/flig:FlightCode/com:CarrierCode",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
},
{
"name" : "flightNumber",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsRequest/flig:FlightCode/com:FlightNumber",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
},
"response" : {
"doReport" : "true",
"payload" : [
{
"name" : "flightStatus",
"xpath" : "/soap:Envelope/soap:Body/flig:getFlightDetailsResponse/flig:FlightStatus",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
}
},
"retrievePassengerListForFlightRequest" : {
"operation" : "retrievePassengerListForFlight",
"oneWay" : "false",
"request" : {
"doReport" : "true",
"payload" : [
{
"name" : "carrierCode",
"xpath" : "/soap:Envelope/soap:Body/flig:retrievePassengerListForFlightRequest/flig:FlightCode/com:CarrierCode",
"namespaces" : [
{
"prefix" : "soap",
"namespace" : "http://schemas.xmlsoap.org/soap/envelope/"
},
{
"prefix" : "flig",
"namespace" : "com.flyinghigh/operations/flightservice"
},
{
"prefix" : "com",
"namespace" : "com.flyinghigh/operations/common"
}
]
}
]
},
"response" : {
"doReport" : "true"
}
}
}

Obviously, you will have to provide the values that make sense for the services you want to the attach the policy to. Note: if you do not define the operationMap property for a particular policy binding, the service execution is reported. However, these reports obviously cannot report the operation name (only the message type) nor any values from the payload.

Click on Apply to confirm the property values.

At this point, the policy is primed for action for the FlightService.

Invoke the SOA Composite and check the SOA domain log file (to find service execution reports logged in the file)

By invoking the various FlightService operations, we can now see the policy in action.

image

The effect of this call is reported by the custom policy in the log-file:

image

A call to another operation results in a similar report:

image

in the log file:

image

The third operation – sendFlightStatusUpdate – is not configured at all in the operationsMap property. When this operation is invoked:

image

The report:

SNAGHTML1007f5a8

Stage 2 – Configuration of resources to route Service Execution Reports to JMS

The reports produced by the policy can be reported to a JMS destination in addition to the log file output. And we need that. So we first need to prepare a simple JMS Queue that we can next configure on the policy to have the JMS reporting going.

Open the WebLogic Administration Console. Open the Services | Messaging node in the Domain Structure Navigator. Create a new JMS Server:

image

Set the name. Then press Next. Select the managed server running the SOA Suite (the engine that runs the SOA Composite applications) as the target.

image

Press Finish.

image

Click on the Services | Messaging | JMS Modules node. Click on the New button to create a new JMS Module.

image

Set the name of the JMS module:

image

Click on Next.

Select the managed server running the SOA Suite as the target for the JMS Module:

image

and press Next.

image

Check the checkbox and press Finish.

image

Open the tab Subdeployments:

image

Click on New to create  a new subdeployment. Set the name:

image

And click on Next.

Select the JMS Server that was created earlier on as the target:

image

Click Finish:

image

Open the Configuration tab. Click on the new button to  create the Connection Factory:

image

Select the right radio button and click Next.

image

Set the name and the JNDI Name:

image

and click Next.

The target for the JMS Module is shown:

image

Click Finish. Create a new resource of type Queue:

image

Set the name and the JNDI Name:

image

Press Next.

Select the appropriate subdeployment and JMS Server (those that were created earlier):

image

Press Finish.

All four JMS artifacts are now created:

image

 

Update the configuration of the policy binding with the JMS destination

The policy was initially uploaded with a global configuration that includes the properties JMSDestination and JMSConnectionFactory set to empty strings. To configure the appropriate JMS artifact references, open the EM FMW Control and navigate to Web Logic Domain – soa_domain | Web Services | WSM Policies.

image

Locate the policy amis/monitoring. Click on Open link. Open the Assertion tab and click on Configuration.

image

Set the properties JMSDestination and JMSConnectionFactory  to “jms/ServiceExecutionReportingQueue” and “jms/ServiceExecutionReportingCF” respectively :

image

Click OK to apply these values.

 

Invoke the SOA Composite and check the JMS Queue monitoring page in the WebLogic Administration Console

From SOAP UI make one call to the service by the SOA composite that has the Service Execution Reporter attached.

image

Both the request en response message will pass through the policy and trigger both an entry in the log file as well as a message sent to the JMS queue. We can verify the latter in the WebLogic Admin Console by checking the Monitoring tab for the queue:

image

Drilling down provides a little more insight into the messages that were published to the queue:

image

image

Invoke the SOA Composite’s service from SoapUI a few more times and the message count on the Monitoring tab for the JMS queue will increase further.

Clearly we have established JMS publication of a MapMessage for each service execution of the FlightService (and any other service that has the ServiceExecutionReporter policy attached.

 

Stage 3 – Monitor Service Execution using Oracle Stream Explorer explorations

The final piece of today’s puzzle is the step from the JMS Queue with its MapMessags to the Stream Explorer exploration that provides a count of recent service executions.

Run Stream Explorer

image

and create a Stream on top of the JMS Queue. Click on Create New Item and Select Stream as the new Item Type to create.

Enter a name and a description and select the Stream’s source type as JMS:

image

Click Next.

Configure the JMS destination (the queue to use as the source) as shown next:

image

The URL is for the WebLogic managed server that hosts the JMS Queue; the admin username and password are used here to access the JMS Queue.

Click Next.

Specify the name for the ‘shape’ – the data structure in Stream Explorer to capture the events from the stream.

image

Select Manual Mapping and define the properties of the shape – corresponding with the properties written in the JMS Map – which are:

service, operation, ecid, stage, executionTimestamp – and whichever payload elements are configured for extraction in the operationsMap.

image

Press Create to create the Stream.

The wizard for creating the Exploration kicks in immediately after completing the Stream definition.

image

Specify name and description and optionally some tags.

image

Press Create. This takes you to the Exploration editor.

A lot is specified for the exploration:

  • The Summary to calculate is a count of the number of events – grouped by service and operation.
  • Filter only the events that have the stage set to request
  • Calculate the Summary over the last one hour and update the count every 10 seconds

image

Invoke several operations on the SOA Composite (several times) and see how the StreamExplorer exploration is updated to provide the latest insight:

 

image

Here we see how first (bottom two entries) some calls were made to the operation retrievePassengerListForFlight – the last two within 10 seconds of each other because an entry with COUNT_of-service equal to 2 is missing. Subsequently, up to 7 calls were made to the getFlightDetails operation – not interrupted by calls to other operations in the FlightService. Noe that calls 5 and 6 were close together – within 10 seconds of each other.

Let’s attach the policy to another SOA composite – just for kicks:

image

image

image

image

Then invoke an operation on the ConversionService composite:

image

followed by a few calls to the FlightService – and see the result in the Stream Explorer report:

image

 

It should hopefully be clear now that we have an way to observe and analyze service execution behavior using Stream Explorer and leveraging the output from the custom Service Execution Reporter policy.

image

About Author

Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Kubernetes & Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press book Oracle SOA Suite 12c Handbook. Frequent presenter on user groups and community events and conferences such as JavaOne, Oracle Code, CodeOne, NLJUG JFall and Oracle OpenWorld.

4 Comments

  1. hi

    i need some advice we have oracle stream which are using dsq the old way we what to design our system using soa can where can we start to start to design new structure

    desmond

    • Lucas Jellema on

      Hi Alex,
      The concept will definitely work on 11g. I am not entirely sure whether the JAR and ZIP files can be installed on 11g as is – I believe the configuration files for custom policies may have changed a little between 11g and 12c. You can find all source code on GitHub (https://github.com/lucasjellema/OWSM-custom-assertion-serviceexecutionreporter) and try out tinkering with it. I may get round to doing that myself at some point – if there seems to be enough interest in that.

      kind regards
      Lucas

      • One more question Lucas – is this just a use case for Stream Explorer or does it bring any benefit to monitoring SOA Suite Services executions?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.