Use OCI Monitoring, Alarms and Notifications for Your Own Custom and Functional Metrics

In earlier articles, I have discussed features in Oracle Cloud Infrastructure for monitoring metrics published by virtually all OCI services. These metrics get published when functions are invoked, files are written, the API Gateway handles a request, events are published, a user is created and a network transfers a packet. The metrics are gathered in a big metrics lake inside OCI where they are retained for 14 days. By querying the Monitoring service for this data, you can understand how well the systems and processes are working to achieve the service levels you commit to your customers. Metrics can be inspected in the console using predefined charts and using the metrics explorer. Metrics can also be retrieved through the CLI and the REST API. Alarms can be defined with query conditions on the metrics; when an Alarm condition is satisfied, the alarm is ‘sounded’: it will publish a notification that results in an email being sent or a web hook being invoked.

It is a common desire to not only monitor the behavior of VMs, network resources and other technical resources, but to also – and even primarily – keep an eye on the functional behavior of the system. To learn about the [trends and fluctuations in time of the] number of orders per product category and country, the increase in the number of tweets on the Corona virus in a city or province and the unexpected absence of traffic on a web site or API. This functional or business activity monitoring is support by the OCI Monitoring service: You can publish your own metrics to Monitoring using the API. You can view charts of your published metrics using the Console , query metrics using the API, and set up alarms using the Console or API.You can access your published custom metrics the same way you access any other metrics stored by the Monitoring service.

Metrics have a name and a set of one or more data points (combinations of a numerical value and a timestamp and optionally a count in case multiple observations with the same value are made at the same time). Metrics are associated with a metrics namespace and optionally with a resource group (which could for example be a stand alone application, division, realm). Metrics can be further qualified with up to 16 dimensions as well as metadata (key / value pairs that further describe the data point).

Metrics should be defined with aggregation in mind: while custom metrics can be posted as frequently as every second , the minimum aggregation interval is one minute – in reports, charts and alarms.

In this article, I will show how to publish custom metrics in order to do functional application monitoring. These custom metrics appear in charts in the console, can be retrieved for reporting through the APIs and are the basis for functional alarms.

The steps I will discuss:

1. Define policies for inspecting metrics in a compartment and for publishing custom metrics to the OCI Monitoring service
2. Publish Custom Metrics through the OCI CLI
3. Inspect custom metrics through the OCI Monitoring | Metrics Explorer in the browser console
4. Inspect custom metrics through the OCI CLI
5. Define an Alarm and corresponding Notification on custom metrics – to be notified when a functional event of note takes place
6. Publish Custom Metrics that trigger the Alarm

 

1. Define policies for inspecting publishing metrics

Users that want to inspect metrics produced in the context of a specific compartment need to be in a group that has been granted permissions to do so through policies.

To read metrics and inspect metrics definitions in a compartment, the required policy statements are (see docs):  “read metrics in compartment” and “inspect metrics in compartment” respectively. In order to publish custom metrics in a custom metrics namespace, the policy statement required is: “use metrics in tenancy where target.metrics.namespace=’mycustomnamespace’ “. To create an alarm, the needed policy statement are “manage alarms in tenancy” and “read metrics in tenancy”. To create a notification topic to link the alarm to, the policy statement is “manage ons-topics”.

Note: To reduce the scope of access to a particular compartment, specify that compartment instead of the tenancy in the policy statement.

I will be using an OCI user that is a member of the group lab-participants. The policy statements shown in the screenshot cover the requirements for this user.

2. Publish Custom Metrics through the OCI CLI

Custom Metrics can be published through the command line interface as well as the REST API and the SDKs. They cannot be published through the OCI console. Documentation on the OCI CLI route are here. The metrics to be published are defined in a JSON document. The easiest way of publishing custom metrics through the OCI CLI is by creating a file that contains this JSON document and then referring to that document in the OCI CLI call. This call looks like this:

oci monitoring metric-data post --endpoint https://telemetry-ingestion.us-ashburn-1.oraclecloud.com --metric-data file://./custom-metrics.json

The endpoint parameter specifies the endpoint for the home region of my cloud tenancy. The file that contains the JSON definition of the custom metrics is called custom-metrics.json.  Here is the contents of that file – that represents orders of products, presumably in a web shop:

[
    {
      "compartmentId": "$compartmentId",
      "datapoints": [
        {
          "count": 10,
          "timestamp": "2020-02-08T04:18:01+00:00",
          "value": 5.0
        },
        {
          "count": 3,
          "timestamp": "2020-02-08T05:11:01+00:00",
          "value": 10.0
        }
      ],
      "dimensions": {
        "product": "ball",
        "country": "NL"
      },
      "metadata": {
        "category": "toys",
        "note": "national holiday"
      },
      "name": "productOrder",
      "namespace": "mymetricsnamespace",
      "resourceGroup": "divisionX"
    },
    {
      "compartmentId": "$compartmentId",
      "datapoints": [
        {
          "count": 7,
          "timestamp": "2020-02-08T03:22:01+00:00",
          "value": 3.0
        },
        {
          "count": 11,
          "timestamp": "2020-02-08T05:08:03+00:00",
          "value": 2
        }
      ],
      "dimensions": {
        "product": "The Road to Nowhere",
        "country": "FR"
      },
      "metadata": {
        "category": "books",
        "note": "start second semester"
      },
      "name": "productOrder",
      "namespace": "mymetricsnamespace",
      "resourceGroup": "divisionY"
    }
  ]

You will note how the custom metrics are published in the context of the metrics namespace mymetricsnamespace  – this name can freely be defined. Note that it is explicitly named on the policy statement regarding publishing custom metrics.

The metrics are published in the context of a compartment. They are associated with two dimensions – product and country- and with a resourceGroup – either DivisionX or DivisionY. Data for different resourceGroups is treated as completely separate in the OCI Monitor. Additional metaData – key-value pairs – are defined as well.

Note: the metrics cannot be too old (the timestamp should be less than two hours in the past) nor too young (not more than 10 minutes in the future). If you want to use this JSON document in your own experiments, you will need to edit both the OCID for the compartment and the timestamp values for the metrics.

This screenshot shows the command executed in the CLI – and the response that looks quite encouraging:

After this statement was processed, my first custom metrics are stored in the OCI Metrics Data Lake.

 

3. Inspect custom metrics through the OCI Monitoring | Metrics Explorer

Open the Metrics Explorer in the OCI Console:

This tool will open with a incomplete query that you need to complete. You need to set the compartment and the metrics namespace. In order to see metrics linked to a resource group, that resource group also has to be explicitly selected. Finally, the metric of interest – productOrder in this example – has to be set. The aggregation interval is set (the drop down list offers 1m, 5m and 1 hour; when you switch to Advanced Mode you can customize the interval in the MQL query definition). The Statistic (aggregation function) is to be selected – Sum is a common choice, although many options are available (min, max, mean, count, rate, P50, P90, P95, P99).

The next screenshot shows the graphical representation of the few custom data points I have just published through the CLI. Nothing impressive – but encouraging nevertheless. I did manage to get my own metrics in the OCI Monitoring framework.

The Data Table view shows the individual data points for the selected resource group DivisionX and the selected metric productOrder.

 

4. Inspect custom metrics through the OCI CLI

The custom metrics can be inspected through the CLI and the API just as they can be searched in the console. A quick example of how we can check our custom metrics through the CLI:

oci monitoring metric-data summarize-metrics-data --from-json file://./get-metrics.json

The query is defined in the JSON document in file get-metrics.json; it is defined as follows:

{
  "compartmentId": "$compartmentId",
  "endTime": "2020-02-08T20:00:00+00:00",
  "namespace": "mymetricsnamespace",
  "queryText": "productOrder[1m].sum()",
  "resolution": "5m",
  "resourceGroup": "divisionX",
  "startTime": "2020-02-07T23:00:00+00:00"
}

The same parameters and query text are used as in the Metrics Explorer in the console.

The result is a JSON document that in this example contains aggregated data points for the productOrder metric in namespace mymetricsnamespace and only for resource group divisionX. In this case, data points have been aggregated per 1 minute, resulting in three different timestamps. The values in the aggregated data points are the summation of the values reported for productOrders for product==ball and country==NL.

 

 

5. Define an Alarm and Notification on custom metrics

In a recent article I have introduced Alarms and Notifications on Oracle Cloud Infrastructure. I will not repeat that introduction – but quickly create an Alarm for the custom metric productOrder. I want to raise an Alarm if and when the number of productOrders in a 5 minute time period is higher than 100.

Here is the full definition of the Alarm:

Click on Save alarm to activate the alarm.

The Alarm is associated with the Notification Topic ExtremeProductOrderVolume, that was defined like this:

The topic is associated with on destination – my email address. When I created that association, I received an email inviting me to confirm the association:

With a link that took me here:

 

6. Publish Custom Metrics to trigger the Alarm

I have edited the custom-metrics.json file – to make sure that the metrics published should trigger the alarm, with higher values for the productOrder metrics and with fresh timestamps. I have then published the metrics from the CLI.

Here is the indication in the console of the triggering status of the Alarm:

and the details:

Here is the email I received as notification for the Alarm:

 

Summary

In this article, I demonstrated custom metrics in the OCI Monitoring Service. I showed how easy it is to publish custom metrics to OCI that describe functional aspects of applications and services and how these metrics can be inspected – using the Metrics Explorer – and how alarms can be defined in terms of these custom metrics in the same way that alarms and notifications can be defined for technical, OCI resource related metrics.

In this article, I used the OCI CLI to publish the metrics. The OCI CLI is not the only way to publish custom metrics. Another way is through the OCI REST API. In fact, the only way is through the OCI REST API – because the OCI CLI is using that API under the hood. It would be easy to create a function (on OCI FaaS) in Node that invokes the REST API to publish custom metrics. This function could be exposed through an API Gateway – private or public – to make it easily accessible for other functions.

With very simple, straightforward REST calls, custom metrics could be published into the OCI Monitoring Data Lake – available for inspecting, reporting and notifying. I will leave that for another time.

 

 

Resources

OCI Documentation – Publishing Custom Metrics: https://docs.cloud.oracle.com/en-us/iaas/Content/Monitoring/Tasks/publishingcustommetrics.htm

CLI Reference for publishing custom metrics: https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/2.9.1/oci_cli_docs/cmdref/monitoring/metric-data/post.html

REST API reference for publishing custom metrics: https://docs.cloud.oracle.com/en-us/iaas/api/#/en/monitoring/latest/MetricData/PostMetricData 

Policies governing publication and viewing metrics: https://docs.cloud.oracle.com/en-us/iaas/Content/Identity/Concepts/commonpolicies.htm#metrics-access

Blog Article: Oracle Cloud Infrastructure Resource Monitoring – Alarm triggers Notification when Metrics satisfy Condition – https://technology.amis.nl/2020/02/03/oracle-cloud-infrastructure-resource-monitoring-alarm-triggers-notification-when-metrics-satisfy-condition/