Composing DAPR Custom State Store Component for Oracle Cloud Object Storage

Dapr is a runtime framework for distributed applications as well as a personal assistant with many generic qualities relevant to any application. Through Dapr, applications can easily benefit from configuration and secret management, from interacting with dozens of external technologies in both inbound and outbound direction, from having messages received and sent on their behalf and from management of state – persisting and retrieving relevant data.

Dapr ships with many tens of implementations of the bindings, state store, pubsub and secret stores components. And Dapr is open – allowing anyone to create their own additional custom components that functional alongside the collection that is included in the open source project. Besides, your own custom Dapr components can – and really should be – offered back to the open source project to be validated and subsequently included as well.

I noticed that currently the Dapr project does not have any component implementations for Oracle technology – apart from MySQL database. None of the OCI services have been opened up through a Dapr component and not even the Oracle Database is supported. Of course the community – that is us or at least that is me. So if I feel that certain services and technologies should have a Dapr component to open up for everyone the synergy between Dapr and the particular services and technologies, why should I not take on that task myself?

In this article I will discuss the development of a custom Dapr state store component for the OCI Object Storage service. The state store component in Daprized applications is used  by the “Dapr personal assistant” to save application’s state and make that state available again to the application whenever required. The state is private to the application; only the application – through the Dapr personal assistant – should access that state. Different applications can each configure their own state store and use the same Dapr component to access these various state stores.

Note: when applications want to access and manipulate data in data stores shared with other processes, then this is typically done using a binding component and treating the data store as an external service (not exclusively owned by one Daprized application).

With the component introduced in this article, applications can create their personal state on OCI Object Storage; each value is stored as file named with key in the bucket that is created specifically for holding the application’s state – very much like the Dapr state store component for Azure BlobStorage.

The steps discussed in this article:

• create a new directory in the locally cloned Dapr components-contrib repository for OCI based state store components
• create a skeleton Go application for the new OCI ObjectStorage state store component – based on one of the existing state store components
• import the new component’s skeleton application into main Daprd application and add the registration of the component
• build the Dapr application into a fresh binary executable that includes the new skeleton component
• add a components.yaml configuring a state store instance based on the new component to the quickstart hello-world application
• run the hello-world quickstart and verify that the new skeleton component is correctly interacted with
• do a small celebration for this intermediate milestone
• create tests
• create meaningful implementations of the interface functions
• Init (along with the definition of the component configuration), Features, Ping
• New<Instance>
• Get
• Set
• Delete
• optionally: BulkSet and BulkDelete
• build, run tests, do linting, try out the new component
• write documentation
• submit the new custom component to the Dapr open source project to have it included – with Alpha status to start with – in a new release of Dapr

The preparation for creating a custom Dapr component includes creating a local development environment. I have described the process of preparing that environment in two earlier articles:

Customizing Dapr–preparing my environment

Extending Local Dapr Development Environment for Custom Components

The interaction from Go through the OCI Go SDK with the OCI Object Storage Service is introduced in this article:

Interacting with OCI Object Storage using Go SDK (first steps)

Create the home for the new Component

Dapr components are defined in the components-contrib repository – separate from the main Dapr repo. The main repo refers to the components repo and uses the definitions for all components. This means that new components can be introduced and existing ones evolved without impacting the core Dapr framework.

For my new component for the OCI Object Storage service I create a new directory in the locally cloned Dapr components-contrib repository. Just like GCP, Azure and AWS, I think OCI should have its own subdirectory that subsequently can contain components for various OCI services. Under state  create directory oci  that contains directory objectstorage. This directory will be the home for my new component.

Create a skeleton Go application for the new OCI ObjectStorage state store component

The new component is implemented through a Go application, typically a single file called after the package and directory: objectstorage.go. I have used an existing state store component as my example – to be specific: azure/blobstorage/blobstorage.go

I have created file oci/objectstorage/objectstorage.go as a skeleton that implements the Dapr statestore interface using the Azure BlobStorage component as my example. This means that the interface functions have been implemented, with almost meaningless but correct implementations including some log statements to be able to see at runtime if the component is engaged as expected.

The interface I have to implement:

An impression of the implementation:

Integrate New Component into Core Dapr

Dapr knows about the custom component – and is able to interpret references to the custom components in application components.yaml files – when that component has been registered at runtime with a simple statement in function main in dapr/cmd/daprd/main.go:

To make this call successful, the package that contains the new component needs to be imported into the main Daprd application, like this:

Build the Dapr application into a fresh binary executable with the new component

In the two earlier mentioned articles about setting up the local Dapr environment I have described how to build a new Dapr binary and replace the local one with the fresh incarnation. On my environment, this takes two statements:

make build

cp /home/lucas/dapr-dev/dapr/dist/linux_amd64/release/daprd ~/.dapr/bin

Configure a Quickstart Application to Leverage New Custom State Store

In order to check the correct integration of the new custom component in my local Dapr runtime, I have to see it in action, And for that to happen, I will run an application with Dapr and make sure that the components.yaml file used configures a state store instance based on the new component.

The components.yaml file contains:

This stipulates that Dapr should instantiate a state store using the oci.objectstorage component (that is now registered inside Daprd in the main function as discussed before).

This file is created in the hello-world\node folder of the quickstart hello-world application.

Run the Daprized Hello-World Quickstart application and see the Custom State Store component in action

I now run the hello-world quickstart application from the node directory that contains both app,.js and components.yaml and I include a reference to the current directory as the components-path:

dapr run –app-id nodeapp –app-port 3000 –dapr-http-port 3500 node app.js  –components-path .

This starts the Dapr runtime and ensures that all components configured in the yaml files on the components path are instantiated.

The logging shows that the Init function in the new custom state store component is invoked. Great news!

When I indirectly manipulate the state store – by having the application write state to the store and trying to retrieve state directly from the store – as is shown in the next screenshot

The logging will show that both the Set and the Get function have been invoked as well in the custom component.

Time for that little celebration. And then time to create a proper implementation of the functions in the Store interface.

Implementing the Init function

I will first focus on Init. This function is invoked by the Dapr runtime when it processes a component definition for a state store of type state.oci.objectstorage. The component definition contains the configuration properties for an OCI user account (tenancy, region, user, fingerprint and private key) as well as the designated OCI compartment, the bucket in which the state is to be managed and the Dapr name for the state store.

Function Init receives all this information from the Dapr runtime and is expected to do necessary validation and initialization: check if all required properties have been provided, validate if operations can be performed against the OCI ObjectStorage Service and make sure that the requested bucket exists.

The next figure gives an outline of function Init and its tasks and dependencies.

The Init method is called first, as we have seen, before the store does anything. This function should take care of processing (and validating) configuration metadata and initialization. It gets the metadata passed in spec/metadata section of the yaml in a form of key-value dictionary structure. Function Init also has access the to StateStore instance that was first created by NewOCIObjectStorage (based on the locally defined type). The function sets the relevant properties in the StateStore instance – after it has created an OCI ObjectStorage client and connected to verify and if necessary realize the existence of the requested bucket.

The configuration information needed for interacting with OCI Object Storage for creating the state store (a bucket) consists of:

• name of bucket
• tenancy (OCID)
• region
• compartment (OCID)
• user (OCID) – note: alternatively, token based authentication can be used
• fingerprint
• private key

The Init function should check if all this meta data is provided. It has to retain the values. And it should ensure that a bucket with the indicated name exists in the designated compartment. This is also a test of all other configuration settings – only if everything is correctly provided can the Object Storage service be queried for the bucket.

The component configuration now will look something like this:

All required data elements are defined as YAML properties, including the private key. Of course, in real life use cases, the private key should be stored in a secret store and referenced from this YAML file.

I have defined a type ObjectStorageMetadata and included it in the StateStore to hold the meta data passed into the Init function (as well as the derived information such as namespace and objectStorageClient:

The two OCI SDK packages needed to interact with OCI Object Storage are imported

I have created function getObjectStorageMetadata to get the values from the meta data parameter passed in and put them in a local struct called meta.

The function is simple and straightforward, processing all meta data elements or YAML properties:

Function initStorageClientAndBucket bridges to functions that directly interact with OCI ObjectStorage Service APIs. This function creates a client that can be used for all subsequent interactions with the OCI service. This client is saved in the StateStore instance. The function also invokes a function that checks if the bucket exists and if not, makes sure that it gets created.

With function Init properly implemented, I can build the Daprd executable again and run a Daprized application. The log messages should demonstrate the effect and the bucket I ask for in the components.yaml file should get created on OCI ObjectStorage. In order to see the debug level log messages, I now will start the Daprized quickstart application with the log-level switch:

dapr run –app-id nodeapp –app-port 3000 –dapr-http-port 3500 node app.js  –components-path . –log-level debug

The relevant log messages appear:

And the bucket is created as expected on OCI Object Storage – ready to accept and provide state:

When I stop and restart the application, the bucket is not recreated of course because it already exists.

Implementing Set and Get

Both Get and Set are implemented in three tiers: the function as defined in the Dapr interface (without any OCI specific elements), a low level function that interacts with OCI ObjectStorage (and knows nothing about Dapr) and the bridge between these two tiers. Visually the functions can be plotted as is show next

The complete code is available in this Gist.

Calls from the Dapr runtime – triggered by applications sending Get, Set and Delete requests – are made to the Get, Set and Delete functions. The Get function uses readDocument that in turn calls getObject, the function that actually talks to the OCI ObjectStorage service API.

The code for this trio that together retrieve the state for a certain key:

To Set the state for a key, a similar combination of functions is used: Set => writeDocument => putObject.

With Set, an additional nuance is introduced: concurrency (protection). State stores in Dapr can support optimistic concurrency control using ETags. The state store implementation discussed here based on OCI ObjectStorage can indeed provide such concurrency control. With one notable element: applications cannot assign ETag values themselves. Instead, when an object is created or updated (replaced, recreated) it gets assigned an ETag by OCI ObjectStorage. Unfortunately, the Dapr API for the Set function does not define a response object. This means that the ETag defined during the creation of the object in OCI ObjectStorage cannot be returned to the application. Instead, the application has to call Get immediately after Set in order to learn the ETag value that was assigned.

In subsequent Set (as well as Delete) requests, application can specify the ETag value for the state they want to update or remove. If they do so – and indicate that the Dapr “firstWrite” concurrence policy should be enforced – then the operation will only succeed if the object on OCI ObjectStorage still has the indicated ETag value and has not been updated in the mean time by any other agent.

Function writeDocument checks if the SetRequest contains Options.Concurrency different from state.FirstWrite and if so, set the etag passed to putObject to nil. If that concurrency option is set however, then the etag sent by the application to Dapr is passed to putObject and used in the OCI ObjectStorage service API call. The call will fail if an etag value is provided that differs from the actual etag value of the object on OCI ObjectStorage.

Implementing Delete

The implementation of the Delete function is somewhat similar to the Set function except that it does not return a value. It will however return an error if the state to be deleted does not exist. This is prescribed in the Dapr API. Alternatively, one could argue that the desired result – the state does not exist – is achieved, so why return an error? The code for Delete looks like this:

Creating Unit Tests for Custom State Store implementation

Not to be left until the end – and I did not leave it to the end in reality – are the unit tests for the custom state store implementation. Before I can submit the custom component to the Dapr project, I have to lint my code and I have to provide unit tests. It seems that many of the components already in the components-contrib repository have only a meagre set of tests provided. That should not stop me of course from creating a better coverage in my test set.

The file objectstorage_test.go colocated with the code for my custom state store contains the test cases.

The Dapr runtime does not play a role in these unit tests, nor does the components.yaml file. The equivalent to the data in the yaml file is defined inside the tests file.

The tests are ran using go test test, from the root of the components-contrib repository / directory as follows:

go test github.com/dapr/components-contrib/state/oci/objectstorage -v

A small code snippet for the tests

The test coverage is fairly good – both in terms of code coverage as well as in terms of feature coverage.

The full test code is available in this Gist.

Note: in order to run these tests, valid OCI configuration details need to be available.

Next Steps

I believe the implementation of the custom state store component is done. The code is working, the tests prove that it is. I have the quickstart applications working with my custom state store component to provide an OCI Object Storage backed state store. I am ready for the next step: submit this component to the Dapr project through a proper pull request.

In a next article –link to be provided – I will report on the submission process (see docs on contributing to Dapr). It will involve:

• fork repository
• create branch and recreate the custom component
• make sure there’s an issue (bug or proposal) raised, which sets the expectations for the contribution I am about to make.
• Update relevant documentation for the change
• Commit with DCO sign-off and open a PR

Wait for the CI process to finish and make sure all checks are green. A maintainer of the project will be assigned, and I can expect a review within a few days.

Resources

Skeleton State Store Component – Gist – https://gist.github.com/lucasjellema/499d4355c521e3e064e2c1864105d6c7

Gist with all code discussed in this article – https://gist.github.com/lucasjellema/5b2784185ce4d9bcac7cefd753fe4463 

Dapr Docs:  Development of Custom Component – instructions (also on make build and replace) – https://github.com/dapr/components-contrib/blob/master/docs/developing-component.md

My preparation stories: Customizing Dapr–preparing my environment ,  Extending Local Dapr Development Environment for Custom Components

The interaction from Go through the OCI Go SDK with the OCI Object Storage Service is introduced in this article: Interacting with OCI Object Storage using Go SDK (first steps)

Excellent video by Geert Baeke – Writing a Dapr custom output binding for InfluxDB 2.0  https://www.youtube.com/watch?v=AGerQCJMDwQ

Implementing Custom Dapr State – step by step overview of the implementation of a custom state store component  by Ivan G – https://dev.to/aloneguid/implementing-custom-dapr-state-3f8h

The Dapr state store component for Azure BlobStorage

Stack Overflow: preserve line breaks in Yaml files: https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-in-yaml-over-multiple-lines/21699210#21699210 

Running Tests in Go: https://ieftimov.com/post/testing-in-go-go-test/ 

Dapr docs on contributing to Dapr