Introduction

In a couple of blog articles I will described how to get started with Elasticsearch and Oracle Fusion Middleware. Elasticsearch is described on the elastic.co website as ‘Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases.’

Blogs 3 and 4 focus on a use case that is centered around Adaptive Case Management (ACM): the solution is part of the application.

Blogs 5 and 6 focus on the use case of ‘handling logging data’

The series cover:

1. Elasticsearch and Oracle Middleware – is there an opportunity?
2. Installation of Elasticsearch: installation and the indexing of – human generated – documents
3. Elasticsearch and Oracle ACM data: example with ACM data
4. Kibana for ACM dashboards: an example of dashboards with ACM data
5. Logstash and Fusion Middleware: how to get log file data into Elasticsearch
6. Beats and Fusion Middleware: a more advanced way to handle log files

Splunk or Elasticsearch for handling log files?

Some Fusion Middleware customers have implemented Splunk for handling their log files, i.e. for meaningful reports and alerting functionality. The ones that I know of are happy with their choice – so far so good. However, when browsing for possible alternatives, Elasticsearch also popped up. There are a couple of comparisons between these and other ‘tools for processing log files’. In general, these comparisons boil down to:

• each tool has its own strengths and weaknesses
• which tool is best, depends on your specific companies use case
• Splunk acknowledges that they ‘are feeling the heat from Open Source alternatives’

So far nothing new.

Search functionality as part of the application

Consider the use case of a customer with an Oracle Adaptive Case Management implementation where case data AND the involved case documents must be searched. That requires a flexible search functionality that can also index documents. This in fact reveals one of the big differences between Splunk and ElasticSearch: their internal search engine. The Splunk engine is geared towards processing machine-generated log data and the ElasticSearch engine – which is Apache Lucene – is geared towards processing human-generated documents. For more information, please look for the answer given here.

An important aspect here is the search functionality implementation: we don’t want to end up in a situation where we have to code our own ‘fuzzy-searching-style’ algorithms based on the data structures that we have to design. We want a google-like search, that will work on whatever type of documents. This use case has a good fit with Elasticsearch for the search functionality.

Please note that I don’t mean to compare Splunk and ElasticSearch. There is lots of material on that. However, I noted that most comparisons don’t focus on the internal search engine capabilities. You should do that, when relevant for your use case.

ELK stack

ElasticSearch is also known under the acronym ELK. The letters stand for ElasticSearch, LogStash, Kibana. These are parts of the ElasticSearch product stack. Even more, recently Elastic.co added Beats into their product mix ‘The Open Source Elastic Stack’. That makes for a very complete architecture (and possibly a difficult acronym: ELKB? Or BELK? Or BLEK?).

The core products of ‘The Open Source Elastic Stack’ are:

• Elasticsearch: a search engine based on Apache Lucene
• Logstash: a data processing pipeline, for transforming data and then send it to Elasticsearch
• Kibana: a BI like product for visualizing data in Elasticsearch
• Beats: lightweight agents that collect data and send it to Logstash or Elasticsearch

A large scale set-up would look like shown below:

Next to these products, the Elastic solution can be extended with other products. These products offer functionality around security, altering, monitoring, reporting and for exploring data relationships. All-in-all, the offering seems quite complete and mature.

And, also Elastic offers a Cloud solution that offers Elasticsearch and Kibana on AWS.

The Elastic product offering is quite vibrant: the recent addition of Beats and the new release 5.0 (October 2016) have resulted in an impressive offering. In the examples, the new release 5.0 is used. So, at the time of writing, this is pretty much ‘the latest and greatest’.

Conclusion

The big question is ‘would you recommend Elasticsearch for implementation of a flexible search functionality with ACM?’. In order to give that answer, there are some topics that still need to be considered for your specific use case:

• Elasticsearch also can be viewed as a data store: will that meet your use case requirements? Will it be a primary data store for your data?
• How can security and authorization be implemented, i.e. who has access to what search results?
• Will the product choice fit with your IT operations requirements
• If your data lacks structure or its structure changes regularly, Elasticsearch as a NoSQL DataStore may be a better choice
• …etc

But, after my first steps with the Elasticsearch product offering, I think that it is an option that you should consider when you need search functionality as part of an application…

A favor

If I can ask you a favor: if I recall correctly, Oracle mentioned somewhere on OOW 2015 that they were also going to use the Elasticsearch engine in one of their products. If you know more details, could you please drop me a mail? I’m kind of curious to know how that ended up?