The future of CDM RuleFrame – the Oracle framework for server-side business rules

In this post I will discuss some thoughts I have on the further development of CDM RuleFrame (the Framework for implementing Business Rules). It seems to me that a) Oracle has not done much on RuleFrame since its first incarnation (production release in 2000) and b) the environment has changed to such an incredible extent that the technical options are so much greater than back in 1999 that RuleFrame would – and should – look considerably different today. Some importants developments: Oracle 9i (or even 10g) RDBMS instead of 7.3 as we used to develop RuleFrame, the current status of Oracle Designer, the rise of Java/J2EE applications and of course all the experience we have collected with developing and implementing CDM RuleFrame.
For more information on what RuleFrame is, see the RuleFrame homepage

Some of you may know that back in 1999, I laid the foundation for CDM RuleFrame, the framework for implementing (server side) business rules offered by Oracle Consulting. At the end of the ODTUG 1999 Conference in Fort Lauderdale, Florida, on a day ruined by lots of rain, Ton, Sigrid and myself sat discussing much of what we had heard during the conference, especially with regard to Business Rules. CDM – Oracle’s methodology for Custom Development – contained an excellent section of Business Rules, especially on classification of Business Rules. However, we had never really worked out the implementation details. For some reason at the end of that conference, that is exactly what we started working on.

I clearly remember that our plane left around 8pm from Miami Intl Airport for Amsterdam and that even as we were boarding, Sigrid and I were making scribbles and drafts. Over the few weeks after our return I was frantically working on the first prototype of CDM RuleFrame – of course that name had not been thought up back then. Somewhere early in September I did the first presentation and demonstration of that prototype; the core of it has not really been changed ever since!

My biggest intellectual triumph at the time was the solution I invented for locking the backdoor to people trying to circumvent business rule enforcement. I introduced a dummy table with a deferred check constraint. All tables that were under RuleFrame supervision were given a generic DML trigger (one that fires on every insert, update or delete-operation). This trigger calls a central procedure that does nothing but perform an update on the dummy table that violates the check constraint. In doing so, we know for sure that the user cannot commit his transaction: the deferred check constraint is violated and the database will not allow the commit to happen. The only code that can correct the situation and return the dummy table to a valid state is the same code that performs the validation of all RuleFrame business rules; that means: no transaction can commit without performing the RuleFrame validations.

Note that the deferred constraint was the only non-Oracle 7 feature used in CDM RuleFrame. At the time, it was a good thing to provide a framework that did not depend on any fancy scarcely available and probably unreliable new database feature. We were pleased with the fact that almost any organisation beyond the Oracle 6 era could use RuleFrame. However, as time evolved, the only 8i features we included in RuleFrame (as far as I can remember) were Native Dynamic SQL instead of DBMS_SQL adn a brief flirt with Global Temporary Tables. I think today it is a waste of many great database facilities to stick to what is basically Oracle 7 code.

I also requested an ON-COMMIT trigger in the database, to allow me to specify my own code to be executed by the database engine just prior to committing a transaction. It was filed into the Bug- and Enhancement Request database of Oracle product development, but unfortunately until today it was not acted upon.

RuleFrame has been adopted by a substantial number of organisations; I do not have the exact number but I would estimate somewhere between 100 and 250. Oracle has released updates of RuleFrame, as part of the iDevelopment Accelerators Suite – that also contains CDM, Headstart Template Package, Headstart Utilities, Headstart for Apps and Oracle Designer Web Assistant. For RuleFrame, only minor enhancements have been developed, ever since we reached a fairly stable release somewhere midway 2000.

I started work on a more structured method for analyzing business rules, called RuleSLang – for Rule Specification Language. This work was inspired by OCL – the Object Constraint Language, part of UML. It evolved even into a simple prototype that could interpret, translate back to natural language and generate code for specification such as: order.customer.country.continent = order.salesagent.country.continent (the continent that contains the country that has the location of the salesagent that sold the order must be the same as the continent that contains the country that houses the customer of this order). I presented on RuleSLang during the OTDUG 2000 conference (see Technical Papers), but afterwards other priorities – mainly Java and JHeadstart – replaced RuleSLang, and largely RuleFrame as well – on our agenda.

In all those years since 1999 I have not yet seen the ideal alternative for RuleFrame; many people discuss Business Rules and their specific – often very specific – ways of implementing them. I have seen many interesting thoughts, inspiring approaches to common problems and useful demonstration of both challenges and solutions in business rule implementations though.

RuleFrame has hardly evolved at all. That is in a period where both demands and opportunities have increased enormously. Back in 2000 we were almost proud of the fact that RuleFrame could be used on a Oracle 7 Database. No new, fancy, upgrade-requiring, potentially immature features were required. RuleFrame was for the masses. However, almost everyone is at least on 8.1.7 right now, many people on 9iR2 or contemplating the move to 9iR2. It is a waste NOT to make use of technology that was added to the database after 1995.

Appeal to the RuleFrame team

I would like to call upon Oracle Consulting, my former colleagues, to get back into gear with regard to RuleFrame. There is plenty of opportunity to improve the existing framework. There is also ample cause to do so. I will present a list of both categories below. In short, I feel that in order to take RuleFrame serious as a meaningful framework in the coming years, action is required. I challenge Oracle to present its response to this post and provide an overview of its intentions, both shortterm and longer term, with RuleFrame.

Since I know a little about staffing issues, priority setting and resource management etc. at Oracle Consulting, I know this will pose a challenge to the team in charge of RuleFrame. I therefore propose an alternative as well. Turn RuleFrame into an open source project. This will not only allow but in fact encourage developers around the world to participate in a project to improve RuleFrame. It allows former Oracle specialists such as Marc Vahsen, Lauri Boyd and myself to continue with their work on RuleFrame. Maybe we can bring in people with very interesting ideas such as Toon Koppelaars (see his website) and maybe some people from the ODTUG Business Rule symposium bunch, people for example from Dulcian (BRIM) and TUSC.

I for one would certainly like to get involved with some of my excellent colleagues at AMIS!

Below I will list some of the reasons why I feel that a major revamp of RuleFrame is in order. Anyone who can add to the list (or maybe delete from it) is invited to comment to this post.

Opportunity for improvement – an area where improvement may not be actually required – because of a bug – but is fervently wished for.

RuleFrame Design Time

• Improved User interface:The design time UI for RuleFrame is an awkward mixture of several Oracle Designer components (RON, Design Editor, ERD) and a number of Headstart Utilities – not the best user interface at the best of times. Switching between environments and different areas within those environments makes it much harder to perform all steps in the Business Rule development process than would be necessary. The combination of Business Function, Events, Trigger Definition and Module Implementation in Oracle Designer, while functional, is not intuitive and far too much exposed to the analyst/designer/developer.

  I would strongly advocate an integrated user interface, either HTML Web or GUI, that supports the entire development life-cycle of the a business rule. For example, instead of forcing the analyst to first create all events for an entity and then select those events required for the event-analysis of a business rule, the UI would allow the analyst to just select events – and have the UI generate the necessary event-objects in the background. Instead of looking hard for all standard Designer properties that also happen to play a role in the life of RuleFrame Business Rules, the UI would only expose the properties that apply to business rule analysis and design. And instead of using the Headstart Utilities WebForms UI to invoke various utilities, these would also be invoked from the integrated UI. Given the strategic direction Oracle is taking with its tools, one could not be blamed for considering JDeveloper as foundation for the RuleFrame IDE.

• Extended Generation of BR for popular patterns: Everything in RuleFrame is in place to support the effortless generation of business rules and their complete implementation code for many frequently occurring rule patterns, such as details should fall within date range set by their master, sum/count/avg of detail-attribute within master should not exceed a certain limit, relationships may not be cyclic (a record cannot through various foreign keys turn out to be its own parent), etc.

  A simple syntax prescription for defining the analysis definition of the rule and a slightly more complicated piece of code to actually generate the validation logic for each class of rules is all that is needed.. Yet, at this point in time RuleFrame still only supports generation of a very small set of rule patterns, mostof them dealing with super-sub-types. I think we can increase the productivity and the appeal of RuleFrame enormously if we start implementing generation capability for some popular patterns.

• Structured Analysis (for example RuleSLang): It’s not on the top of my list, interesting and fascinating as it may be, but following the previous item of generation for popular rule patterns is a more generic way of describing rules in a ‘language’ that would allow for generation of the design definition and implementation of the rules. The work we already did on RuleSLang as well as the OCL language within the UML specification can help us get started here.

  Since event analysis proves essential for complete and correct business rule implementation, the language and its interpreter should at the very least be able to derive the entire event-scope for a business rule from its structured specification. Expressing standard patterns through such a language should likewise be fairly easy. It is crucial however to create a language that is not much harder than SQL; many attempts in this direction end up being much too formal, mathematical. I would strive for usability – sacrificing completeness and rigor along the way. If we can generate 90% of the business rules from easy to write specifications, I am a happy camper. I do not believe in 100% here.

• Dependence on Oracle Designer: RuleFrame has a strong dependency on Oracle Designer, in two respects:
  a) we use Oracle Designer to store the meta-data for Rule definitions, tied to Entity and Table Definitions and recorded through Business Functions, Events, Triggers and PL/SQL modules.
  b) RuleFrame relies on the TAPI (table API) generated by Oracle Designer as key component in the Run-time architecture.

  Given the strategic direction within Oracle with regard to Oracle Designer and also the fairly limited presence of Oracle Designer among organizations engaging in Oracle based development activities – including purely Java/J2EE development organisations – this strong dependency will prove an inhibitor to growth and perhaps even survival of RuleFrame. In my opinion, RuleFrame makes perfect sense to anyone using an Oracle database, regardless of the applications run against that database or the technology in which they are developed. It would seem sensible then to provide RuleFrame to organisations not using Oracle Designer. I would propose to severe the strong and currently unavoidable ties that RuleFrame has with Oracle Designer. That means that meta-data should not – only- be tied to Oracle Designer. It also means that an alternative to the TAPI should be supported.

• Dependency on Headstart Utilities and WebForms (for Design Time UI):As stated before, in the section on the user interface, is the dependency the RuleFrame design time currently has on the Headstart Utilities (UI) somewhat unfortunate. Even though the utilities themselves run as pure server side operations, the user interface is implemented using WebForms. While a fine technology, the burden on setting up a development environment that supports RuleFrame become much larger for organisations not necessarily doing Forms development.

  The UI itself is rather simple and straightforward and can easily be implemented using for example the Web PL/SQL Toolkit in an HTML UI – which could then for example be integrated with the Repository Object Browser (pka ODWA). It can also be implemented with either HTML frontend of Java Client UI using Java technology. Of course, I would suggest considerable improvements to the current UI while we are at it. The Headstart Utilities startform for example is very generic; it has no specific knowledge about individual utilities and can therefore only provide limited and inflexible user feedback.

Runtime

• Use of Oracle 8i and 9i Database Features: This one probably alone would justify an overhaul of RuleFrame. My recent analysis of 8i and 9i features for the Oracle 7Up Workshop I presented gave me a very good insight in the many features and facilities in 8i and 9iR2 that would help to make RuleFrame much better: faster, more efficient with resources, easier to administrate and easier to develop and maintain. See below the discussion on OLD values (for dynamic rules) as just an example of what a close analysis of the current RuleFrame implementation compared with state of the database technology could produce.
• Integration with Oracle RDBMS declarative constraints:RuleFrame always had a difficult relationship with declarative constraints. It was clearly not right to give up declarative integrity constraints when introducing RuleFrame. Yet because of RuleFrame’s transaction focus and intention to present all rule violations at once to the end-user, we never felt comfortable with the declarative constraints. I fear that we sort of ended up allowing people to go for Primary Key, Unique and Foreign Key constraints but somewhat dissuading the use of Check Constraints.

  I feel a new approach is in order here. Wherever you can implement a business rule using a declarative constraint, you should do so. I would suggest setting all declarative constraints to DEFERRABLE INITIALLY DEFERRED. Furthermore, I would extend the RuleFrame runtime core with the generic capability to:
  a) set all constraints immediate and check whether a violation occurs; if not, continue with normal processing
  b)if there is a violation, turn all constraints one by one from deferred to immediate. Those that give no violation can be ignored
  c) for the ones that return a violation, either return generic messages specifying which constraints were violated or go the extra mile and
  d) find out exactly which records among the ones touched in the transaction violate the constraint; for this latter bit, the constraint specification must be retrieved from the data dictionary and turned into a SQL statements that finds the violating records. Unfortunately, there does not seem to be a quicker way to find out the culprits (also see my tete-a-tete with Tom Kyte)

• Run-time rule-administration: I have proposed a RuleFrame run-time monitor long time ago. I bring it up afresh. I can see much value in a run-time administration facility for RuleFrame, not unlike the meta-facilities in the database for declarative constraints. It could provide such facilities as:
  a) disable/enable business rules (in general and per session)
  b) provide switch: validate all/fail on first
  c) provide switch: validate at statement level/validate at transaction level
  d) parametrization of BRs: provide support for business rules with configurable parameters; for example: ‘Managers may not earn more than x% higher salaries than the average salary of their subordinates’ or ‘an order may not contain more than y order lines’. In these examples, x and y are parameters that are defined outside the Business Rule validation code. The ‘monitor’ could provide a user interface to maintain the parameter-values
  e) collecting statistics (which rules are never validated, which are frequently violated, which ones take long to validate)
  f) provide debug/trace facilities
  g) provide ‘exceptions into’ option with validate_all_static (like EXCEPTIONS clause for ENABLE declarative constraints)
• Virtual Private Business Rules: With the advent of global web applications and the ASP model it becomes even more natural to require facilities to have business rules applied to a subset of the data or a subset of the user-community. For example: our order-processing application is hosted – in one database schema – for five retail-organisations. Each has their own business process and associated business rules. Therefore, although business rules are implemented in the database, they should not all be enforced for all data and all transactions/users. Depending on which retail-organisation’s data and users are involved, certain rules apply and others do not. Likewise, the parameters of business rules can be ‘virtually private’: depending on user or organisation, the parameter settings may vary.

Necessity – areas where improvements or even corrections are needed.

• Performance and efficiency: For OLTP applications with relatively small transactions, performance and resource usage were never a big issue with RuleFrame. For substantially sized batch-operations, this is slightly different and the validate_all_static operation, which allows you to see whether all data in a table adheres to all – possibly new or modified – business rules, in some cases never completes at all because of resource- or performance issues.

  I would suggest – as Lauri and myself discussed long ago to address the validate_all_static issues – to leave the procedural record by record validation model we currently have in RuleFrame (and PL/SQL table in which we retain all old and new values) and go to a set-based model where we use SQL statements to find records that violate our rules. This will render (most of) the CAPI logic redundant. It will change the way business rule validation code is programmed, but I think it will prove more natural, more reusable and much more efficient to perform. We may provide a way to generate such SQL from simple rule specifications.

• Plugging a Loophole: there is currently a loophole in RuleFrame through which incorrect data may enter the database. There is an easy fix possible by the way. The situation occurs for entity and inter-entity rules, and is probably best explained with an example.

  A department may not have more than two Clerks. Two transactions – A and B – are executed in parallel. Transaction A inserts a Clerk in the SALES department. That brings the total number of Clerks in SALES to 2. At about the same time, Transaction B transfers a Clerk from the Research department to SALES. When Transaction A validates the business rule, it finds two Clerks in SALES: the one that already existed there and the one newly created. No violation. Before Transaction A commits, Transaction B also validates the rule. It too finds two Clerks in SALES: the one already there and the one transfered from RESEARCH. At this point both Transaction A and B commit. The final result is three Clerks in SALES, a violation of the Business Rule.

  The time between validation and commit is the weak spot in the RuleFrame architecture. The solution is simple though – thought up by my AMIS colleague Ton Elie and myself as we were landing at Miami Intl Airport prior to attending the ODTUG 2003 Conference. In order to start validation of the business rule for Department SALES in the example, the transaction must take a lock (using dbms_lock) on that combination (rule and department). Only one transaction can validate at the same time. Commit or rollback releases the lock. Some mechanisms need to be devised to prevent deadlocks!

• Support for new datatypes such as LOBs, XMLType and Object Types that require a different way of enforcing Business Rules (for example: no documents must be entered in this CLOB column that are about sex (and not gender), implemted using the Oracle Text theme-indexing)
• Change event without dml – use dbms_job to schedule the operation as part of transaction: RuleFrame documentation always stated that Business Rules of type Change Event without DML (the standard example being sending an email or writing a file) were not linked to the transaction and should therefore be executed after all validations had taken place and we could be fairly sure that the transaction would succeed – and the Change Event was allowed to be performed.

  It seems that using dbms_job is a better way of doing this: a job gets only schedules when the transaction that schedules it gets committed. That means that now it does not matter when in the transaction the CEW gets scheduled, it gets only executed when the transaction succeeds completely. No longer can it happen that an email gets send inadvertently from a CEW rule from a transaction that got rolled back after all. Note: scheduling the event will also mean it is typically not actually done when the user gets notice of the successful commit; even jobs scheduled for immediate execution will experience a small delay before the background job-handler processes pick them up. This means on the one hand that user-feedback can be faster than before. On the other hand, it can be confusing in certain situations where the end-users would not expect successful commit unless the CEW is completely performed.

Sideline:
I have engaged in a conversation with Tom Kyte on how to capture the records that violate deferred constraints. It turned into an argument from Mr Kyte on the validity of generic engines, especially rule engines. I have to admit, I feel somewhat at a loss for his fairly harsh comments. Is it true that we are trying to do too much with stuff like RuleFrame?

Old-values for Dynamic Rules – a discussion on efficiency and use of new database features
Dynamic rules frequently require the availability of both old and new column values. Such rules include ‘the salary may not be increased by more than 10%’ and ‘you can become gold customer only if you previously were a silver customer’.

RuleFrame provided old values in not extremely efficient way: it retained all old values for all columns in all records that participated in the transaction. Although this does the job, it is clearly a somewhat massive approach, especially for larger transactions.

There are at least six ways to improve on the current implementation regarding old-values:
1. only hold old-values for records in tables that actually have dynamic rules defined against them (instead of all records of all tables)
2. only hold old-values for records that trigger the dynamic rules (instead of all records participating in the transaction)
3. only hold old-values for the columns that are required for the validation (instead of old values for all columns)

These three require a more intelligent CAPI generator. No new database features are used for them.

4. Open a cursor for each record that is touched in the transaction and for which old values are required. A cursor will return the values from the time it was opened, so if we ensure that it is opened before any changes are applied, we will get the old, pre-change values, when we fetch from the cursor during validation. This approach can result in a substantial number of cursors to be held during transactions.

5. Use autonomous transaction to get to the old values: procedure run in an autonomous transaction do not see the changes made by their parent-transaction. That means that if you query a record inside an AT – a record that has been changed in the parent transaction – you will find the unchanged (aka the old) values of the record.

6. Use flashback query: the Oracle 9i feature of FlashBack query allows us to query the database ‘in the past’. That means that we can select values for a certain record as they existed prior to the current transaction, by simply using the SELECT …FROM … WHERE… AS OF scn where scn is the System Change Number that identifies transactions. If we capture the SCN at the beginning of our transaction (using dbms_flashback.get_system_change_number), we can then at any time get the values as they existed before the beginning of the transaction (in all fairness I have to point out that this suggestion was done to me by Toon Koppelaars of Centraal Boekhuis, although I am confident that as part of my research for the 7Up workshop, I would have come up with it myself).

Note:A summary of this piece is posted on the OTN Discussion Forum for Headstart and RuleFrame. Comments/reactions may be found there as well – I hope.