You are in trouble when more than 8 kernel options are enabled in oracle executable

A brief headsup for everyone responsible for maintaining kernel options in the oracle executable.

The environment where I discovered this is a 3 node racattack cluster with Oracle GI & RDBMS 12.1.0.2.0. After enabling Unified auditing in my 12.1.0.2 database I noticed that all of a sudden database startup started failing with an almost  unlimited amount of core dump files being generated.

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 14 07:57:47 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected.
ERROR: ORA-01012: not logged on
Process ID: 0 Session ID: 0 Serial number: 0

Before I put them aside in the dustbin I looked inside one of them (racdba1_q002_32146_i138141.trc) and it showed following:

Dump file /acfs/oradb/diag/rdbms/racdba/racdba1/incident/incdir_138141/racdba1_q002_32146_i138141.trc
DDE recursive invocation exceeded max allowed
DDE was called in a 'No Invocation Mode'
DDE encountered the following error:
ORA-49203: Too many recursive DDE invocations
ORA-00600: internal error code, arguments: [9999], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kpotcfnd-4], [600],</pre>
<pre>[ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [], [], ..

Fortunately we do have an Oracle support contract over here so we could use the ORA-600/ORA-7445/ORA-700 Error Look-up Tool from Doc ID 153788.1. It turns out to be a known bug (2036911: ORA-600[9999] / Cannot enable more than 8 kernel options (such as uniaud , olap, lbac etc..) and fixed in upcoming 12.2 release.

It turns out that enabling Unified Auditing was just the last straw (in Dutch we have a saying about a drop of water and subsequent bucket overflow..) that led to this problem. After disabling an option I knew I wouldn’t be using like  olap option in the oracle kernel it all started to work as expected! Is it neccessary to mention that RAC databases have to be stopped first?

[oracle@collabn2 bin]$ chopt disable olap
Writing to /u01/app/oracle/12.1.0.2/rachome1/install/disable_olap.log...
/usr/bin/make -f /u01/app/oracle/12.1.0.2/rachome1/rdbms/lib/ins_rdbms.mk olap_off ORACLE_HOME=/u01/app/oracle/12.1.0.2/rachome1
/usr/bin/make -f /u01/app/oracle/12.1.0.2/rachome1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/12.1.0.2/rachome1

After this fix I was able to logon to my RAC database again:

Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
Advanced Analytics, Oracle Database Vault, Real Application Testing and Unified Auditing options

The bug reference is:

20369110

12.2.0.0

ORA-600[9999] / Cannot enable more than 8 kernel options (such as uniaud , olap, lbac etc..)

 

An easy way to count the enabled oracle kernel options is to look at the banner you get back from the rdbms after login in sqlplus. Below each enabled kernel option is marked with an ellips: