A brief headsup for everyone responsible for maintaining kernel options in the oracle executable.
The environment where I discovered this is a 3 node racattack cluster with Oracle GI & RDBMS 18.104.22.168.0. After enabling Unified auditing in my 22.214.171.124 database I noticed that all of a sudden database startup started failing with an almost unlimited amount of core dump files being generated.
SQL*Plus: Release 126.96.36.199.0 Production on Wed Sep 14 07:57:47 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected. ERROR: ORA-01012: not logged on Process ID: 0 Session ID: 0 Serial number: 0
Before I put them aside in the dustbin I looked inside one of them (racdba1_q002_32146_i138141.trc) and it showed following:
Dump file /acfs/oradb/diag/rdbms/racdba/racdba1/incident/incdir_138141/racdba1_q002_32146_i138141.trc DDE recursive invocation exceeded max allowed DDE was called in a 'No Invocation Mode' DDE encountered the following error: ORA-49203: Too many recursive DDE invocations ORA-00600: internal error code, arguments: , , , , , , , , , , ,  ORA-00600: internal error code, arguments: [kpotcfnd-4], ,</pre> <pre>[ORA-00600: internal error code, arguments: , , , , , , , , , , ..
Fortunately we do have an Oracle support contract over here so we could use the ORA-600/ORA-7445/ORA-700 Error Look-up Tool from Doc ID 153788.1. It turns out to be a known bug (2036911: ORA-600 / Cannot enable more than 8 kernel options (such as uniaud , olap, lbac etc..) and fixed in upcoming 12.2 release.
It turns out that enabling Unified Auditing was just the last straw (in Dutch we have a saying about a drop of water and subsequent bucket overflow..) that led to this problem. After disabling an option I knew I wouldn’t be using like olap option in the oracle kernel it all started to work as expected! Is it neccessary to mention that RAC databases have to be stopped first?
[oracle@collabn2 bin]$ chopt disable olap Writing to /u01/app/oracle/188.8.131.52/rachome1/install/disable_olap.log... /usr/bin/make -f /u01/app/oracle/184.108.40.206/rachome1/rdbms/lib/ins_rdbms.mk olap_off ORACLE_HOME=/u01/app/oracle/220.127.116.11/rachome1 /usr/bin/make -f /u01/app/oracle/18.104.22.168/rachome1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/22.214.171.124/rachome1
After this fix I was able to logon to my RAC database again:
Connected to: Oracle Database 12c Enterprise Edition Release 126.96.36.199.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security, Advanced Analytics, Oracle Database Vault, Real Application Testing and Unified Auditing options
The bug reference is:
|20369110||188.8.131.52||ORA-600 / Cannot enable more than 8 kernel options (such as uniaud , olap, lbac etc..)|
An easy way to count the enabled oracle kernel options is to look at the banner you get back from the rdbms after login in sqlplus. Below each enabled kernel option is marked with an ellips: