and
How I resolved it …. My Way
235 … 235 … isn’t it an Interstate in Oklahoma? Is Oracle telling me that I’m now working with only 0.72% of all naturally occurring uranium? Was I testing the maximum speed of a BMW 3series? Or did somehow an Un-locked Control File Transaction (ORA-235) occurred during my OMS-system patch? No.
All I did, was testing a patch with OMSPatcher 13.8.0.2 and it failed to login into the EM-repository of our Grid Control 13.2: the OMSPatcher failed with error code 235 and none of the published workarounds helped to make it work.
It all began some weeks ago when we were suddenly nagged by hundreds of EM notification mails about “Java.lang.NullPointerException:oracle.sysman.gcagent.state.StateMgr:1096” (see also Doc ID 2260803.1). We knew there was nothing wrong, so I tried to stop them with the given workarounds and we tried to ignore the notifications which still occured. But as a side-effect we started to ignore – or beter – started to overlook other more important notifications from OMS as well – and that was worrying.
It was time to patch our OMS 13c rel. 2 and all its agents to the highest possible version because at least the penultimate version at that time, bundle patch 180430, should prevent this notification for good.
So I began the “Study of OMS-Patching” which is sometimes very confusing and began by applying the latest OMSPatcher and OPatch patches to 13.8.0.2 and 13.9.3.2 respectively. Then I wanted to apply the first OMS system patch which always starts with an analyze or test of the patch to be applied:
omspatcher apply analyze ....
After I gave the login data for user SYSMAN, waited a short while, the command came back with:
OMSPatcher failed with error code 235
I adapted the url, the password, the protocol, nothing helped. Again and again the answer was: OMSPatcher failed with error code 235.
With WLST, the initial url was no problem at all, the login to the repository succeded. Only the OMSPatcher version 13.8.0.2 refused to login and patch!
Of course we searched the internet and MOS and found a couple of Doc Id’s like Doc ID 2343447.1 or 2233103.1, which gave workarounds for this error, but none of them worked for us.
In the meantime I opened an SR, for which I answered lots of questions, attached multiple log- and config files, showed Oracle the live environment … and waited for the inspired tip. It never came and no changes to the OMSPatcher have been further applied. And no patches could be applied without returning to a former version of the OMSPatcher.
After days of waiting for Oracle Support I looked at my list of things I had still to patch and realized that an other problem should be completely indipendent of the OMSPatcher and could be repaired without it. The problem was, that passwords of users incl. our administrative superusers could not be changed via the EM13c GUI. The administrators could change their passwords directly in the repository database. To remedy this it was according to Doc ID 2241358.1 necessary to change the JDK version of the OMS and the agents to JDK SE 1.7_131 (not 151 or anything else!!). And this I applied, tested it successfully and my collegues could finally change passwords in the GUI.
I cannot say why or how I got the idea to test the OMSPatcher again directly after it was clear that the password problem had successfully been repaired. But to my incredible amazement the still unchanged OMSPatcher 13.8.0.2 now suddenly worked as intended!!! Just by changing the JDK, amazing!
And none of the papers, articles and documents we had found so far had mentioned this as a workaround or solution… and not even Oracle Support asked us to switch to a higher JDK!
“Blog, Blog”, called a colleague immediately from a cross the table, who had observed my weeks of struggle, and now saw me dancing through the room to fetch my “Victory-coffee”.
Later I attached the output of opatch lsinventory -detail to the SR, where you can see that some jar-files have been touched during the applied patches but Oracle never made any comment which of these might have been a possible cause. The support engineer only thanked me for the update when I triumphantly announced that I found the solution to my problem … and asked some more questions and for more logs.
All I ever got to know from Oracle Support was that “more occurences like this have been recorded, but they where on other OS’ses like Windows”.
Meaning, even Oracle support can not (or will not) explain what happened here?
I quickly developed a theory of what happened which involved an assumption of an applied patch which has never been checked by Oracle support….but, who am I.
So, when you run into an “error code 235” with your freshly patched OMSPatcher, try to upgrade your JDK to version1.7._131 and watch what happens!
I hope this little story saves one of you a lot of time and work.
Have fun!
