Oracle Management Cloud has several services for operational management and analysis of applications, platform and infrastructure components. Among the platform components that we can monitor are of course JCS instances – WebLogic in the Oracle Public Cloud. OMC provides a platform for gathering, retaining and analyzing metrics and log data collected from across many platforms and infrastructure components, both in the cloud and on premises. OMC works with agents that are provisioned on the components to be monitored; these agents send metrics and logging to the central OMC instance.
At the moment, the OMC agents are not automatically deployed and configured when a new instance of JCS or DBaaS is provisioned. I would expect Oracle to very shortly provide that option, at least as part of a cloud stack. That means that we have to take care of our own provisioning on JCS of the Cloud Agent that collects and posts the log file entries to OMC Log Analytics (and likewise the APM Agent). The steps are similar to the actions that need to be taken on premises.
I will briefly discuss the steps I took and the experiences I had. I had assistance from my colleagues Job Oprel and Eugene van der Voort and also from Oracle’s Mahesh Sharma.
Preparation
We need SSH access to the Oracle Compute VM that runs the WebLogic Domain. That means we need the private key for the SSH Key Pair of which the Public Key was uploaded to Oracle Cloud during configuration of the JCS provision job and also the public IP address for the VM.
We need a way to copy a file (the AgentInstaller, downloaded from OMC Console) to the VM. I used a tool called MobaXTerm (see MobaXTerm) for this on Windows (it provides sftp over SSH channels)
We need the name of the WebLogic Domain and the names of the managed server(s); the name of the admin server is <domain_name>_adminserver. We also need the name of the host of the WebLogic Domain – in my case amispaas-db-java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal. Finally, we need the listen ports and SSL listen portsfor the admin server (JCS default is 9071 and 9072) and the managed servers (the first managed server listens at port 9073 and 9074 by default).
We need the Web Logic Domain Home (available in EM FMW Control, on JCS defined as: /u01/data/domains/<name of domain>) and the WebLogic Home (Oracle software installation home, typically under the Fusion Middleware Home; on JCS set to: /u01/app/oracle/middleware/wlserver).
Configuration of the Cloud Agent on JCS
Download the Agent Installer from OCS. On the OCS Agents Tab, on the Download sub tab, click on the download link. This will download a file called AgentInstall.zip.
Click on the subtab Registration Keys. Create a registration key if you do not have one yet. Copy a registration key to scratch pad or temp file; you will need to use that key later on. You can also download the key in a text file.
Open an SSH session to the Compute VM. You need to provide public ip address, private key and possibly the passphrase associated with the private key file. The SSH session is always connected as opc.
Extract the file AgentInstall.sh from the zip file downloaded from OMC. Copy the file to the Compute VM. My steps:
- from my laptop: copy AgentInstall.sh to home directory for user opc (using MobaXTerm drag and drop from laptop to MobaXTerm)
- copy AgentInstall.sh to /tmp (to make it accessible to user oracle)
- alternative: put AgentInstall.sh on Dropbox or something similar and use wget https://dl.dropboxusercontent.com/u/<your id>/AgentInstall.sh to fetch the file to the Compute cloud
Now switch to user oracle:
sudo su – oracle
mkdir omc_stage
mkdir omc_cloudstage
cp /tmp/AgentInstall.sh omc_stage
./AgentInstall.sh AGENT_TYPE=cloud_agent STAGE_LOCATION=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloudstage AGENT_REGISTRATION_KEY=RMxMm7chywi- -download_only
Note: instead of downloading and installing in once action, which can also be done, I chose to go about this in two stages. The main reason: my first attempt was in one step; then the process seemed to be stuck on downloading and I aborted the attempt.
The previous command will write a single line of feedback:
‘Downloading lama agent software…’
This download took very long for me. After several minutes (and without any further feedback), the download is complete ( with ps -ef | grep AgentInstall I verified that the process was no longer running). It case created two files in directory omc_cloudstage:
Navigate to omc_cloudstage:
cd omc_cloudstage
and run this command that will provision the cloud agent from the stages download:
./AgentInstall.sh AGENT_TYPE=cloud_agent AGENT_REGISTRATION_KEY=RMxMm7chywi-J-VZ7_UfxY5XUU AGENT_BASE_DIR=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent -staged
The following configuration scripts need to be executed as the root user
#Root script to run : /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/core/1.12.0/root.sh
First exit from user oracle back to opc
exit
then assume root user:
sudo su
then run the script
/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/core/1.12.0/root.sh
feedback:
Creating /etc/oragchomelist file…
The Cloud Agent is now listed in the Oracle Management Cloud Agents overview.
In order to allow monitoring of the Linux Host we have to make a small change in a configuration file of the Cloud Agent. Return to user oracle:
exit
sudo su – oracle
then navigate to directory:
cd /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/plugins/oracle.em.sgfm.zip/1.12.0/configs
open a file for editing in vi
vi discovery.properties
comment out the parameter: disable_monitoring_for_entitytype=omc_host_linux
save the modified file.
The Cloud Agent can be accessed through its command line interface omcli in directory:
/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin
For example to restart the agent:
./omcli stop agent
./omcli start agent
to check the current list of targets:
./omcli config agent listtargets
At this point, no logging is actually flowing to the OMC cloud. The entities currently listed as targets are however already visible, with some associated log sources (in Log Analytics console, click configuration):
Configuring the Entities to Collect Logging For
The documentation provides instructions on adding entities – the actual platform components that logging is to be collected for. For example an Oracle Database, a WebLogic Domain or a WLS Managed Server. Using the omcli tool and a JSON file format for specifying the components, the entities are added. Based on the attributes provided for these entities and the built in knowledge in the cloud agent about the log files written to by the entity type indicated, the cloud agent knows which files are to be harvested for log entries that are sent to the OMC cloud.
I have created a JSON file weblogic_amispaas.json for registering the WebLogic Domain and the Admin server:
{
"entities":
[
{
"name": "amispaas_domain",
"type": "omc_weblogic_domain",
"displayName": "Domain_amispaas_domain",
"timezoneRegion": "CET",
"properties":
{
"port":
{
"displayName": "Port",
"value": "7001"
},
"domain_home":
{
"displayName": "Domain Home",
"value": "/u01/data/domains/amispaas_domain"
},
"admin_server_host":
{
"displayName": "Admin Server Host",
"value": "amispaas-db-java-stack-jaas-wls-
1.compute-amispaas.oraclecloud.internal"
},
"webLogic_home":
{
"displayName": "WebLogic Home",
"value": "/u01/app/oracle/middleware/wlserver"
}
},
"associations":
[
{
"assocType": "omc_contains",
"sourceEntityName": "amispaas_domain",
"sourceEntityType": "omc_weblogic_domain",
"destEntityName": "amispaas_adminserver",
"destEntityType": "omc_weblogic_j2eeserver"
}
]
},
{
"name":"amispaas_adminserver",
"type":"omc_weblogic_j2eeserver",
"displayName":"Admin_Server_AMISPAAS",
"timezoneRegion":"CET",
"properties":{
"host_name":
{"displayName":"Weblogic Host","value":"amispaas-db-
java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal"},
"domain_home":
{"displayName":"Domain
Home","value":"/u01/data/domains/amispaas_domain"},
"listen_port":
{"displayName":"Listen Port","value":"9071"},
"listen_port_enabled":
{"displayName":"Listen Port Enabled","value":"true"},
"ssl_listen_port":
{"displayName":"SSL Listen Port","value":"9072"},
"server_names":
{"displayName":"Server
Names","value":"amispaas_adminserver"}
},
"associations":[
{ "assocType":"omc_monitored_by",
"sourceEntityName":"amispaas_adminserver",
"sourceEntityType":"omc_weblogic_j2eeserver",
"destEntityName":"amispaas_domain",
"destEntityType":"omc_weblogic_domain"}
]
}
]
}
The entities are registered from file weblogic_amispaas.json using this command:
/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli add_entity agent /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/my_entities/weblogic_amispaas.json
I then created a second file for the managed server: oracle-loganalytics-weblogic-managedserver-entities.json
{
"entities":
[
{
"name":"amispaas_server_1",
"type":"omc_weblogic_j2eeserver",
"displayName":"Managed Server 1 AMISPAAS",
"timezoneRegion":"CET",
"properties":{
"host_name":
{"displayName":"Weblogic Host","value":"amispaas-db-
java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal"},
"domain_home":
{"displayName":"Domain
Home","value":"/u01/data/domains/amispaas_domain"},
"listen_port":
{"displayName":"Listen Port","value":"9073"},
"listen_port_enabled":
{"displayName":"Listen Port Enabled","value":"true"},
"ssl_listen_port":
{"displayName":"SSL Listen Port","value":"9074"},
"server_names":
{"displayName":"Server
Names","value":"amispaas_server_1"}
},
"associations":[
{ "assocType":"omc_monitored_by",
"sourceEntityName":"amispaas_server_1",
"sourceEntityType":"omc_weblogic_j2eeserver",
"destEntityName":"amispaas_domain",
"destEntityType":"omc_weblogic_domain"}
]
}
]
}
and used a similar command to add this entity (note: I could have added all entities in a single file with the same effect):
/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli add_entity agent /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/my_entities/weblogic_amispaas_managedserver1.json
Using
/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli config agent listtargets
I then checked whether all entities had been added:
In de OMC Log Analytics Console, the entities are visible as well:
At this point, the log file entries should start to appear in OMC.
Unfortunately, they did not. This turned out to be bug (that may well have been fixed by the time you are reading this).
Fix Bug in Configuration File
No log file entries were published at this point from the JCS VM. Investigation by Eugene van der Voort made clear that a problem existed in the out of the box agent configuration files:
1) Agent Logfile with error messages: $AGENT_HOME/sysman/log/gcagent.log ( /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/log/gcagent.log)
inspect log file /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/log/gcagent.log
2) Locate Error in logfile:
2017-01-19 10:44:00,066 [61:CAD9F7AE] WARN – Some issues to read/parse /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/ApplicationsState/loganalytics/logrules_os_file.xml
java.util.regex.PatternSyntaxException: Unmatched closing ‘)’ near index 58
([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+\(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*).*
^
3) Modify offending file: /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/ApplicationsState/loganalytics/logrules_os_file.xml
4) Locate tag:
<Name>tm_server.log</Name>
<Regex>([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+\(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*)</Regex>
5) Change regular expression to :
<Regex>([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*)</Regex>
6) Then stop and start the agent:
- omcli stop agent
- omcli start agent
Now the log files from the JCS VM finally became visible in Log Analytics:
The log file shows something of the activity of the agent:
it also indicates an existing problem with file path /var/log/audit. I need to check out that one. Perhaps I need to provide a Credential to the Cloud Agent.
Easy solution: provide read access on that path to user oracle. As root:
chmod 754 /var/log/audit
