Configuring Cloud Agent for Log Analytics on JCS instance to Collect WebLogic Log Files

0

image

 

Oracle Management Cloud has several services for operational management and analysis of applications, platform and infrastructure components. Among the platform components that we can monitor are of course JCS instances – WebLogic in the Oracle Public Cloud. OMC provides a platform for gathering, retaining and analyzing metrics and log data collected from across many platforms and infrastructure components, both in the cloud and on premises. OMC works with agents that are provisioned on the components to be monitored; these agents send metrics and logging to the central OMC instance.

At the moment, the OMC agents are not automatically deployed and configured when a new instance of JCS or DBaaS is provisioned. I would expect Oracle to very shortly provide that option, at least as part of a cloud stack. That means that we have to take care of our own provisioning on JCS of the Cloud Agent that collects and posts the log file entries to OMC Log Analytics (and likewise the APM Agent). The steps are similar to the actions that need to be taken on premises.

I will briefly discuss the steps I took and the experiences I had. I had assistance from my colleagues Job Oprel and Eugene van der Voort and also from Oracle’s Mahesh Sharma.

 

Preparation

We need SSH access to the Oracle Compute VM that runs the WebLogic Domain. That means we need the private key for the SSH Key Pair of which the Public Key was uploaded to Oracle Cloud during configuration of the JCS provision job and also the public IP address for the VM.

We need a way to copy a file (the AgentInstaller, downloaded from OMC Console) to the VM. I used a tool called MobaXTerm (see MobaXTerm) for this on Windows (it provides sftp over SSH channels)

We need the name of the WebLogic Domain and the names of the managed server(s); the name of the admin server is <domain_name>_adminserver. We also need the name of the host of the WebLogic Domain – in my case amispaas-db-java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal. Finally, we need the listen ports and SSL listen portsfor the admin server (JCS default is 9071 and 9072) and the managed servers (the first managed server listens at port 9073 and 9074 by default).

We need the Web Logic Domain Home (available in EM FMW Control, on JCS defined as: /u01/data/domains/<name of domain>) and the WebLogic Home (Oracle software installation home, typically under the Fusion Middleware Home; on JCS set to: /u01/app/oracle/middleware/wlserver).

 

Configuration of the Cloud Agent on JCS

Download the Agent Installer from OCS. On the OCS Agents Tab, on the Download sub tab,   click on the download link. This will download a file called AgentInstall.zip.

image

Click on the subtab Registration Keys. Create a registration key if you do not have one yet. Copy a registration key to scratch pad or temp file; you will need to use that key later on. You can also download the key in a text file.

SNAGHTML328cbf6

Open an SSH session to the Compute VM. You need to provide public ip address, private key and possibly the passphrase associated with the private key file. The SSH session is always connected as opc.

Extract the file AgentInstall.sh from the zip file downloaded from OMC. Copy the file to the Compute VM. My steps:

  • from my laptop: copy AgentInstall.sh to home directory for user opc (using MobaXTerm drag and drop from laptop to MobaXTerm)
  • copy AgentInstall.sh to /tmp (to make it accessible to user oracle)
  • alternative: put AgentInstall.sh on Dropbox or something similar and use wget https://dl.dropboxusercontent.com/u/<your id>/AgentInstall.sh to fetch the file to the Compute cloud

 

Now switch to user oracle:

sudo su – oracle

SNAGHTML32fee45

mkdir omc_stage

mkdir omc_cloudstage

cp /tmp/AgentInstall.sh omc_stage

./AgentInstall.sh AGENT_TYPE=cloud_agent STAGE_LOCATION=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloudstage AGENT_REGISTRATION_KEY=RMxMm7chywi- -download_only

Note: instead of downloading and installing in once action, which can also be done, I chose to go about this in two stages. The main reason: my first attempt was in one step; then the process seemed to be stuck on downloading and I aborted the attempt.

The previous command will write a single line of feedback:

‘Downloading lama agent software…’

This download took very long for me. After several minutes (and without any further feedback), the download is complete ( with ps -ef | grep AgentInstall I verified that the process was no longer running). It case created two files in directory omc_cloudstage:

image

 

Navigate to omc_cloudstage:

cd omc_cloudstage

and run this command that will provision the cloud agent from the stages download:

./AgentInstall.sh AGENT_TYPE=cloud_agent AGENT_REGISTRATION_KEY=RMxMm7chywi-J-VZ7_UfxY5XUU AGENT_BASE_DIR=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent  -staged

 

image

 

The following configuration scripts need to be executed as the root user

#Root script to run : /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/core/1.12.0/root.sh

First exit from user oracle back to opc

exit

then assume root user:

sudo su

then run the script

/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/core/1.12.0/root.sh

feedback:

Creating /etc/oragchomelist file…

 

The Cloud Agent is now listed in the Oracle Management Cloud Agents overview.

image

In order to allow monitoring of the Linux Host we have to make a small change in a configuration file of the Cloud Agent. Return to user oracle:

exit

sudo su – oracle

then navigate to directory:

cd /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/plugins/oracle.em.sgfm.zip/1.12.0/configs

open a file for editing in vi

vi discovery.properties

comment out the parameter: disable_monitoring_for_entitytype=omc_host_linux

save the modified file.

The Cloud Agent can be accessed through its command line interface omcli in directory:

/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin

For example to restart the agent:

./omcli stop agent

./omcli start agent

to check the current list of targets:

./omcli config agent listtargets

image

At this point, no logging is actually flowing to the OMC cloud. The entities currently listed as targets are however already visible, with some associated log sources (in Log Analytics console, click configuration):

image

 

Configuring the Entities to Collect Logging For

The documentation provides instructions on adding entities – the actual platform components that logging is to be collected for. For example an Oracle Database, a WebLogic Domain or a WLS Managed Server. Using the omcli tool and a JSON file format for specifying the components, the entities are added. Based on the attributes provided for these entities and the built in knowledge in the cloud agent about the log files written to by the entity type indicated, the cloud agent knows which files are to be harvested for log entries that are sent to the OMC cloud.

I have created a JSON file weblogic_amispaas.json for registering the WebLogic Domain and the Admin server:

{
    "entities": 
[
{
"name": "amispaas_domain",
"type": "omc_weblogic_domain",
"displayName": "Domain_amispaas_domain",
"timezoneRegion": "CET",
"properties": 
{
"port": 
{
"displayName": "Port",
"value": "7001"
},
"domain_home": 
{
"displayName": "Domain Home",
"value": "/u01/data/domains/amispaas_domain"
},
"admin_server_host": 
{
"displayName": "Admin Server Host",
"value": "amispaas-db-java-stack-jaas-wls-
1.compute-amispaas.oraclecloud.internal"
},
"webLogic_home": 
{
"displayName": "WebLogic Home",
"value": "/u01/app/oracle/middleware/wlserver"
}
},
"associations": 
[
{
"assocType": "omc_contains",
"sourceEntityName": "amispaas_domain",
"sourceEntityType": "omc_weblogic_domain",
"destEntityName": "amispaas_adminserver",
"destEntityType": "omc_weblogic_j2eeserver"
}
]
},
{
        "name":"amispaas_adminserver",
        "type":"omc_weblogic_j2eeserver",
        "displayName":"Admin_Server_AMISPAAS",
        "timezoneRegion":"CET",
        "properties":{
                "host_name":
                        {"displayName":"Weblogic Host","value":"amispaas-db-
java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal"},
                "domain_home":
                        {"displayName":"Domain 
Home","value":"/u01/data/domains/amispaas_domain"},
                "listen_port":
                        {"displayName":"Listen Port","value":"9071"},
                "listen_port_enabled":
                        {"displayName":"Listen Port Enabled","value":"true"},
                "ssl_listen_port":
                        {"displayName":"SSL Listen Port","value":"9072"},
"server_names":
{"displayName":"Server 
Names","value":"amispaas_adminserver"}
        },
        "associations":[
                { "assocType":"omc_monitored_by", 
                  "sourceEntityName":"amispaas_adminserver", 
                  "sourceEntityType":"omc_weblogic_j2eeserver", 
                  "destEntityName":"amispaas_domain", 
                  "destEntityType":"omc_weblogic_domain"}
        ]
}
]
}

The entities are registered from file weblogic_amispaas.json using this command:

/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli add_entity agent /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/my_entities/weblogic_amispaas.json

I then created a second file for the managed server: oracle-loganalytics-weblogic-managedserver-entities.json

{
    "entities": 
[
{
        "name":"amispaas_server_1",
        "type":"omc_weblogic_j2eeserver",
        "displayName":"Managed Server 1 AMISPAAS",
        "timezoneRegion":"CET",
        "properties":{
                "host_name":
                        {"displayName":"Weblogic Host","value":"amispaas-db-
java-stack-jaas-wls-1.compute-amispaas.oraclecloud.internal"},
                "domain_home":
                        {"displayName":"Domain 
Home","value":"/u01/data/domains/amispaas_domain"},
                "listen_port":
                        {"displayName":"Listen Port","value":"9073"},
                "listen_port_enabled":
                        {"displayName":"Listen Port Enabled","value":"true"},
                "ssl_listen_port":
                        {"displayName":"SSL Listen Port","value":"9074"},
"server_names":
{"displayName":"Server 
Names","value":"amispaas_server_1"}
        },
        "associations":[
                { "assocType":"omc_monitored_by", 
                  "sourceEntityName":"amispaas_server_1", 
                  "sourceEntityType":"omc_weblogic_j2eeserver", 
                  "destEntityName":"amispaas_domain", 
                  "destEntityType":"omc_weblogic_domain"}
        ]
}
]
}

and used a similar command to add this entity (note: I could have added all entities in a single file with the same effect):

/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli add_entity agent /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/my_entities/weblogic_amispaas_managedserver1.json

Using

/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli config agent listtargets

I then checked whether all entities had been added:

image

 

In de OMC Log Analytics Console, the entities are visible as well:

image

At this point, the log file entries should start to appear in OMC.

Unfortunately, they did not. This turned out to be bug (that may well have been fixed by the time you are reading this).

Fix Bug in Configuration File

No log file entries were published at this point from the JCS VM. Investigation by Eugene van der Voort made clear that a problem existed in the out of the box agent configuration files:

1)      Agent Logfile with error messages: $AGENT_HOME/sysman/log/gcagent.log    ( /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/log/gcagent.log)

inspect log file /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/log/gcagent.log

2)      Locate Error in logfile:

2017-01-19 10:44:00,066 [61:CAD9F7AE] WARN – Some issues to read/parse /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/ApplicationsState/loganalytics/logrules_os_file.xml

java.util.regex.PatternSyntaxException: Unmatched closing ‘)’ near index 58

([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+\(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*).*

^

3)      Modify offending file: /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/sysman/ApplicationsState/loganalytics/logrules_os_file.xml

4)      Locate tag:

<Name>tm_server.log</Name>

<Regex>([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+\(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*)</Regex>

5)      Change regular expression to :

<Regex>([\d]{2}:[\d]{2}:[\d]{2},[\d]{3})\s+(.*?)\s+\[(.*)\]\s+(.*)\s+\[(.*)\]\s+([\d]{4})-([\d]{2})-([\d]{2})\s+([\d]{2}):([\d]{2}):([\d]{2}),([\d]{3})(.*)</Regex>

6)      Then stop and start the agent:

  1. omcli stop agent
  2. omcli start agent

Now the log files from the JCS VM finally became visible in Log Analytics:

image

 

The log file shows something of the activity of the agent:

image

it also indicates an existing problem with file path /var/log/audit. I need to check out that one. Perhaps I need to provide a Credential to the Cloud Agent.

image

Easy solution: provide read access on that path to user oracle. As root:

chmod 754 /var/log/audit

About Author

Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press books: Oracle SOA Suite 11g Handbook and Oracle SOA Suite 12c Handbook. Frequent presenter on community events and conferences such as JavaOne, Oracle Code and Oracle OpenWorld.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.