Creating and scaling Dynamic Clusters using wlst

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

In my previous article, Creating and scaling Dynamic Clusters in Weblogic 12c, I described the creation and scaling of Dynamic Clusters. I used the Weblogic Console to create the Dynamic Clusters and change the number of servers.

Most of the time you will use some wlst scripting to create and manage your Weblogic environments.
In this article I will show you how to create Dynamic Clusters en how you can scale them.

The example scripts from the Oracle documentation where used as base for the following script.
It is just a simple create script to show you how easy it is to create a Dynamic Cluster via wlst. So no fancy functions and exception handling in there. Yet …

createDynamicCluster.py

print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
serverTemplate='dyna-server-Template'
serverNamePrefix='dyna-server-'
listenAddress='192.168.100.4${id}'
listenPort=8000
listenPortSSL=9000
maxServerCount=2

print '--- Connect to the AdminServer'
try:
connect('weblogic','Welcome01','t3://wls01.domain.local:7001')
except err:
print "--- Can't connect to AdminServer, "+err
sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Creating the server template '+serverTemplate+' for the dynamic servers and set the attributes'
dynamicServerTemplate=cmo.createServerTemplate(serverTemplate)
dynamicServerTemplate.setListenAddress(listenAddress)
dynamicServerTemplate.setListenPort(listenPort)
dynamicServerTemplateSSL=dynamicServerTemplate.getSSL()
dynamicServerTemplate.setListenPort(listenPortSSL)

print '--- Creating the dynamic cluster '+clusterName+', set the number of dynamic servers and designate the server template to it.'
dynamicCluster=cmo.createCluster(clusterName)
dynamicServers=dynamicCluster.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)
dynamicServers.setServerTemplate(dynamicServerTemplate)

print '--- Designating the Cluster to the ServerTemplate'
dynamicServerTemplate.setCluster(dynamicCluster)

print '--- Set the servername prefix to '+serverNamePrefix
dynamicServers.setServerNamePrefix(serverNamePrefix)

print '--- Set Calculate Listen Port and Machinename based on server template'
dynamicServers.setCalculatedMachineNames(true)
dynamicServers.setCalculatedListenPorts(true)

print '--- Save and activate the changes'
save()
activate()
serverConfig()

Running the script with wlst will produce the following output and will create a Dynamic Cluster with two Dynamic Servers.

[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh createDynamicCluster.py
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Creating the server template dyna-server-Template for the dynamic servers and set the attributes
--- Creating the dynamic cluster dyna-cluster, set the number of dynamic servers and designate the server template to it.
--- Designating the Cluster to the ServerTemplate
--- Set the servername prefix to dyna-server-
--- Set Calculate Listen Port and Machinename based on server template
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As you might expect, it is way faster than clicking through the Weblogic Console.
Next step will be to scale the Dynamic Cluster up to four Dynamic Servers.

scaleDynamicCluster.py

print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
maxServerCount=4

print '--- Connect to the AdminServer'
try:
connect('weblogic','Welcome01','t3://wls01.domain.local:7001')
except err:
print "Can't connect to AdminServer, "+err
sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Change the maximum number of dynamic servers'
cd('/Clusters/%s' % clusterName )
dynamicServers=cmo.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)

print '--- Save and activate the changes'
save()
activate()
serverConfig()

Running the script with wlst will produce the following output and will scale up to four Dynamic Servers.

[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh scaleDynamicCluster.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

--- Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Change the maximum number of dynamic servers
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As mentioned before, the scripts are very limited and just show you how easy it is to create Dynamic Clusters using wlst. The scripts can be made as comprehensive as you need (want) them to be.
I will create some more examples and post them as I get them ready.

Imagine the possibilities when you create scripts you can connect to your monitoring system. Capacity on demand!

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

Creating and scaling Dynamic Clusters in Weblogic 12c

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

Introduced in Weblogic 12.1.2, dynamic clusters is a great feature to scale your private cloud.
Dynamic clusters provides you easy scaling of Weblogic clusters by adding and removing managed server instances on demand. They contain one or more dynamic servers. These dynamic servers are based on a single servertemplate that guarantees that every member of the cluster is exactly the same.

Creating Dynamic Clusters

Let’s take a look at some of the possibilities as we create a dynamic cluster.

I have created a virtualbox environment.
This environment consists of four VM’s with the following specs.

  • 2 vCPU’s
  • 4 Gb memory
  • 50 Gb disk
  • Oracle Linux 6.6
  • Java 1.7.0_75
  • Weblogic 12.1.3.0.2

I created a simple domain called demo_domain with only an AdminServer and four machines.
After unpacking the domain to the four servers, the nodemanagers where started and are reachable by the AdminServer.

Domain-pic1Now let go through the process of creating a dynamic cluster.

Open the Weblogic Console and navigate to Environment -> Clusters
Lock and Edit the domain in the Change Center
note. I make it a good practice always creating a domain in production mode, even in Development and Test.

Create a new dynamic cluster

Domain-cap1

New -> Dynamic Cluster

Provide the Clustername

Domain-cap2
Cluster name: dyna-cluster
Click Next

We will start of with a cluster containing two dynamic servers.

Domain-cap3
Number of Synamic Servers: 2
Server Name Prefix: dyna-server-
Click Next

For this demo all machines will take part.

Domain-cap4
Select ‘Use any machine configured in this domain’
Click Next

Assign each dynamic server unique listen ports

Domain-cap5
Listen Port for First Server: 8000
SSL Listen Port for First Server: 9000
Click Next

Summary screen

Domain-cap6
Click Finish

With the creation of the Dynamic Cluster there is also a Server Template created for it.

Server templates

A single server template provides the basis for the creation of the dynamic servers. Using this single template provides the possibility of every member being created with exactly the same attributes. Where some of the server-specific attributes like Servername, listen-ports, machines, etc. can be calculated based upon tokens.
You can pre-create server templates and let Weblogic clone one when a Dynamic Cluster is created.
When none is available a server template is create with the Dynamic Cluster. The name and the listen ports are the only server template attributes that you provide during Dynamic Cluster creation.

Before we activate the changes to the domain, we are going to make a change to the server template.
As an example we are going to demonstrate the use of tokens for server-specific configuration.

Navigate to Environment -> Clusters -> Server Templates

Domain-cap8
Click on the name: dyna-server-Template

We are going to use the ${ID} token in the Listen Address

Domain-cap10
Listen Address: 192.168.100.4${ID}
Click Save

The last digit of the listen address is used to make the listen address dynamic.

Activate changes in the Change Center of the Weblogic Console.
After activation the cluster and two managed servers are created.

Domain-cap12Domain-cap11

We can now start the two servers.

In the previous steps we have added a dynamic cluster with two dynamic servers, based on a single server template, to the domain.

Domain-pic2

Scaling a Dynamic Cluster

When the capacity is insufficient and you need to scale-up, you can add dynamic servers on demand.
It requires only a few clicks.

Navigate to Environment -> Clusters

Domain-cap12
Click dyna-cluster

On the Configuration tab go to the Servers tab

Domain-cap13
Change the Maximum Number of Dynamic Servers to: 4
Click save

Activate changes in the Change Center of the Weblogic Console.
After activation two Dynamic Servers are added to the Dynamic Cluster.

Start the two new Dynamic Servers and you have doubled you capacity.

Domain-cap14

Domain-pic3
Scaling down works exactly the same.
Just lower the Maximum Number of DynamicServers and activate.

A few points to keep in mind when scaling up or down.

Up

  • New dynamic servers are not started upon creation
  • Think before you act with the use of Tokens.
    For example.
    In our demo, the number of Dynamic servers can’t grow beyond nine servers, since we use the ${ID} as last digit of the listen address.

Down

  • Dynamic Servers above the new Maximum have to be shutdown before the change can be activated.
  • Dynamic Servers are removed in order, Last -> First
    (In our demo dyna-server-4 gets removed first, then dyna-server-3, etc..)
  • You cannot remove a Dynamic Server directly from the Environment -> Servers page
Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

Cloud Control Agent 12c managing many objects

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

An Oracle Enterprise Management Agent 12c that has to manage hundreds of objects needs extra tweaking. If not, it will not start, or it will die soon, leaving you with an unmanaged infrastructure.

Situation

The customer runs over 200 databases on 1 host. Intel bases host with 384GB RAM and over 4TB of storage. Linux OS. Works as a charm.
Enterprise Manager 12c Release 4, Agent Version 12.1.0.4.0.

The databases should be managed with Oracle Cloud Control 12c. Installing an agent on the database host was done as a routine job. The next step would be to (auto)discover the databases and adding them to the repository. Based on some previous experiences I decided to add them in random chunks of about 20 per time, wait until all databases were managed (the green arrow pointing upward) and proceed with the next random chunk.

This went very well for the first 100-ish databases but then suddenly the newly added databases did not appear in the main databases screen as managed. Even worse: all databases eventually became unmanaged.

Investigation

It looked like the agent was not running so I logged on to the database host to investigate further. The agent was down indeed. So, I tried to start it again. After waiting and waiting and seeing the number of dotted lines grow to five or six (!) the startup eventually failed.

Time for deeper investigation. I’ll save you the failed attempts (and my frustration about that) and skip to: I completely removed the agent installation from the host and from the Enterprise repository and redid it. That went smooth as ever. I started adding databases to the repository and in the mean time frantically checked the status of the agent.

[oracle@s-xxxx-db-11 oracle]$ /u01/app/oracle/Agent12c/agent_inst/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/Agent12c/agent_inst
Agent Log Directory : /u01/app/oracle/Agent12c/agent_inst/sysman/log
Agent Binaries : /u01/app/oracle/Agent12c/core/12.1.0.4.0
Agent Process ID : 41118
Parent Process ID : 41039
Agent URL : https://s-xxxx-db-11:3872/emd/main/
Local Agent URL in NAT : https://s-xxxx-db-11:3872/emd/main/
Repository URL : https://s-xxxx-em-01.xxx.com:4903/empbs/upload
Started at : 2014-11-03 16:07:51
Started by user : oracle
Operating System : Linux version 2.6.32-504.el6.x86_64 (amd64)
Last Reload : (none)
Last successful upload : 2014-11-03 16:49:59
Last attempted upload : 2014-11-03 16:49:59
Total Megabytes of XML files uploaded so far : 10
Number of XML files pending upload : 280
Size of XML files pending upload(MB) : 0.25
Available disk space on upload filesystem : 32.81%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2014-11-03 16:48:58
Last successful heartbeat to OMS : 2014-11-03 16:48:58
Next scheduled heartbeat to OMS : 2014-11-03 16:49:59
Receivelet Interaction Manager Current Activity: Outstanding receivelet event tasks
----------------------------------
TargetID = oracle_database.VALIDD.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:49 CET
TargetID = oracle_database.PSP.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:54 CET
TargetID = oracle_database.VACQD.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:45 CET
TargetID = oracle_database.TSM14SP.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:37 CET
TargetID = oracle_database.ZOND.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:54 CET
TargetID = oracle_database.TSM13RT.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:34 CET
TargetID = oracle_database.skbc.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:51 CET
TargetID = oracle_database.TSM17RT.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:48:50 CET
TargetID = oracle_database.MGRMP.xxx.com - EventType - TARGET_EVENT for operation SAVE_TARGET submitted at 2014-11-03 16:46:10 CET

Target Manager Current Activity : Compute Dynamic Properties (total operations: 37, active: 9, finished: 28)

Current target operations in progress
-------------------------------------
oracle_database.ZOND.xxx.com - ADD_TARGET running for 261 seconds
oracle_database.TSM14SP.xxx.com - ADD_TARGET running for 261 seconds
oracle_database.PSP.xxx.com - ADD_TARGET running for 261 seconds
oracle_database.skbc.xxx.com - ADD_TARGET running for 261 seconds
oracle_database.VALIDD.xxx.com - ADD_TARGET running for 262 seconds
oracle_database.MGRMP.xxx.com - ADD_TARGET running for 262 seconds
oracle_database.TSM13RT.xxx.com - ADD_TARGET running for 262 seconds
oracle_database.VACQD.xxx.com - ADD_TARGET running for 262 seconds
oracle_database.TSM17RT.xxx.com - ADD_TARGET running for 262 seconds

Dynamic property executor tasks running
------------------------------
---------------------------------------------------------------
Agent is Running and Ready

This was new to me. And allthough the agent claimed to be running and ready, in reality it was useless.
Eventually we raised an SR at Oracle. Their response was to the point but no solution; I’ll quote the important part here:

Agent will be configured to start with minimum memory allocation.
Whenever the memory configured is not enough for the agent opereations, agent will restart and during restart auto tune and increase the default memory allocated to higher value so as to fulfil the requirement.
The memory allocated will not be sufficient when the number of targets monitored are high and hence need to set more memory for agent .

Also when starting the Agent, agent will collect and load metedata of all its targets and then only report the status as RUNNING and READY. When the targets are high, this may take few minutes and hence if the status is checked immediately after start up, it report ‘Agent is Running but not ready’. It will report status as Ready after the collection is completed and this is not an issue.

Perform the steps below to increase the Agent memory settings

1.Stop the Agent

2.take a backup and edit /sysman/config/emd.properties file
Change
agentJavaDefines=-Xmx673M -XX:MaxPermSize=96M
to
agentJavaDefines=-Xmx1024M -XX:MaxPermSize=96M

Save the file

3.Start the Agent and monitor its status
/bin>./emctl start agent
after 5- 10 minutes
/bin>./emctl status agent
/bin>./emctl upload

This can be found in the manual and I had already tried it. I even went as far as Xmx10240M, in small steps, but there was no noticable difference.
Apart from the SR I managed to contact Kellyn Pot’Vin, the author of an interesting Oracle blog. She seemed very knowlegable when it comes to Enterprise Manager related stuff. Her first attempt was to have me check a new option in Enterprise Manager to view the data from the collections graph and look at that performance analysis to see what may be backlogging collections and impacting the performance. Sadly I just got an empty graph.

Solution

Upon studying my logfiles she came up with a simple question:

Could you send me the results of:

ulimit -Su
ulimit -Hu

Okay:

[oracle@s-xxxx-db-11 bin]$ ulimit -Su
8192
[oracle@s-xxxx-db-11 bin]$ ulimit -Hu
3100271

Her next request was da bomb:

Could you set both of these to unlimited and restart the agent?

Bingo! The agent started in a jiffy, not even half a line of dots. Apart from that, it was running smooth and I could easily add another 100 databases to the repository. Problem solved!
Off course, afterward you need to have your system administrator find the right setting for those ulimits, which in our case turned out to be only slightly higher than the above published.

Not only did the agent start fast, it kept running as it’s supposed to do, for weeks now.
This information was not in the manual. Maybe it will eventually. Many thanks to Kellyn!

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

Christmas Masterclass Oracle SOA Suite 12c

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

On Friday, December 19, AMIS organises a special XMas-terclass on SOA Suite 12.

SOA Suite 12c (June 2014) was a major release for Oracle’s flagship integration product. This release introduces new functionality, higher developer productivity, more robust run-time scalability and performance. 12c brings together development and administration of Service Bus and SOA composite applications, it ships new technology adapters as well as a SDK to develop custom adapters and provides across the stack support for REST-style services and JSON-format messages. New facilities were added to the stack, like Managed File Transfer and Enterprise Scheduling Services, and others better integrated, such as Oracle Event Processing and Business Activity Monitoring.

This masterclass, based on over a year’s beta program participation and extensive research and experimentation, deals with many of the essential topics for SOA Suite 12c. In one day, presenters Robert van Mölken and Lucas Jellema will discuss how SOA Suite 12c will change and enhance the way SOA projects develop services. They will show many live demonstrations of the tools in action – providing you concrete insight in new and improved features. At the end of this masterclass, attendees will know how SOA Suite 12c can benefit their organizations and how they can get started with building skills, migrating applications and using this new release properly.

Throughout the day, participants and presenters have enough opportunity to discuss real world experience and brainstorm on how to best make use of SOA Suite 12c in their organizations.

All demo code and slides will be made available. Participants will also receive a voucher for the Oracle SOA Suite 12c Handbook by Lucas Jellema (Oracle Press, Spring 2015).

The SOA Suite 12c Xmas-terclass is aimed at developers, solution architects and administrators with SOA Suite 11g experience.

More information & registration

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

MAF 2.0 : Loading Images in a Background Process – Part I

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page

Images are heavily used in Mobile apps. For instance a list that contains employees usually shows the images of these employees. This works well when you have a WIFI connection, but what if you are using slower connections ? If you look at a mobile twitter app you will see that, depending on connectivity, images are loaded instantaneously or delayed. In this post I explain how you can load the images of a List in a background process, after the other content has been loaded. Continue reading

Share this on .. Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someoneShare on TumblrBuffer this page