Collaborate 08 – Day 4 – The Final Day

Being the last day of the conference
there was a shorter program, only until early afternoon.

I must say, this has been a very
valuable conference for me. I have seen great presentations, received
very valuable information on current and future products of Oracle
and, not the least important, I met some great people.

Today I attended two sessions:

• RAC on E-Business Suite –
  Sweating the details – by Kurt Forshee, DCC Services, LLC
• Setup Management for E-Business
  Suite with iSetup – by Uma Prabhala, Oracle

RAC on E-Business Suite – Sweating the details

Kurt Forshee, DCC Services, LLC

Anyone wonder why I would attend this session? No kidding! I am an
expert in the same area, and wanted to hear about Kurt’s experiences.

Kurt started off with a short introduction what RAC on E-Business
Suite really is, what you can use it for and what not. His vision was
clear: Not for Cost savings, not for performance (where did I hear
that earlier?;-), probably for availability and most likely for
scalability. All of this because it is a complex product that
requires a lot of skills: OS, database, application, network, storage
to name a few. RAC isn’t cheap, it isn’t easy. So what should a
typical RAC environment consist of?

• Smaller machines (preferably
  Linux-based), because it is cheaper and easier to manage
• NAS Storage (over SAN storage), because it is cheaper and
  delivers almost equal performance as SAN
• ASM (over CFS, because of the caching mechanism which is
  optimized for Oracle, delivering much better performance)
• At least 3 nodes

To all of the above I can put some remarks to some extend. They are all true in a way, however, each of them should be considered in the context of the customers situation.
Smaller machines? Yes, but the more machines you add the less
performance (percentage wise) you will gain by scaling out. This for
sure is a negative side-effect of RAC and clustering in general. My
personal opinion on this is that you should keep the number of nodes
in your cluster as low as possible. This keeps the cluster management
overhead to a minimum, from which overall performance will
benefit.
NAS over SAN? I am quite OK with that, but coming to a
customer that already has a SAN in place, I will not be on the front
line to get this customer to change it’s storage solution to
NAS. Besides, NAS will have good performance, assuming that your
production environment has more or less exclusive access, and doesn’t
have to share it with other environments. I agree, there are
solutions to this as well; with a new infrastructure to build, my
personal preference will most likely be with NAS as well.
ASM over
CFS or raw? I don’t agree. If RAC is complex enough to handle for a
customer, adding another layer of complexity (ASM ain’t
easy either!) will not be beneficial, to say the least. On top of
that, the DBA will get some sort of responsibility for the storage
and it’s management; something I am not sure every DBA is a) waiting
for and b) can handle. Besides, when on a SAN, ASM will be the second
black box – the SAN being the first. That is asking for
fingerpointing in case of problems.
At least 3 nodes? for (3^rd party) Clusterware, I would agree, however Oracle RDBMS with RAC is perfectly able of providing so-called Single Node Quorum. It is not mandatory, however still recommended though.

Kurt is right in stating the above, at least when establishing a completely new infrastructure, however I have not seen many customers do that. Many times I meet customers that want to migrate their existing environment, or the project has been underway for a significant amount of time and changing the direction on the level of technology is usually not appreciated.
My view on this subject: When customers consider Real Applications Cluster technology for their E-Business Suite, it is of vital importance to get an Architect on board that knows E-Business Suite Technology, knows Real Applications Cluster technology and the adjacent technologies like OS, Storage and Network. As early in the project as possible! I also know these experts are not easily found. But that doesn’t make it less important. OK, sofar my sales talk 😉

When considering RAC for E-Business Suite, the first motto is:
Architecture, Architecture, Architecture. The second motto is: Think,
rethink and then rethink again. In case of an advanced architecture
that RAC implies, it is very important, if not vital, for success to
be very concious about your architecture. For example, in case of a
failure the cluster should consist of at least 3 nodes and the number
of nodes should always be odd. This doesn’t mean that it is necessary
to have 3 RAC nodes. 2 RAC nodes can handle a failure, one will
remain in case of a failure, but on Cluster level (Clusterware or 3^rd
pary cluster solution) you should have an odd number of nodes. This is because of so-called Quorum. A majority of the nodes should be
available in order to proceed. If not, then the cluster should go
down immediately. 1 out of 2 is not a majority. Therefore, in 2-node
active-active clusters, when a failure occurs on one of the nodes,
the second will go down with it.
If you don’t evaluate your architecture thoroughly, you will encounter issues with it after a while. And if this is after go-live, what are you going to do? Who will tell the management? I cannot emphasize enough. Go by the book (many articles have been written on cluster architecture), conform to what is recommended on every detail. And don’t save on budget there. You will pay for it later.

According to Kurt Forshee, the optimum number of nodes in a RAC
cluster for E-Business Suite would be 5. 3 of these should then be
used for OLTP activities and the remaining 2 for OLAP activities like
Discoverer etc. Apart from this, the sysadmin should establish node
affinity on module level for the OLTP nodes. This means that a
certain module will only be working on a given instance in the RAC
cluster, preventing nasty things like block pinging with concurrent
processes. Also, the Concurrent Managers should not be given the
806_BALANCE two-task, because that could cause concurrent management
activities of the same batch process to end up on two different nodes
which is disastrous for performance.

One of the other point Kurt was making, and I cannot agree more
with him is that you should have a carefully considered instance
strategy. At least one of your non-production environments should be
a mirror of your production environment, and when developing in a
separate chain of instances, there should be a RAC environment
available there as well. I agree with him on this point, based on
personal experience.

Kurt also dug a little deeper into the various software elements
that a RAC environment typically consists of, and gave a listing of
the possible options:

• OS
  -Kernel Parameters! So very important to have them
  tuned up
  -Clock Synchronization between RAC nodes
  -Linux: the
  Hangcheck Timer
  -If possible: Use 3^rd party
  clusterware, because a) I/O fencing is much better than with Oracle
  Clusterware, a Volume Manager can be used for logical volumes, NICs
  can be managed far better, and CRS control can be either disabled or
  deferred, giving it to the external cluster.
• Clusterware
  -Use a separate ORACLE_HOME (I thought this
  was mandatory?)
  -Host equivalency: Essential to have when
  installing, afterwards optional.
  -Network: Have at least 2
  private interconnect networks and 1 public
  -cluvfy (cluster
  verify utility): It often doesn’t work, as long as you did what is
  necesary, no problem! Just continue.
• ASM
  -Use a separate ORACLE_HOME
  -Use a separate
  listener (do not use LISTENER_<hostname> because this will be
  created by E-Business Suite autoconfig!)
  -Consider mirroring
  within ASM (surely for Voting Disk and Cluster Registry, optional
  for data disks)
  -Use an spfile
• RDBMS
  -Use 10gR2
  -Use OATM (the Oracle Applications
  Tablespace Model, introduced in 11.5.9)
  -Do not register with CRS
  so root cause analysis can be done after failure – I do not agree
  here, because it is easy to disable the auto-start feature that is
  enabled by default when registering the database to CRS.
  -Use the
  init.ora ifile for customizations, since autoconfig overwrites the
  init.ora.

Down here a simple copy and paste of one of the last slides with
general recommendations, which are very valuable indeed (I am sure he
will approve;-):

• Define SLA’s for performance and availability for each
  service or application
• Use Grid Control to manage CRS, ASM, Database and
  Application.
• All changes to the production environment must be previously
  tested on a separate
  environment
• Apply changes to one system element at a time, first on test
  then on production.
• Keep a detailed change log
• Implement services where possible to manage workload for OLAP
  and customizations
• Configure OSWatcher to have handy information about the OS
  layer in case of need, see on
  Metalink Note:301137.1, OS Watcher
  User Guide
• Configure RDA to have handy information to Oracle Support in
  case of need, see on
• Metalink Note:314422.1, Remote Diagnostic Agent Getting
  Started. RDA 4.5+ includes RAC
  data collection capability. It can
  be used in place of RAC Diagnostics tool RACDDT.
• Establish support mechanisms and escalation procedures.
• Make sure DBA’s have well tested procedures about how to deal
  with problems and collect
  required diagnostics.
• Use Racdiag.sql to check database during normal behavior and
  be able to compare results,
  see Metalink Note:135714.1

Setup Management for E-Business Suite with iSetup

Uma Prabhala, Oracle

I have been hearing a lot of iSetup lately. Especially with the
release of R12, iSetup was mentioned as a key feature of R12. I was
just very curious about what it does and how it works. I knew on a
high level already, but attending this session was worthwhile
indeed.

iSetup is positioned in the Product LifeCycle Management portfolio
of Oracle. Other products in this portfolio (for E-Business Suite)
are:

• Implementation
  – Rapid Install and Rapid Clone
  – iSetup
• Management
  – Oracle Applications Manager
  – Applications
  Management Pack for Oracle Enterprise Manager Grid Control
  –
  Oracle E-Business Suite Diagnostics
• Upgrade
  – Patch Wizard
  – Apps DBA Utilities

iSetup is used to

• transform (a subset of) setup information from one instance
  to another
• document reports on setups across timelines and/or instances

The key features of iSetup are

• Promoting functional configurations through CRP –
  Production
• Expediting implementations by providing out-of-the-box
  migration templates tailored to suit various implementation phases
• Orchestrating functional configuration deployment and
  enforcing business validation throughout the migration cycle
• Providing an enterprise wide repository to manage functional
  configuration snapshots per business and/or industry best practices.

The iSetup Migrator extracts functional configuration data, using
BC4J APIs, FNDLOADER and BSO, to optionally transform this data based
on given parameterization and upload it into
another E-Business Suite instance. This makes it easy to copy
functional setup across instances, without having to go through the
whole setup again. A really nice feature of this Migration tool is
that it performs Business Validation, which takes care of the
completeness of the set of data that is extracted from the source
environment, but it also validates the data uploaded into the
target.
It is possible to use filter criteria to exclude/include
certain setup data into the snapshots and also transforming (part of)
this data. One other key feature is that iSetup Migrator is able to
restart a failed (incomplete) load.

The iSetup Reporter can generate standard and
comparison reports, based on any snapshot. It also is able to filter
on objects. This makes it possible to perform a comparison between
two snapshots with overlapping data, only showing part of this
overlapping data. Reports – because BI technology is being used –
can be generated in PDF, RTF and Excel format.

The requirements for iSetup are as follows:

• 11.5.10 CU2
• AZ.H Minipack + delta 1
• Financials OU Transformations (I
  personally have no idea what this is, I am a technical guy;-)

iSetup comes with a set of templates in
different categories:

• Generic Templates
  General
  Foundation
  Product Foundation
  Financials
  etc.
• Operational
  Templates
  Suppliers
  Employees
  etc.
• Independent Templates
  Profile
  options
  Daily Rates
  XML Publisher
  etc.

iSetup can be a very powerful tool to manage
your functional setup across multiple instances. Uma gave a live demo
exporting data from one instance to the other and comparison reports
between two environments.

Very interesting stuff to watch and a good
conclusion to a very good conference
My compliments to the
Conference committee for organizing such a great event.