In my previous post Network considerations on an ODA X5-2 I described a solution where I created a private network to optimize network performance between the components running on an Oracle Database Appliance. I am talking about a Virtualized setup in this article, and rebuilding the second pair of public interfaces to become a private network bond, in order to get a faster connection between the virtual machines.
In this article I will describe the steps how to create a private network on an ODA X5-2 and configure an additional SCAN-listener for this private network.
Connect the nodes
Let’s start by connecting the two nodes of the ODA directly to each other using Cat. 6 Ethernet cables. Make sure that the cables are long enough to route them through the Cable Management Arm. This way you don’t have to disconnect the cables when you have to do maintenance on the nodes.
We will use the second pair of interfaces for the configuration of the private network. Connect interfaces net2 and net3 on node0 to net2 and net3 on node1. This will create a 10 Gb network between the two nodes.
Pict. 1 – Oracle Database Appliance X5-2 Back Panel with net2 and net3 connected to each other
Now actually, you have a private network configured that you can assign to the VM’s.
Configure the interface on the ODA_BASE
In Dom0 the pair of interfaces is known as net2 and is assigned to the ODA_BASE.
Within the ODA_BASE the interface is known as eth1.
In this example we will use subnet 192.168.10.0/24 for the private network.
Node0 : 192.168.10.10
Node1 : 192.168.10.11
There are several ways to assign an IP addresses to eth1 on the ODA_BASE on both nodes.
Using the Oracle Appliance Manager Configurator
If you are preparing for Oracle Database Appliance Installation and Deployment you configure the interface and ip-address during the deployment. The Oracle Appliance Manager Configurator will guide you through the process.
Using oakcli
If eth1 is unconfigured, you can use oakcli on the ODA_BASE of both nodes to configure it.
oakcli configure additionalnet
Provide Interface Name, DHCP, IP and Netmask
Manually change the IP address
Note. Don’t change the primary public IP addresses this way!!!
If you need to change the IP Address of the primary public interface, you need to follow the procedure described in
ODA: How to Change Oracle Database Appliance IP Addresses Post Deployment (Doc ID 1504734.1) on Oracle support.
If eth1 is already configured and you want to give it another IP address we can edit the ifcfg-eth1 file on the ODA_BASE of both nodes.
On Node0
cd /etc/sysconfig/network-scripts vi ifcfg-eth1 #auto generated by OAK DEVICE=eth1 ONBOOT=yes BOOTPROTO=none TYPE=ETHERNET IPADDR=192.168.10.10 NETMASK=255.255.255.0
Enable eth1 interface once.
ifup eth1
The next time the ODA_BASE is restarted, the interface will start automatically with the new address.
On Node1
cd /etc/sysconfig/network-scripts vi ifcfg-eth1 #auto generated by OAK DEVICE=eth1 ONBOOT=yes BOOTPROTO=none TYPE=ETHERNET IPADDR=192.168.10.11 NETMASK=255.255.255.0
Enable eth1 interface once.
ifup eth1
The next time the ODA_BASE is restarted, the interface will start automatically with the new address.
Note that the eth1 interface on the ODA_BASE is also known as net2 on the Dom0 and in the GRID infrastructure. Continuing this article I will use the term net2 for the private network.
Configure an additional SCAN Listener
When an ODA is installed and deployed, the Oracle Appliance Manager Configurator configures the SCAN listener on the interface defined as firstnet. Usually this will be net1.
Now that we have a private network, we wat to be able to use it for connections to the database. So we are going to add a SCAN listener to net2.
As a guide to creating another SCAN listener I am using ODA (Oracle Database Appliance): HowTo Configure Multiple Public Network on GI (Grid Infrastructure) 12c (Doc ID 2101109.1)
Configure hosts file and DNS
Before you configure the SCAN listeners we are going to add some entries to the /etc/hosts file of the ODA_BASE on both nodes
After initial deployment, the /etc/hosts file is filled with default values. For example.
# Following added by OneCommand 127.0.0.1 localhost.localdomain localhost # PUBLIC HOSTNAMES # PRIVATE HOSTNAMES 192.168.16.27 oda11-priv0.foo.bar oda11-priv0 192.168.16.28 oda12-priv0.foo.bar oda12-priv0 # VIP HOSTNAMES 10.10.1.13 oda11-vip.foo.bar oda11-vip 10.10.1.14 oda12-vip.foo.bar oda12-vip # NET(0-3) HOSTNAMES 10.10.1.10 oda11.foo.bar oda11 10.10.1.11 oda12.foo.bar oda12 # Below are SCAN IP addresses for reference. # SCAN_IPS=(10.10.1.7 10.10.1.8) # ASR entry 141.146.1.169 transport.oracle.com
Where 10.10.1.0/24 is the subnet for net1 in this example.
Add the VIP hostnames of the newly created private network on net2.
To make things easy, used the same hostname as the public hostname and added -int to it.
# INTERNAL HOSTNAMES, ADDED MANUALLY 192.168.10.10 oda11-int.foo.bar oda11-int 192.168.10.11 oda12-int.foo.bar oda12-int # INTERNAL VIP HOSTNAMES, ADDED MANUALLY 192.168.10.12 oda11-int-vip.foo.bar oda11-int-vip 192.168.10.11 oda12-int-vip.foo.bar oda12-int-vip
Also add the vip addess of the SCAN listener to DNS before you create the new SCAN listener. On an ODA by default there are two ip addresses registered for the SCAN listener, so for the new SCAN listener we will also register two addresses on the private interface.
192.168.10.7 oda1-int-scan 192.168.10.8 oda1-int-scan
Configure Listener in GI Infrastructure
Logon to the ODA_BASE of node0 as user grid
Set environment to GI
[grid@oda11 ~]$ . oraenv ORACLE_SID = [grid] ? +ASM1 The Oracle base has been set to /u01/app/grid
Set interface type to pubic for eth1
[grid@oda11 ~]$ oifcfg setif -global eth1/192.168.10.0:public [grid@odadbsandta01 ~]$ oifcfg getif eth0 10.10.1.0 global public ibbond0 192.168.16.0 global cluster_interconnect,asm eth1 192.168.10.0 global public
Logon to the ODA_BASE of node0 as user root
Set environment to GI
[root@odadbsandta01 ~]# . oraenv ORACLE_SID = [root] ? +ASM1 The Oracle base has been set to /u01/app/grid
Add a network and vip addresses to GI
[root@oda11 grid]# srvctl add network -netnum 2 -S 192.168.10.0/255.255.255.0/eth1 -w static [root@oda11 grid]# srvctl add vip -n oda11 -A oda11-int-vip.foo.bar /255.255.255.0 -netnum 2 [root@oda11 grid]# srvctl add vip -n oda12 -A oda12-int-vip.foo.bar/255.255.255.0 -netnum 2
Logon to the ODA_BASE of node0 as user grid
Set environment to GI
[grid@oda11 ~]$ . oraenv ORACLE_SID = [grid] ? +ASM1 The Oracle base has been set to /u01/app/grid
Start the newly created vip’s
[grid@oda11 ~]$ srvctl start vip -i oda11-int-vip.foo.bar [grid@oda11 ~]$ srvctl start vip -i oda12-int-vip.foo.bar
Check if vip’s are enabled and running on both nodes
[grid@oda11 ~]# srvctl status vip -n oda11 VIP oda11-vip.foo.bar is enabled VIP oda11-vip.foo.bar is running on node: oda11 VIP oda11-int-vip.foo.bar is enabled VIP oda11-int-vip.foo.bar is running on node: oda11 [grid@oda11 ~]# srvctl status vip -n oda12 VIP oda12-vip.foo.bar is enabled VIP oda12-vip.foo.bar is running on node: oda12 VIP oda12-int-vip.foo.bar is enabled VIP oda12-int-vip.foo.bar is running on node: oda12
Add a listener on the net2 interface and start it
[grid@oda11 ~]$ srvctl add listener -l INT_LISTENER -p 1521 -netnum 2 -s [grid@oda11 ~]$ srvctl start listener -l INT_LISTENER [grid@oda11 ~]$ srvctl status listener Listener INT_LISTENER is enabled Listener INT_LISTENER is running on node(s): oda12,oda11 Listener LISTENER is enabled Listener LISTENER is running on node(s): oda12,oda11
Logon to the ODA_BASE of node0 as user root
Set environment to GI
[root@odadbsandta01 ~]# . oraenv ORACLE_SID = [root] ? +ASM1 The Oracle base has been set to /u01/app/grid
Add SCAN and SCAN listener
[root@oda11 grid]# srvctl add scan -scanname oda1-int-scan -netnum 2 [root@oda11 grid]# srvctl start scan -netnum 2 [root@oda11 grid]# srvctl add scan_listener -netnum 2 -listener int -endpoints TCP:1521 [root@oda11 grid]# srvctl start scan_listener -netnum 2
[grid@oda11 ~]$ srvctl status scan SCAN VIP scan1 is enabled SCAN VIP scan1 is running on node oda12 SCAN VIP scan2 is enabled SCAN VIP scan2 is running on node oda12 [grid@oda11 ~]$ srvctl status scan_listener SCAN Listener LISTENER_SCAN1 is enabled SCAN listener LISTENER_SCAN1 is running on node oda12 SCAN Listener LISTENER_SCAN2 is enabled SCAN listener LISTENER_SCAN2 is running on node oda12
Register databases
In normal circumstances the databases should register themselves to the new listener. I have seen this working on an ODA X5-2 running version 12.1.2.4.0.
So normally you are done with the database part.
Unfortunately version 12.1.2.6.0 introduced a bug which prevents the registration of the database with the new listener.
The bug is related to bug 19884033 : ORA-2097 AND ORA-119 ATTEMPTING TO SET LISTENER_NETWORKS PARAMETER BY THE AGENT
Unfortunately the fix for this bug didn’t make it to the next release, nor the patchset of version 12.1.2.7.0.
DO NOT try to install the normal server patchset Patch 19884033!! I have tried this and almost ruined our ODA installation.
Please wait for a patch specifically for the ODA.
Waiting for the bugfix, Oracle support supplied the following workaround.
ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node1_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance1>'; ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node2_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance2>';
Login to both the DB instances and run:
SQL> alter system register;
I am still working with Oracle support to get a one-off patch for this issue.
Using the private network
To use the private network, just add another network adapter to the vm or vmtemplate.
oakcli configure vm <vm name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']" oakcli configure vmtemplate <vm template name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']"
In the VM the interfaces will be present as eth0 and eth1
Let’s take a look at the pictures I used in the previous blog, which are part of the design as an example.
In this design I configured the Oracle Traffic Director (OTD) and Fusion Middleware (in this case SOA Suite and MFT) VM’s with two interfaces.
The OTD is configured to listen on the public interface and routes traffic to the FMW components on the private interface.
The FMW components are configured to listen only on the private network. That way all traffic to the FMW components have to come from the OTD.
Also all cluster communication between the FMW components is routed through the private interface.
The FMW components route database traffic to and from the ODA_BASE through the private interface.
Note. The public interface on the FMW VM’s is only used for SSH.
Summary
By creating a private network on an ODA, you are not only able to improve performance. You are also capable of separating traffic from public to private. This creates a real one-box solution and, if implemented correctly, increases security. You are able to hide the back-end systems from the clients.
At customer sites where I have implemented the private network, they are very pleased with the solution.
Related Documents
Oracle Database Appliance Documentation
ODA: How to Change Oracle Database Appliance IP Addresses Post Deployment (Doc ID 1504734.1)
ODA (Oracle Database Appliance): HowTo Configure Multiple Public Network on GI (Grid Infrastructure) 12c (Doc ID 2101109.1)
Oracle White Paper on Oracle Single Client Access Name (SCAN)
Oracle Support Bug 19884033 : ORA-2097 AND ORA-119 ATTEMPTING TO SET LISTENER_NETWORKS PARAMETER BY THE AGENT
Network considerations on an ODA X5-2
