Advertisements
Skip to content
February 28, 2015 / Shivananda Rao P

Node Addition in RAC 11gR2

This article demonstrates on adding a node to the RAC cluster.

Environment:
OS: OEL Linux 5.5
Existing Nodes : 10gnode1, 10gnode2
Node to be added: 10gnode3
Database Name: PRIMT
DB Version: 11.2.0.3

Here is the details of Public, Private and Virtual IP addresses of the all the nodes in the cluster environment. This is same across all the 3 nodes.

 

[oracle@10gnode1 ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

# PUBLIC
192.168.2.101 10gnode1.mydomain 10gnode1
192.168.2.102 10gnode2.mydomain 10gnode2
192.168.2.103 10gnode3.mydomain 10gnode3
# PRIVATE
192.168.0.101 10gnode1-priv.mydomain 10gnode1-priv
192.168.0.102 10gnode2-priv.mydomain 10gnode2-priv
192.168.0.103 10gnode3-priv.mydomain 10gnode3-priv
# VIRTUAL
192.168.2.111 10gnode1-vip.mydomain 10gnode1-vip
192.168.2.112 10gnode2-vip.mydomain 10gnode2-vip
192.168.2.113 10gnode3-vip.mydomain 10gnode3-vip

# SCAN
192.168.2.203 node-scan.mydomain node-scan
192.168.2.204 node-scan.mydomain node-scan
192.168.2.205 node-scan.mydomain node-scan

Next step would be to verify and establish passwordless SSH between the running nodes and the new node.

 


[oracle@10gnode1 ~]$ ssh 10gnode3 date
The authenticity of host '10gnode3 (192.168.2.103)' can't be established.
RSA key fingerprint is d2:e4:b0:df:11:11:cb:1d:16:15:25:79:f7:be:96:04.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10gnode3,192.168.2.103' (RSA) to the list of known hosts.
Sat Jan 24 17:43:37 IST 2015
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$ ssh 10gnode3 hostname
10gnode3.mydomain
[oracle@10gnode2 ~]$ ssh 10gnode3 date
The authenticity of host '10gnode3 (192.168.2.103)' can't be established.
RSA key fingerprint is d2:e4:b0:df:11:11:cb:1d:16:15:25:79:f7:be:96:04.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10gnode3,192.168.2.103' (RSA) to the list of known hosts.
Sat Jan 24 17:44:01 IST 2015
[oracle@10gnode2 ~]$
[oracle@10gnode2 ~]$ ssh 10gnode3 hostname
10gnode3.mydomain
[oracle@10gnode3 ~]$ ssh 10gnode1 date
Sat Jan 24 17:44:17 IST 2015
[oracle@10gnode3 ~]$
[oracle@10gnode3 ~]$ ssh 10gnode1 hostname
10gnode1.mydomain
[oracle@10gnode3 ~]$
[oracle@10gnode3 ~]$ ssh 10gnode2 date
Sat Jan 24 17:44:27 IST 2015
[oracle@10gnode3 ~]$
[oracle@10gnode3 ~]$ ssh 10gnode2 hostname
10gnode2.mydomain
[oracle@10gnode3 ~]$

Using CLUVFY, let’s check the pre-requisite on the new node. This is done by running the following the command on the running node.

 


cluvfy stage -pre nodeadd -n <new-node-to-be-added> -verbose


[oracle@10gnode1 ~]$ cd $ORA_CRS_HOME/bin
[oracle@10gnode1 bin]$ pwd
/u01/app/11.2.0/grid3/bin
[oracle@10gnode1 bin]$
[oracle@10gnode1 bin]$ cluvfy stage -pre nodeadd -n 10gnode3 -verbose
[oracle@10gnode1 bin]$ ./cluvfy stage -pre nodeadd -n 10gnode3 -verbose

Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "10gnode1"
Destination Node Reachable?
------------------------------------ ------------------------
10gnode3 yes
Result: Node reachability check passed from node "10gnode1"
Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Status
------------------------------------ ------------------------
10gnode3 passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
10gnode1 passed
10gnode2 passed
10gnode3 passed

Verification of the hosts config file successful
Interface information for node "10gnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.101 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth0 192.168.2.203 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth0 192.168.2.111 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth1 192.168.0.101 192.168.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500
eth1 169.254.125.10 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500
eth2 192.168.75.103 192.168.75.0 0.0.0.0 UNKNOWN 08:00:27:73:78:FF 1500
Interface information for node "10gnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.102 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500
eth0 192.168.2.112 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500
eth1 192.168.0.102 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500
eth1 169.254.60.182 169.254.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500
eth2 192.168.75.102 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:76:C0:28 1500
Interface information for node "10gnode3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.103 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:53:19:C6 1500
eth1 192.168.0.103 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:BA:22:46 1500
eth2 192.168.75.104 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:22:DE:35 1500
Check: Node connectivity for interface "eth0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1[192.168.2.101] 10gnode1[192.168.2.203] yes
10gnode1[192.168.2.101] 10gnode1[192.168.2.111] yes
10gnode1[192.168.2.101] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.101] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.101] 10gnode3[192.168.2.103] yes
10gnode1[192.168.2.203] 10gnode1[192.168.2.111] yes
10gnode1[192.168.2.203] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.203] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.203] 10gnode3[192.168.2.103] yes
10gnode1[192.168.2.111] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.111] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.111] 10gnode3[192.168.2.103] yes
10gnode2[192.168.2.102] 10gnode2[192.168.2.112] yes
10gnode2[192.168.2.102] 10gnode3[192.168.2.103] yes
10gnode2[192.168.2.112] 10gnode3[192.168.2.103] yes
Result: Node connectivity passed for interface "eth0"
Check: TCP connectivity of subnet "192.168.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1:192.168.2.101 10gnode1:192.168.2.203 passed
10gnode1:192.168.2.101 10gnode1:192.168.2.111 passed
10gnode1:192.168.2.101 10gnode2:192.168.2.102 passed
10gnode1:192.168.2.101 10gnode2:192.168.2.112 passed
10gnode1:192.168.2.101 10gnode3:192.168.2.103 passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "10gnode1"
The Oracle Clusterware is healthy on node "10gnode2"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
PRVG-1013 : The path "/u01/app/11.2.0/grid3" does not exist or cannot be created on the nodes to be added
Result: Shared resources check for node addition failed
Checking node connectivity...

Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
10gnode1 passed
10gnode2 passed
10gnode3 passed

Verification of the hosts config file successful
Interface information for node "10gnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.101 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth0 192.168.2.203 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth0 192.168.2.111 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500
eth1 192.168.0.101 192.168.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500
eth1 169.254.125.10 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500
eth2 192.168.75.103 192.168.75.0 0.0.0.0 UNKNOWN 08:00:27:73:78:FF 1500
Interface information for node "10gnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.102 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500
eth0 192.168.2.112 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500
eth1 192.168.0.102 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500
eth1 169.254.60.182 169.254.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500
eth2 192.168.75.102 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:76:C0:28 1500
Interface information for node "10gnode3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.2.103 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:53:19:C6 1500
eth1 192.168.0.103 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:BA:22:46 1500
eth2 192.168.75.104 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:22:DE:35 1500
Check: Node connectivity for interface "eth0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1[192.168.2.101] 10gnode1[192.168.2.203] yes
10gnode1[192.168.2.101] 10gnode1[192.168.2.111] yes
10gnode1[192.168.2.101] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.101] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.101] 10gnode3[192.168.2.103] yes
10gnode1[192.168.2.203] 10gnode1[192.168.2.111] yes
10gnode1[192.168.2.203] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.203] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.203] 10gnode3[192.168.2.103] yes
10gnode1[192.168.2.111] 10gnode2[192.168.2.102] yes
10gnode1[192.168.2.111] 10gnode2[192.168.2.112] yes
10gnode1[192.168.2.111] 10gnode3[192.168.2.103] yes
10gnode2[192.168.2.102] 10gnode2[192.168.2.112] yes
10gnode2[192.168.2.102] 10gnode3[192.168.2.103] yes
10gnode2[192.168.2.112] 10gnode3[192.168.2.103] yes
Result: Node connectivity passed for interface "eth0"
Check: TCP connectivity of subnet "192.168.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1:192.168.2.101 10gnode1:192.168.2.203 passed
10gnode1:192.168.2.101 10gnode1:192.168.2.111 passed
10gnode1:192.168.2.101 10gnode2:192.168.2.102 passed
10gnode1:192.168.2.101 10gnode2:192.168.2.112 passed
10gnode1:192.168.2.101 10gnode3:192.168.2.103 passed
Result: TCP connectivity check passed for subnet "192.168.2.0"
Check: Node connectivity for interface "eth1"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1[192.168.0.101] 10gnode2[192.168.0.102] yes
10gnode1[192.168.0.101] 10gnode3[192.168.0.103] yes
10gnode2[192.168.0.102] 10gnode3[192.168.0.103] yes
Result: Node connectivity passed for interface "eth1"
Check: TCP connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
10gnode1:192.168.0.101 10gnode2:192.168.0.102 passed
10gnode1:192.168.0.101 10gnode3:192.168.0.103 passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 2.9718GB (3116120.0KB) 1.5GB (1572864.0KB) passed
10gnode1 2.9718GB (3116120.0KB) 1.5GB (1572864.0KB) passed
Result: Total memory check passed

Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 2.9093GB (3050644.0KB) 50MB (51200.0KB) passed
10gnode1 1.9784GB (2074528.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed

Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 9.767GB (1.0241396E7KB) 2.9718GB (3116120.0KB) passed
10gnode1 9.767GB (1.0241396E7KB) 2.9718GB (3116120.0KB) passed
Result: Swap space check passed

Check: Free disk space for "10gnode3:/u01/app/11.2.0/grid3"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/11.2.0/grid3 10gnode3 /u01 18.6406GB 5.5GB passed
Result: Free disk space check passed for "10gnode3:/u01/app/11.2.0/grid3"

Check: Free disk space for "10gnode1:/u01/app/11.2.0/grid3"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/11.2.0/grid3 10gnode1 /u01 9.3252GB 5.5GB passed
Result: Free disk space check passed for "10gnode1:/u01/app/11.2.0/grid3"

Check: Free disk space for "10gnode3:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp 10gnode3 / 26.9141GB 1GB passed
Result: Free disk space check passed for "10gnode3:/tmp"

Check: Free disk space for "10gnode1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp 10gnode1 / 26.917GB 1GB passed
Result: Free disk space check passed for "10gnode1:/tmp"

Check: User existence for "oracle"
Node Name Status Comment
------------ ------------------------ ------------------------
10gnode3 passed exists(500)
10gnode1 passed exists(500)

Checking for multiple users with UID value 500
Result: Check for multiple users with UID value 500 passed
Result: User existence check passed for "oracle"

Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 5 3,5 passed
10gnode1 5 3,5 passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
10gnode1 hard 65536 65536 passed
10gnode3 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
10gnode1 soft 1024 1024 passed
10gnode3 soft 1024 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
10gnode1 hard 16384 16384 passed
10gnode3 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
10gnode1 soft 2047 2047 passed
10gnode3 soft 2047 2047 passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 i686 x86 passed
10gnode1 i686 x86 passed
Result: System architecture check passed

Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 2.6.18-8.el5 2.6.18 passed
10gnode1 2.6.18-8.el5 2.6.18 passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 250 250 250 passed
10gnode3 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 32000 32000 32000 passed
10gnode3 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 100 100 100 passed
10gnode3 100 100 100 passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 128 128 128 passed
10gnode3 128 128 128 passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 2147483648 2147483648 1595453440 passed
10gnode3 2147483648 2147483648 1595453440 passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 4096 4096 4096 passed
10gnode3 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 2097152 2097152 2097152 passed
10gnode3 2097152 2097152 2097152 passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 6815744 6815744 6815744 passed
10gnode3 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
10gnode3 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 1048576 1048576 262144 passed
10gnode3 1048576 1048576 262144 passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 4194304 4194304 4194304 passed
10gnode3 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 262144 262144 262144 passed
10gnode3 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 1048576 1048576 1048576 passed
10gnode3 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
10gnode1 1048576 1048576 1048576 passed
10gnode3 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "ocfs2-tools"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 ocfs2-tools-1.2.7-1.el5 ocfs2-tools-1.2.7 passed
10gnode1 ocfs2-tools-1.2.7-1.el5 ocfs2-tools-1.2.7 passed
Result: Package existence check passed for "ocfs2-tools"

Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 make-3.81-1.1 make-3.81 passed
10gnode1 make-3.81-1.1 make-3.81 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 binutils-2.17.50.0.6-2.el5 binutils-2.17.50.0.6 passed
10gnode1 binutils-2.17.50.0.6-2.el5 binutils-2.17.50.0.6 passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 gcc-4.1.2-48.el5 gcc-4.1.2 passed
10gnode1 gcc-4.1.2-48.el5 gcc-4.1.2 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 gcc-c++-4.1.2-48.el5 gcc-c++-4.1.2 passed
10gnode1 gcc-c++-4.1.2-48.el5 gcc-c++-4.1.2 passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "libgomp"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libgomp-4.4.0-6.el5 libgomp-4.1.2 passed
10gnode1 libgomp-4.4.0-6.el5 libgomp-4.1.2 passed
Result: Package existence check passed for "libgomp"

Check: Package existence for "libaio"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libaio-0.3.106-5 libaio-0.3.106 passed
10gnode1 libaio-0.3.106-5 libaio-0.3.106 passed
Result: Package existence check passed for "libaio"

Check: Package existence for "glibc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 glibc-2.5-49 glibc-2.5-24 passed
10gnode1 glibc-2.5-49 glibc-2.5-24 passed
Result: Package existence check passed for "glibc"

Check: Package existence for "compat-libstdc++-33"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed
10gnode1 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33"

Check: Package existence for "elfutils-libelf"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed
10gnode1 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed
Result: Package existence check passed for "elfutils-libelf"

Check: Package existence for "elfutils-libelf-devel"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed
10gnode1 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 glibc-common-2.5-49 glibc-common-2.5 passed
10gnode1 glibc-common-2.5-49 glibc-common-2.5 passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 glibc-devel-2.5-12 glibc-devel-2.5 passed
10gnode1 glibc-devel-2.5-12 glibc-devel-2.5 passed
Result: Package existence check passed for "glibc-devel"

Check: Package existence for "glibc-headers"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 glibc-headers-2.5-12 glibc-headers-2.5 passed
10gnode1 glibc-headers-2.5-12 glibc-headers-2.5 passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "libaio-devel"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libaio-devel-0.3.106-5 libaio-devel-0.3.106 passed
10gnode1 libaio-devel-0.3.106-5 libaio-devel-0.3.106 passed
Result: Package existence check passed for "libaio-devel"

Check: Package existence for "libgcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libgcc-4.1.2-48.el5 libgcc-4.1.2 passed
10gnode1 libgcc-4.1.2-48.el5 libgcc-4.1.2 passed
Result: Package existence check passed for "libgcc"

Check: Package existence for "libstdc++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libstdc++-4.1.2-48.el5 libstdc++-4.1.2 passed
10gnode1 libstdc++-4.1.2-48.el5 libstdc++-4.1.2 passed
Result: Package existence check passed for "libstdc++"

Check: Package existence for "libstdc++-devel"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 libstdc++-devel-4.1.2-48.el5 libstdc++-devel-4.1.2 passed
10gnode1 libstdc++-devel-4.1.2-48.el5 libstdc++-devel-4.1.2 passed
Result: Package existence check passed for "libstdc++-devel"

Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 sysstat-7.0.2-3.el5 sysstat-7.0.2 passed
10gnode1 sysstat-7.0.2-3.el5 sysstat-7.0.2 passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
10gnode3 ksh-20060214-1.4 ksh-20060214 passed
10gnode1 ksh-20060214-1.4 ksh-20060214 passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
10gnode3 passed
10gnode1 passed

Check for consistency of root user's primary group passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency
Result: Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
10gnode3,10gnode1
Result: Clock synchronization check using Network Time Protocol(NTP) failed
Checking to make sure user "oracle" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
10gnode3 passed does not exist
10gnode1 passed does not exist
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
PRVF-5622 : search entry does not exist in file "/etc/resolv.conf" on nodes: "10gnode3"
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
10gnode1 failed
10gnode3 failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: 10gnode1,10gnode3

File "/etc/resolv.conf" is not consistent across nodes
Pre-check for node addition was unsuccessful on all the nodes.

From the above output, I see that all checks have been passed except the “DNS response time”. I’m ignoring this in my case as I haven’t configured DNS and not using this.
Now, let’s add the new node using the “addNode.sh” using “silent” method. The parameters passed here are “CLUSTER_NEW_NODES” which is the new node that needs to be added and “CLUSTER_NEW_VIRTUAL_HOSTNAMES” which is the VIP alias of the new node to be added.

addNode.sh is located at “GRID_HOME/oui/bin” and is run from one of the running nodes. I’m running it here from 10gnode1.

 


[oracle@10gnode1 ~]$ cd /u01/app/11.2.0/grid3/oui/bin
[oracle@10gnode1 bin]$
[oracle@10gnode1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={10gnode3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={10gnode3-vip}" -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes 10gnode2,10gnode3 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/11.2.0/grid3
New Nodes
Space Requirements
New Nodes
10gnode3
/u01: Required 5.16GB : Available 17.36GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.4
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Server) 11.2.0.3.0
Installation Plugin Files 11.2.0.3.0
Universal Storage Manager Files 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Automatic Storage Management Assistant 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.3.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Oracle Net Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Enterprise Manager plugin Common Files 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.3.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Cluster Verification Utility Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle LDAP administration 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
PL/SQL 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Cluster Ready Services Files 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Saturday, January 24, 2015 7:13:26 PM IST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Saturday, January 24, 2015 7:13:29 PM IST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Saturday, January 24, 2015 7:22:28 PM IST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/11.2.0/grid3/root.sh #On nodes 10gnode3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0/grid3 was successful.
Please check '/tmp/silentInstall.log' for more details.

We need to run root.sh script on the new node (10gnode3) as ROOT user.

 

[root@10gnode3 ~]# /u01/app/11.2.0/grid3/root.sh
Check /u01/app/11.2.0/grid3/install/root_10gnode3.mydomain_2015-01-24_19-25-31.log for the output of root script
[root@10gnode3 ~]#

Let’s check the status of the resources on the newly added node.

 

[root@10gnode3 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.asm
ONLINE ONLINE 10gnode1 Started
ONLINE ONLINE 10gnode2 Started
ONLINE ONLINE 10gnode3 Started
ora.gsd
OFFLINE OFFLINE 10gnode1
OFFLINE OFFLINE 10gnode2
OFFLINE OFFLINE 10gnode3
ora.net1.network
ONLINE ONLINE 10gnode1
ONLINE ONLINE 10gnode2
ONLINE ONLINE 10gnode3
ora.ons
ONLINE ONLINE 10gnode1
ONLINE ONLINE 10gnode2
ONLINE ONLINE 10gnode3
ora.registry.acfs
ONLINE ONLINE 10gnode1
ONLINE ONLINE 10gnode2
ONLINE ONLINE 10gnode3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.10gnode1.vip
1 ONLINE ONLINE 10gnode1
ora.10gnode2.vip
1 ONLINE ONLINE 10gnode2
ora.10gnode3.vip
1 ONLINE ONLINE 10gnode3
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE 10gnode2
ora.cvu
1 ONLINE ONLINE 10gnode3
ora.oc4j
1 ONLINE ONLINE 10gnode2
ora.primt.db
1 ONLINE ONLINE 10gnode1 Open
2 ONLINE ONLINE 10gnode2 Open
ora.scan1.vip
1 ONLINE ONLINE 10gnode2

Run the CLUVFY utility to verify everything is fine after adding the new node to the cluster.

 


[oracle@10gnode1 ~]$ cd /u01/app/11.2.0/grid3/bin
[oracle@10gnode1 bin]$
[oracle@10gnode1 bin]$ ./cluvfy stage -post nodeadd -n 10gnode3

============================DB=============================

 

Now it’s time to add the DB Home installed on the new node, in other words we need to add the new node needs to be added to the DB cluster.

 

[oracle@10gnode1 ~]$ cd /u01/app/oracle/product/11.2.0/db3/oui/bin/
[oracle@10gnode1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={10gnode3}" -silent

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "10gnode1"
Checking user equivalence...
User equivalence check passed for user "oracle"

WARNING:
Node "10gnode3" already appears to be part of cluster

Pre-check for node addition was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes 10gnode2,10gnode3 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/oracle/product/11.2.0/db3
New Nodes
Space Requirements
New Nodes
10gnode3
/u01: Required 4.14GB : Available 13.02GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.5.0.1
Oracle ODBC Driverfor Instant Client 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
SSL Required Support Files for InstantClient 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.3.0
Oracle Database Vault J2EE Application 11.2.0.3.0
Oracle Label Security 11.2.0.3.0
Oracle Data Mining RDBMS Files 11.2.0.3.0
Oracle OLAP RDBMS Files 11.2.0.3.0
Oracle OLAP API 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle Database Vault option 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.3.0
Oracle SQL Developer 11.2.0.3.0
Oracle Application Express 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
SQLJ Runtime 11.2.0.3.0
Database Workspace Manager 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
Enterprise Manager Repository Core Files 10.2.0.4.4
Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
Enterprise Manager Grid Control Core Files 10.2.0.4.4
Enterprise Manager Common Core Files 10.2.0.4.4
Enterprise Manager Agent Core Files 10.2.0.4.4
RDBMS Required Support Files 11.2.0.3.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Parser Generator Required Support Files 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Multimedia Annotator 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Sample Schema Data 11.2.0.3.0
Oracle Starter Database 11.2.0.3.0
Oracle Message Gateway Common Files 11.2.0.3.0
Oracle XML Query 11.2.0.3.0
XML Parser for Oracle JVM 11.2.0.3.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.3.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.3.0
Oracle Containers for Java 11.2.0.3.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.3.0
Secure Socket Layer 11.2.0.3.0
Oracle Universal Connection Pool 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Precompiler Common Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
PL/SQL 11.2.0.3.0
Generic Connectivity Common Files 11.2.0.3.0
Oracle Database Gateway for ODBC 11.2.0.3.0
Oracle Programmer 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Call Interface (OCI) 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle XML Development Kit 11.2.0.3.0
Database Configuration and Upgrade Assistants 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Advanced Security 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Enterprise Manager Console DB 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Oracle Text 11.2.0.3.0
Oracle Net Services 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
Oracle OLAP 11.2.0.3.0
Oracle Spatial 11.2.0.3.0
Oracle Partitioning 11.2.0.3.0
Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Sunday, January 25, 2015 7:13:56 PM IST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, January 25, 2015 7:14:04 PM IST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, January 25, 2015 7:30:04 PM IST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/db3/root.sh #On nodes 10gnode3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db3 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@10gnode1 bin]$

Its time for now to run the root.sh script from the DB home on the newly added node -10gnode3 as ROOT user.

 


[root@10gnode3 ~]# /u01/app/oracle/product/11.2.0/db3/root.sh
Check /u01/app/oracle/product/11.2.0/db3/install/root_10gnode3.mydomain_2015-01-25_19-37-11.log for the output of root script
[root@10gnode3 ~]#

Here is the outcome of the ROOT.SH script that was run on 10gnode3.

 


[root@10gnode3 ~]# cat /u01/app/oracle/product/11.2.0/db3/install/root_10gnode3.mydomain_2015-01-25_19-37-11.log

Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/db3
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@10gnode3 ~]#

We now need to add a new instance “primt3” running from 10gnode3 to the database “primt”. But before we do that, we need to have few of the cluster parameters modified according to the instance.

 

Eg:
instance_number
Thread
Online Redo log groups for the new thread
UNDO_TABLESPACE

 


[oracle@10gnode1 ~]$ . oraenv
ORACLE_SID = [oracle] ? primt1
The Oracle base has been set to /u01/app/oracle
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 26 19:38:41 2015

Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select group#,thread#,bytes/1024/1024,members from v$log;

GROUP# THREAD# BYTES/1024/1024 MEMBERS
------ ------- --------------- ----------
1      1       100             2
2      1       100             2
3      2       100             2
4      2       100             2



GROUP# MEMBER
------ --------------------------------------------------
1      +DATA/primt/onlinelog/group_1.281.854648695
1      +FRA/primt/onlinelog/group_1.269.854648697
2      +DATA/primt/onlinelog/group_2.282.854648701
2      +FRA/primt/onlinelog/group_2.270.854648703
3      +DATA/primt/onlinelog/group_3.289.854738095
3      +FRA/primt/onlinelog/group_3.271.854738099
4      +DATA/primt/onlinelog/group_4.290.854738101
4      +FRA/primt/onlinelog/group_4.272.854738107

8 rows selected.

SQL> alter database add logfile thread 3 group 5 size 100M;

Database altered.

SQL> alter database add logfile thread 3 group 6 size 100M;

Database altered.

SQL> select group#,thread#,bytes/1024/1024,members from v$log;

GROUP# THREAD# BYTES/1024/1024 MEMBERS
------ ------- --------------- -------------
1      1       100             2
2      1       100             2
3      2       100             2
4      2       100             2
5      3       100             2
6      3       100             2

6 rows selected.

Log Files have been added to the new thread 3 by creating new Groups 5 and 6.
Now, let’s move on the undo_tablespace parameter. We shall create a new undo tablespace called “UNDOTBS3” and assign it as a default to instance 3.

 

SQL> select inst_id,name,value from gv$parameter where name like 'undo_tablespa%';

INST_ID NAME                      VALUE
------- ------------------------- -------------------------
1       undo_tablespace           UNDOTBS1
2       undo_tablespace           UNDOTBS2

SQL> create undo tablespace UNDOTBS3 datafile size 400M;

Tablespace created.
SQL> alter system set undo_tablespace=UNDOTBS3 scope=spfile sid='primt3';

System altered.

Assign an instance number to the instance that would be running from newly added node.

 

SQL> ALTER SYSTEM SET instance_number=3 SCOPE=SPFILE SID='primt3';

System altered.

Thread number for the instance that would be added newly is assigned as 3.

 

SQL> alter system set thread=3 scope=spfile sid='primt3';

System altered.

Shutdown the database (including all the instances)

 

[oracle@10gnode1 ~]$ srvctl stop database -d primt
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$ srvctl status database -d primt
Instance primt1 is not running on node 10gnode1
Instance primt2 is not running on node 10gnode2

Current DB configuration is as follows:

 

[oracle@10gnode1 ~]$ srvctl config database -d primt
Database unique name: primt
Database name:
Oracle home: /u01/app/oracle/product/11.2.0/db3
Oracle user: oracle
Spfile:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: primt
Database instances: primt1,primt2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed

Now its time to add the instance that will be running on the newly added node 10gnode3.

 

[oracle@10gnode1 ~]$ srvctl add instance -d primt -i primt3 -n 10gnode3
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$ srvctl status database -d primt -v -f
Instance primt1 is not running on node 10gnode1
Instance primt2 is not running on node 10gnode2
Instance primt3 is not running on node 10gnode3

Copy the PFILE from one of the running nodes to 10gnode3. The pfile would contain only the location of the SPFILE which is hosted on ASM.

 

[oracle@10gnode1 ~]$ cd $ORACLE_HOME/dbs
[oracle@10gnode1 dbs]$ ls -lrt initprimt1*
-rw-r--r-- 1 oracle oinstall 38 Aug 19 2014 initprimt1.ora
[oracle@10gnode1 dbs]$
[oracle@10gnode1 dbs]$
[oracle@10gnode1 dbs]$ cat initprimt1.ora
SPFILE='+data/primt/spfileprimt.ora'


[oracle@10gnode1 dbs]$
[oracle@10gnode1 dbs]$ scp initprimt1.ora oracle@10gnode3:/u01/app/oracle/product/11.2.0/db3/dbs/initprimt3.ora
initprimt1.ora 100% 38 0.0KB/s 00:00
[oracle@10gnode1 dbs]$
[oracle@10gnode3 ~]$ cd $ORACLE_HOME/dbs
[oracle@10gnode3 dbs]$ ls -lrt init*
-rw-r--r-- 1 oracle oinstall 2851 Jan 25 19:25 init.ora
-rw-r--r-- 1 oracle oinstall 38 Feb 24 20:02 initprimt3.ora

[/sourcode]

Start the Database using SRVCTL:


[oracle@10gnode1 ~]$ srvctl start database -d primt
PRCR-1079 : Failed to start resource ora.primt.db
CRS-5017: The resource action "ora.primt.db start" encountered the following error:
ORA-01618: redo thread 3 is not enabled - cannot mount
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid3/log/10gnode3/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.primt.db' on '10gnode3' failed
CRS-2632: There are no more servers to try to place resource 'ora.primt.db' on that would satisfy its placement policy

[oracle@10gnode1 ~]$ srvctl status database -d primt -v -f
Instance primt1 is running on node 10gnode1. Instance status: Open.
Instance primt2 is running on node 10gnode2. Instance status: Open.
Instance primt3 is not running on node 10gnode3

From the above error, it could be noticed that THREAD 3 was not enabled and due to which the instance primt3 wasn’t started.
From any one of the running instance, enable THREAD 3 as shown below.

 

[oracle@10gnode1 ~]$ echo $ORACLE_SID
primt1
[oracle@10gnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 26 21:02:31 2015

Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> alter database enable public thread 3;

Database altered.

Now, lets’ start the instance primt3 on 10gnode3.

 


[oracle@10gnode1 ~]$ srvctl start instance -i primt3 -d primt
[oracle@10gnode1 ~]$
[oracle@10gnode1 ~]$ srvctl status database -d primt -v -f
Instance primt1 is running on node 10gnode1. Instance status: Open.
Instance primt2 is running on node 10gnode2. Instance status: Open.
Instance primt3 is running on node 10gnode3. Instance status: Open.

Finally add the local_listener value to the instance “primt3” based on the listener configuration running on the new node 10gnode3.
Here we go !!

 

 

COPYRIGHT

© Shivananda Rao P, 2012, 2013, 2014, 2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: