Node Addition in RAC 11gR2
This article demonstrates on adding a node to the RAC cluster.
Environment:
OS: OEL Linux 5.5
Existing Nodes : 10gnode1, 10gnode2
Node to be added: 10gnode3
Database Name: PRIMT
DB Version: 11.2.0.3
Here is the details of Public, Private and Virtual IP addresses of the all the nodes in the cluster environment. This is same across all the 3 nodes.
[oracle@10gnode1 ~]$ cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 # PUBLIC 192.168.2.101 10gnode1.mydomain 10gnode1 192.168.2.102 10gnode2.mydomain 10gnode2 192.168.2.103 10gnode3.mydomain 10gnode3 # PRIVATE 192.168.0.101 10gnode1-priv.mydomain 10gnode1-priv 192.168.0.102 10gnode2-priv.mydomain 10gnode2-priv 192.168.0.103 10gnode3-priv.mydomain 10gnode3-priv # VIRTUAL 192.168.2.111 10gnode1-vip.mydomain 10gnode1-vip 192.168.2.112 10gnode2-vip.mydomain 10gnode2-vip 192.168.2.113 10gnode3-vip.mydomain 10gnode3-vip # SCAN 192.168.2.203 node-scan.mydomain node-scan 192.168.2.204 node-scan.mydomain node-scan 192.168.2.205 node-scan.mydomain node-scan
Next step would be to verify and establish passwordless SSH between the running nodes and the new node.
[oracle@10gnode1 ~]$ ssh 10gnode3 date The authenticity of host '10gnode3 (192.168.2.103)' can't be established. RSA key fingerprint is d2:e4:b0:df:11:11:cb:1d:16:15:25:79:f7:be:96:04. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10gnode3,192.168.2.103' (RSA) to the list of known hosts. Sat Jan 24 17:43:37 IST 2015 [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ ssh 10gnode3 hostname 10gnode3.mydomain [oracle@10gnode2 ~]$ ssh 10gnode3 date The authenticity of host '10gnode3 (192.168.2.103)' can't be established. RSA key fingerprint is d2:e4:b0:df:11:11:cb:1d:16:15:25:79:f7:be:96:04. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10gnode3,192.168.2.103' (RSA) to the list of known hosts. Sat Jan 24 17:44:01 IST 2015 [oracle@10gnode2 ~]$ [oracle@10gnode2 ~]$ ssh 10gnode3 hostname 10gnode3.mydomain [oracle@10gnode3 ~]$ ssh 10gnode1 date Sat Jan 24 17:44:17 IST 2015 [oracle@10gnode3 ~]$ [oracle@10gnode3 ~]$ ssh 10gnode1 hostname 10gnode1.mydomain [oracle@10gnode3 ~]$ [oracle@10gnode3 ~]$ ssh 10gnode2 date Sat Jan 24 17:44:27 IST 2015 [oracle@10gnode3 ~]$ [oracle@10gnode3 ~]$ ssh 10gnode2 hostname 10gnode2.mydomain [oracle@10gnode3 ~]$
Using CLUVFY, let’s check the pre-requisite on the new node. This is done by running the following the command on the running node.
cluvfy stage -pre nodeadd -n <new-node-to-be-added> -verbose
[oracle@10gnode1 ~]$ cd $ORA_CRS_HOME/bin [oracle@10gnode1 bin]$ pwd /u01/app/11.2.0/grid3/bin [oracle@10gnode1 bin]$ [oracle@10gnode1 bin]$ cluvfy stage -pre nodeadd -n 10gnode3 -verbose [oracle@10gnode1 bin]$ ./cluvfy stage -pre nodeadd -n 10gnode3 -verbose Performing pre-checks for node addition Checking node reachability... Check: Node reachability from node "10gnode1" Destination Node Reachable? ------------------------------------ ------------------------ 10gnode3 yes Result: Node reachability check passed from node "10gnode1" Checking user equivalence... Check: User equivalence for user "oracle" Node Name Status ------------------------------------ ------------------------ 10gnode3 passed Result: User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ 10gnode1 passed 10gnode2 passed 10gnode3 passed Verification of the hosts config file successful Interface information for node "10gnode1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth0 192.168.2.203 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth0 192.168.2.111 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500 eth1 169.254.125.10 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500 eth2 192.168.75.103 192.168.75.0 0.0.0.0 UNKNOWN 08:00:27:73:78:FF 1500 Interface information for node "10gnode2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500 eth0 192.168.2.112 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500 eth1 169.254.60.182 169.254.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500 eth2 192.168.75.102 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:76:C0:28 1500 Interface information for node "10gnode3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.103 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:53:19:C6 1500 eth1 192.168.0.103 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:BA:22:46 1500 eth2 192.168.75.104 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:22:DE:35 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1[192.168.2.101] 10gnode1[192.168.2.203] yes 10gnode1[192.168.2.101] 10gnode1[192.168.2.111] yes 10gnode1[192.168.2.101] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.101] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.101] 10gnode3[192.168.2.103] yes 10gnode1[192.168.2.203] 10gnode1[192.168.2.111] yes 10gnode1[192.168.2.203] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.203] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.203] 10gnode3[192.168.2.103] yes 10gnode1[192.168.2.111] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.111] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.111] 10gnode3[192.168.2.103] yes 10gnode2[192.168.2.102] 10gnode2[192.168.2.112] yes 10gnode2[192.168.2.102] 10gnode3[192.168.2.103] yes 10gnode2[192.168.2.112] 10gnode3[192.168.2.103] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1:192.168.2.101 10gnode1:192.168.2.203 passed 10gnode1:192.168.2.101 10gnode1:192.168.2.111 passed 10gnode1:192.168.2.101 10gnode2:192.168.2.102 passed 10gnode1:192.168.2.101 10gnode2:192.168.2.112 passed 10gnode1:192.168.2.101 10gnode3:192.168.2.103 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Checking CRS integrity... Clusterware version consistency passed The Oracle Clusterware is healthy on node "10gnode1" The Oracle Clusterware is healthy on node "10gnode2" CRS integrity check passed Checking shared resources... Checking CRS home location... PRVG-1013 : The path "/u01/app/11.2.0/grid3" does not exist or cannot be created on the nodes to be added Result: Shared resources check for node addition failed Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ 10gnode1 passed 10gnode2 passed 10gnode3 passed Verification of the hosts config file successful Interface information for node "10gnode1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth0 192.168.2.203 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth0 192.168.2.111 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:AF:22:42 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500 eth1 169.254.125.10 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:0E:E3:05 1500 eth2 192.168.75.103 192.168.75.0 0.0.0.0 UNKNOWN 08:00:27:73:78:FF 1500 Interface information for node "10gnode2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500 eth0 192.168.2.112 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:8D:58:4D 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500 eth1 169.254.60.182 169.254.0.0 0.0.0.0 192.168.2.1 08:00:27:32:B8:15 1500 eth2 192.168.75.102 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:76:C0:28 1500 Interface information for node "10gnode3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.103 192.168.2.0 0.0.0.0 192.168.2.1 08:00:27:53:19:C6 1500 eth1 192.168.0.103 192.168.0.0 0.0.0.0 192.168.2.1 08:00:27:BA:22:46 1500 eth2 192.168.75.104 192.168.75.0 0.0.0.0 192.168.2.1 08:00:27:22:DE:35 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1[192.168.2.101] 10gnode1[192.168.2.203] yes 10gnode1[192.168.2.101] 10gnode1[192.168.2.111] yes 10gnode1[192.168.2.101] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.101] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.101] 10gnode3[192.168.2.103] yes 10gnode1[192.168.2.203] 10gnode1[192.168.2.111] yes 10gnode1[192.168.2.203] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.203] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.203] 10gnode3[192.168.2.103] yes 10gnode1[192.168.2.111] 10gnode2[192.168.2.102] yes 10gnode1[192.168.2.111] 10gnode2[192.168.2.112] yes 10gnode1[192.168.2.111] 10gnode3[192.168.2.103] yes 10gnode2[192.168.2.102] 10gnode2[192.168.2.112] yes 10gnode2[192.168.2.102] 10gnode3[192.168.2.103] yes 10gnode2[192.168.2.112] 10gnode3[192.168.2.103] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1:192.168.2.101 10gnode1:192.168.2.203 passed 10gnode1:192.168.2.101 10gnode1:192.168.2.111 passed 10gnode1:192.168.2.101 10gnode2:192.168.2.102 passed 10gnode1:192.168.2.101 10gnode2:192.168.2.112 passed 10gnode1:192.168.2.101 10gnode3:192.168.2.103 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Check: Node connectivity for interface "eth1" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1[192.168.0.101] 10gnode2[192.168.0.102] yes 10gnode1[192.168.0.101] 10gnode3[192.168.0.103] yes 10gnode2[192.168.0.102] 10gnode3[192.168.0.103] yes Result: Node connectivity passed for interface "eth1" Check: TCP connectivity of subnet "192.168.0.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- 10gnode1:192.168.0.101 10gnode2:192.168.0.102 passed 10gnode1:192.168.0.101 10gnode3:192.168.0.103 passed Result: TCP connectivity check passed for subnet "192.168.0.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed for subnet "192.168.0.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Check: Total memory Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 2.9718GB (3116120.0KB) 1.5GB (1572864.0KB) passed 10gnode1 2.9718GB (3116120.0KB) 1.5GB (1572864.0KB) passed Result: Total memory check passed Check: Available memory Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 2.9093GB (3050644.0KB) 50MB (51200.0KB) passed 10gnode1 1.9784GB (2074528.0KB) 50MB (51200.0KB) passed Result: Available memory check passed Check: Swap space Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 9.767GB (1.0241396E7KB) 2.9718GB (3116120.0KB) passed 10gnode1 9.767GB (1.0241396E7KB) 2.9718GB (3116120.0KB) passed Result: Swap space check passed Check: Free disk space for "10gnode3:/u01/app/11.2.0/grid3" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /u01/app/11.2.0/grid3 10gnode3 /u01 18.6406GB 5.5GB passed Result: Free disk space check passed for "10gnode3:/u01/app/11.2.0/grid3" Check: Free disk space for "10gnode1:/u01/app/11.2.0/grid3" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /u01/app/11.2.0/grid3 10gnode1 /u01 9.3252GB 5.5GB passed Result: Free disk space check passed for "10gnode1:/u01/app/11.2.0/grid3" Check: Free disk space for "10gnode3:/tmp" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /tmp 10gnode3 / 26.9141GB 1GB passed Result: Free disk space check passed for "10gnode3:/tmp" Check: Free disk space for "10gnode1:/tmp" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /tmp 10gnode1 / 26.917GB 1GB passed Result: Free disk space check passed for "10gnode1:/tmp" Check: User existence for "oracle" Node Name Status Comment ------------ ------------------------ ------------------------ 10gnode3 passed exists(500) 10gnode1 passed exists(500) Checking for multiple users with UID value 500 Result: Check for multiple users with UID value 500 passed Result: User existence check passed for "oracle" Check: Run level Node Name run level Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 5 3,5 passed 10gnode1 5 3,5 passed Result: Run level check passed Check: Hard limits for "maximum open file descriptors" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- 10gnode1 hard 65536 65536 passed 10gnode3 hard 65536 65536 passed Result: Hard limits check passed for "maximum open file descriptors" Check: Soft limits for "maximum open file descriptors" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- 10gnode1 soft 1024 1024 passed 10gnode3 soft 1024 1024 passed Result: Soft limits check passed for "maximum open file descriptors" Check: Hard limits for "maximum user processes" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- 10gnode1 hard 16384 16384 passed 10gnode3 hard 16384 16384 passed Result: Hard limits check passed for "maximum user processes" Check: Soft limits for "maximum user processes" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- 10gnode1 soft 2047 2047 passed 10gnode3 soft 2047 2047 passed Result: Soft limits check passed for "maximum user processes" Check: System architecture Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 i686 x86 passed 10gnode1 i686 x86 passed Result: System architecture check passed Check: Kernel version Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 2.6.18-8.el5 2.6.18 passed 10gnode1 2.6.18-8.el5 2.6.18 passed Result: Kernel version check passed Check: Kernel parameter for "semmsl" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 250 250 250 passed 10gnode3 250 250 250 passed Result: Kernel parameter check passed for "semmsl" Check: Kernel parameter for "semmns" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 32000 32000 32000 passed 10gnode3 32000 32000 32000 passed Result: Kernel parameter check passed for "semmns" Check: Kernel parameter for "semopm" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 100 100 100 passed 10gnode3 100 100 100 passed Result: Kernel parameter check passed for "semopm" Check: Kernel parameter for "semmni" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 128 128 128 passed 10gnode3 128 128 128 passed Result: Kernel parameter check passed for "semmni" Check: Kernel parameter for "shmmax" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 2147483648 2147483648 1595453440 passed 10gnode3 2147483648 2147483648 1595453440 passed Result: Kernel parameter check passed for "shmmax" Check: Kernel parameter for "shmmni" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 4096 4096 4096 passed 10gnode3 4096 4096 4096 passed Result: Kernel parameter check passed for "shmmni" Check: Kernel parameter for "shmall" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 2097152 2097152 2097152 passed 10gnode3 2097152 2097152 2097152 passed Result: Kernel parameter check passed for "shmall" Check: Kernel parameter for "file-max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 6815744 6815744 6815744 passed 10gnode3 6815744 6815744 6815744 passed Result: Kernel parameter check passed for "file-max" Check: Kernel parameter for "ip_local_port_range" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed 10gnode3 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed Result: Kernel parameter check passed for "ip_local_port_range" Check: Kernel parameter for "rmem_default" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 1048576 1048576 262144 passed 10gnode3 1048576 1048576 262144 passed Result: Kernel parameter check passed for "rmem_default" Check: Kernel parameter for "rmem_max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 4194304 4194304 4194304 passed 10gnode3 4194304 4194304 4194304 passed Result: Kernel parameter check passed for "rmem_max" Check: Kernel parameter for "wmem_default" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 262144 262144 262144 passed 10gnode3 262144 262144 262144 passed Result: Kernel parameter check passed for "wmem_default" Check: Kernel parameter for "wmem_max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 1048576 1048576 1048576 passed 10gnode3 1048576 1048576 1048576 passed Result: Kernel parameter check passed for "wmem_max" Check: Kernel parameter for "aio-max-nr" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ 10gnode1 1048576 1048576 1048576 passed 10gnode3 1048576 1048576 1048576 passed Result: Kernel parameter check passed for "aio-max-nr" Check: Package existence for "ocfs2-tools" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 ocfs2-tools-1.2.7-1.el5 ocfs2-tools-1.2.7 passed 10gnode1 ocfs2-tools-1.2.7-1.el5 ocfs2-tools-1.2.7 passed Result: Package existence check passed for "ocfs2-tools" Check: Package existence for "make" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 make-3.81-1.1 make-3.81 passed 10gnode1 make-3.81-1.1 make-3.81 passed Result: Package existence check passed for "make" Check: Package existence for "binutils" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 binutils-2.17.50.0.6-2.el5 binutils-2.17.50.0.6 passed 10gnode1 binutils-2.17.50.0.6-2.el5 binutils-2.17.50.0.6 passed Result: Package existence check passed for "binutils" Check: Package existence for "gcc" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 gcc-4.1.2-48.el5 gcc-4.1.2 passed 10gnode1 gcc-4.1.2-48.el5 gcc-4.1.2 passed Result: Package existence check passed for "gcc" Check: Package existence for "gcc-c++" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 gcc-c++-4.1.2-48.el5 gcc-c++-4.1.2 passed 10gnode1 gcc-c++-4.1.2-48.el5 gcc-c++-4.1.2 passed Result: Package existence check passed for "gcc-c++" Check: Package existence for "libgomp" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libgomp-4.4.0-6.el5 libgomp-4.1.2 passed 10gnode1 libgomp-4.4.0-6.el5 libgomp-4.1.2 passed Result: Package existence check passed for "libgomp" Check: Package existence for "libaio" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libaio-0.3.106-5 libaio-0.3.106 passed 10gnode1 libaio-0.3.106-5 libaio-0.3.106 passed Result: Package existence check passed for "libaio" Check: Package existence for "glibc" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 glibc-2.5-49 glibc-2.5-24 passed 10gnode1 glibc-2.5-49 glibc-2.5-24 passed Result: Package existence check passed for "glibc" Check: Package existence for "compat-libstdc++-33" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed 10gnode1 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed Result: Package existence check passed for "compat-libstdc++-33" Check: Package existence for "elfutils-libelf" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed 10gnode1 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed Result: Package existence check passed for "elfutils-libelf" Check: Package existence for "elfutils-libelf-devel" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed 10gnode1 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed Result: Package existence check passed for "elfutils-libelf-devel" Check: Package existence for "glibc-common" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 glibc-common-2.5-49 glibc-common-2.5 passed 10gnode1 glibc-common-2.5-49 glibc-common-2.5 passed Result: Package existence check passed for "glibc-common" Check: Package existence for "glibc-devel" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 glibc-devel-2.5-12 glibc-devel-2.5 passed 10gnode1 glibc-devel-2.5-12 glibc-devel-2.5 passed Result: Package existence check passed for "glibc-devel" Check: Package existence for "glibc-headers" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 glibc-headers-2.5-12 glibc-headers-2.5 passed 10gnode1 glibc-headers-2.5-12 glibc-headers-2.5 passed Result: Package existence check passed for "glibc-headers" Check: Package existence for "libaio-devel" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libaio-devel-0.3.106-5 libaio-devel-0.3.106 passed 10gnode1 libaio-devel-0.3.106-5 libaio-devel-0.3.106 passed Result: Package existence check passed for "libaio-devel" Check: Package existence for "libgcc" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libgcc-4.1.2-48.el5 libgcc-4.1.2 passed 10gnode1 libgcc-4.1.2-48.el5 libgcc-4.1.2 passed Result: Package existence check passed for "libgcc" Check: Package existence for "libstdc++" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libstdc++-4.1.2-48.el5 libstdc++-4.1.2 passed 10gnode1 libstdc++-4.1.2-48.el5 libstdc++-4.1.2 passed Result: Package existence check passed for "libstdc++" Check: Package existence for "libstdc++-devel" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 libstdc++-devel-4.1.2-48.el5 libstdc++-devel-4.1.2 passed 10gnode1 libstdc++-devel-4.1.2-48.el5 libstdc++-devel-4.1.2 passed Result: Package existence check passed for "libstdc++-devel" Check: Package existence for "sysstat" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 sysstat-7.0.2-3.el5 sysstat-7.0.2 passed 10gnode1 sysstat-7.0.2-3.el5 sysstat-7.0.2 passed Result: Package existence check passed for "sysstat" Check: Package existence for "ksh" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- 10gnode3 ksh-20060214-1.4 ksh-20060214 passed 10gnode1 ksh-20060214-1.4 ksh-20060214 passed Result: Package existence check passed for "ksh" Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed Check: Current group ID Result: Current group ID check passed Starting check for consistency of primary group of root user Node Name Status ------------------------------------ ------------------------ 10gnode3 passed 10gnode1 passed Check for consistency of root user's primary group passed Checking OCR integrity... OCR integrity check passed Checking Oracle Cluster Voting Disk configuration... Oracle Cluster Voting Disk configuration check passed Check: Time zone consistency Result: Time zone consistency check passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... The NTP configuration file "/etc/ntp.conf" is available on all nodes NTP Configuration file check passed No NTP Daemons or Services were found to be running PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s): 10gnode3,10gnode1 Result: Clock synchronization check using Network Time Protocol(NTP) failed Checking to make sure user "oracle" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ 10gnode3 passed does not exist 10gnode1 passed does not exist Result: User "oracle" is not part of "root" group. Check passed Checking consistency of file "/etc/resolv.conf" across nodes Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined File "/etc/resolv.conf" does not have both domain and search entries defined Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes... domain entry in file "/etc/resolv.conf" is consistent across nodes Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes... PRVF-5622 : search entry does not exist in file "/etc/resolv.conf" on nodes: "10gnode3" Checking file "/etc/resolv.conf" to make sure that only one search entry is defined All nodes have one search entry defined in file "/etc/resolv.conf" Checking DNS response time for an unreachable node Node Name Status ------------------------------------ ------------------------ 10gnode1 failed 10gnode3 failed PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: 10gnode1,10gnode3 File "/etc/resolv.conf" is not consistent across nodes Pre-check for node addition was unsuccessful on all the nodes.
From the above output, I see that all checks have been passed except the “DNS response time”. I’m ignoring this in my case as I haven’t configured DNS and not using this.
Now, let’s add the new node using the “addNode.sh” using “silent” method. The parameters passed here are “CLUSTER_NEW_NODES” which is the new node that needs to be added and “CLUSTER_NEW_VIRTUAL_HOSTNAMES” which is the VIP alias of the new node to be added.
addNode.sh is located at “GRID_HOME/oui/bin” and is run from one of the running nodes. I’m running it here from 10gnode1.
[oracle@10gnode1 ~]$ cd /u01/app/11.2.0/grid3/oui/bin
[oracle@10gnode1 bin]$
[oracle@10gnode1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={10gnode3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={10gnode3-vip}" -silent
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes 10gnode2,10gnode3 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/11.2.0/grid3
New Nodes
Space Requirements
New Nodes
10gnode3
/u01: Required 5.16GB : Available 17.36GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.4
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Server) 11.2.0.3.0
Installation Plugin Files 11.2.0.3.0
Universal Storage Manager Files 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Automatic Storage Management Assistant 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.3.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Oracle Net Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Enterprise Manager plugin Common Files 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.3.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Cluster Verification Utility Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle LDAP administration 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
PL/SQL 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Cluster Ready Services Files 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Saturday, January 24, 2015 7:13:26 PM IST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Saturday, January 24, 2015 7:13:29 PM IST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Saturday, January 24, 2015 7:22:28 PM IST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/11.2.0/grid3/root.sh #On nodes 10gnode3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/11.2.0/grid3 was successful.
Please check '/tmp/silentInstall.log' for more details.
We need to run root.sh script on the new node (10gnode3) as ROOT user.
[root@10gnode3 ~]# /u01/app/11.2.0/grid3/root.sh Check /u01/app/11.2.0/grid3/install/root_10gnode3.mydomain_2015-01-24_19-25-31.log for the output of root script [root@10gnode3 ~]#
Let’s check the status of the resources on the newly added node.
[root@10gnode3 bin]# ./crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.asm ONLINE ONLINE 10gnode1 Started ONLINE ONLINE 10gnode2 Started ONLINE ONLINE 10gnode3 Started ora.gsd OFFLINE OFFLINE 10gnode1 OFFLINE OFFLINE 10gnode2 OFFLINE OFFLINE 10gnode3 ora.net1.network ONLINE ONLINE 10gnode1 ONLINE ONLINE 10gnode2 ONLINE ONLINE 10gnode3 ora.ons ONLINE ONLINE 10gnode1 ONLINE ONLINE 10gnode2 ONLINE ONLINE 10gnode3 ora.registry.acfs ONLINE ONLINE 10gnode1 ONLINE ONLINE 10gnode2 ONLINE ONLINE 10gnode3 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.10gnode1.vip 1 ONLINE ONLINE 10gnode1 ora.10gnode2.vip 1 ONLINE ONLINE 10gnode2 ora.10gnode3.vip 1 ONLINE ONLINE 10gnode3 ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 10gnode2 ora.cvu 1 ONLINE ONLINE 10gnode3 ora.oc4j 1 ONLINE ONLINE 10gnode2 ora.primt.db 1 ONLINE ONLINE 10gnode1 Open 2 ONLINE ONLINE 10gnode2 Open ora.scan1.vip 1 ONLINE ONLINE 10gnode2
Run the CLUVFY utility to verify everything is fine after adding the new node to the cluster.
[oracle@10gnode1 ~]$ cd /u01/app/11.2.0/grid3/bin [oracle@10gnode1 bin]$ [oracle@10gnode1 bin]$ ./cluvfy stage -post nodeadd -n 10gnode3
============================DB=============================
Now it’s time to add the DB Home installed on the new node, in other words we need to add the new node needs to be added to the DB cluster.
[oracle@10gnode1 ~]$ cd /u01/app/oracle/product/11.2.0/db3/oui/bin/
[oracle@10gnode1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={10gnode3}" -silent
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "10gnode1"
Checking user equivalence...
User equivalence check passed for user "oracle"
WARNING:
Node "10gnode3" already appears to be part of cluster
Pre-check for node addition was successful.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes 10gnode2,10gnode3 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/oracle/product/11.2.0/db3
New Nodes
Space Requirements
New Nodes
10gnode3
/u01: Required 4.14GB : Available 13.02GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.5.0.1
Oracle ODBC Driverfor Instant Client 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
SSL Required Support Files for InstantClient 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.3.0
Oracle Database Vault J2EE Application 11.2.0.3.0
Oracle Label Security 11.2.0.3.0
Oracle Data Mining RDBMS Files 11.2.0.3.0
Oracle OLAP RDBMS Files 11.2.0.3.0
Oracle OLAP API 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle Database Vault option 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.3.0
Oracle SQL Developer 11.2.0.3.0
Oracle Application Express 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
SQLJ Runtime 11.2.0.3.0
Database Workspace Manager 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
Enterprise Manager Repository Core Files 10.2.0.4.4
Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
Enterprise Manager Grid Control Core Files 10.2.0.4.4
Enterprise Manager Common Core Files 10.2.0.4.4
Enterprise Manager Agent Core Files 10.2.0.4.4
RDBMS Required Support Files 11.2.0.3.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Parser Generator Required Support Files 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Multimedia Annotator 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Sample Schema Data 11.2.0.3.0
Oracle Starter Database 11.2.0.3.0
Oracle Message Gateway Common Files 11.2.0.3.0
Oracle XML Query 11.2.0.3.0
XML Parser for Oracle JVM 11.2.0.3.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.3.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.3.0
Oracle Containers for Java 11.2.0.3.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.3.0
Secure Socket Layer 11.2.0.3.0
Oracle Universal Connection Pool 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Precompiler Common Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
PL/SQL 11.2.0.3.0
Generic Connectivity Common Files 11.2.0.3.0
Oracle Database Gateway for ODBC 11.2.0.3.0
Oracle Programmer 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Call Interface (OCI) 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle XML Development Kit 11.2.0.3.0
Database Configuration and Upgrade Assistants 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Advanced Security 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Enterprise Manager Console DB 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Oracle Text 11.2.0.3.0
Oracle Net Services 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
Oracle OLAP 11.2.0.3.0
Oracle Spatial 11.2.0.3.0
Oracle Partitioning 11.2.0.3.0
Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Sunday, January 25, 2015 7:13:56 PM IST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Sunday, January 25, 2015 7:14:04 PM IST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Sunday, January 25, 2015 7:30:04 PM IST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/db3/root.sh #On nodes 10gnode3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db3 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@10gnode1 bin]$
Its time for now to run the root.sh script from the DB home on the newly added node -10gnode3 as ROOT user.
[root@10gnode3 ~]# /u01/app/oracle/product/11.2.0/db3/root.sh Check /u01/app/oracle/product/11.2.0/db3/install/root_10gnode3.mydomain_2015-01-25_19-37-11.log for the output of root script [root@10gnode3 ~]#
Here is the outcome of the ROOT.SH script that was run on 10gnode3.
[root@10gnode3 ~]# cat /u01/app/oracle/product/11.2.0/db3/install/root_10gnode3.mydomain_2015-01-25_19-37-11.log Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/db3 Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Finished product-specific root actions. [root@10gnode3 ~]#
We now need to add a new instance “primt3” running from 10gnode3 to the database “primt”. But before we do that, we need to have few of the cluster parameters modified according to the instance.
Eg:
instance_number
Thread
Online Redo log groups for the new thread
UNDO_TABLESPACE
[oracle@10gnode1 ~]$ . oraenv ORACLE_SID = [oracle] ? primt1 The Oracle base has been set to /u01/app/oracle [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 26 19:38:41 2015 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> select group#,thread#,bytes/1024/1024,members from v$log; GROUP# THREAD# BYTES/1024/1024 MEMBERS ------ ------- --------------- ---------- 1 1 100 2 2 1 100 2 3 2 100 2 4 2 100 2 GROUP# MEMBER ------ -------------------------------------------------- 1 +DATA/primt/onlinelog/group_1.281.854648695 1 +FRA/primt/onlinelog/group_1.269.854648697 2 +DATA/primt/onlinelog/group_2.282.854648701 2 +FRA/primt/onlinelog/group_2.270.854648703 3 +DATA/primt/onlinelog/group_3.289.854738095 3 +FRA/primt/onlinelog/group_3.271.854738099 4 +DATA/primt/onlinelog/group_4.290.854738101 4 +FRA/primt/onlinelog/group_4.272.854738107 8 rows selected. SQL> alter database add logfile thread 3 group 5 size 100M; Database altered. SQL> alter database add logfile thread 3 group 6 size 100M; Database altered. SQL> select group#,thread#,bytes/1024/1024,members from v$log; GROUP# THREAD# BYTES/1024/1024 MEMBERS ------ ------- --------------- ------------- 1 1 100 2 2 1 100 2 3 2 100 2 4 2 100 2 5 3 100 2 6 3 100 2 6 rows selected.
Log Files have been added to the new thread 3 by creating new Groups 5 and 6.
Now, let’s move on the undo_tablespace parameter. We shall create a new undo tablespace called “UNDOTBS3” and assign it as a default to instance 3.
SQL> select inst_id,name,value from gv$parameter where name like 'undo_tablespa%'; INST_ID NAME VALUE ------- ------------------------- ------------------------- 1 undo_tablespace UNDOTBS1 2 undo_tablespace UNDOTBS2 SQL> create undo tablespace UNDOTBS3 datafile size 400M; Tablespace created. SQL> alter system set undo_tablespace=UNDOTBS3 scope=spfile sid='primt3'; System altered.
Assign an instance number to the instance that would be running from newly added node.
SQL> ALTER SYSTEM SET instance_number=3 SCOPE=SPFILE SID='primt3'; System altered.
Thread number for the instance that would be added newly is assigned as 3.
SQL> alter system set thread=3 scope=spfile sid='primt3'; System altered.
Shutdown the database (including all the instances)
[oracle@10gnode1 ~]$ srvctl stop database -d primt [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ srvctl status database -d primt Instance primt1 is not running on node 10gnode1 Instance primt2 is not running on node 10gnode2
Current DB configuration is as follows:
[oracle@10gnode1 ~]$ srvctl config database -d primt Database unique name: primt Database name: Oracle home: /u01/app/oracle/product/11.2.0/db3 Oracle user: oracle Spfile: Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: primt Database instances: primt1,primt2 Disk Groups: Mount point paths: Services: Type: RAC Database is administrator managed
Now its time to add the instance that will be running on the newly added node 10gnode3.
[oracle@10gnode1 ~]$ srvctl add instance -d primt -i primt3 -n 10gnode3 [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ srvctl status database -d primt -v -f Instance primt1 is not running on node 10gnode1 Instance primt2 is not running on node 10gnode2 Instance primt3 is not running on node 10gnode3
Copy the PFILE from one of the running nodes to 10gnode3. The pfile would contain only the location of the SPFILE which is hosted on ASM.
[oracle@10gnode1 ~]$ cd $ORACLE_HOME/dbs [oracle@10gnode1 dbs]$ ls -lrt initprimt1* -rw-r--r-- 1 oracle oinstall 38 Aug 19 2014 initprimt1.ora [oracle@10gnode1 dbs]$ [oracle@10gnode1 dbs]$ [oracle@10gnode1 dbs]$ cat initprimt1.ora SPFILE='+data/primt/spfileprimt.ora'
[oracle@10gnode1 dbs]$ [oracle@10gnode1 dbs]$ scp initprimt1.ora oracle@10gnode3:/u01/app/oracle/product/11.2.0/db3/dbs/initprimt3.ora initprimt1.ora 100% 38 0.0KB/s 00:00 [oracle@10gnode1 dbs]$ [oracle@10gnode3 ~]$ cd $ORACLE_HOME/dbs [oracle@10gnode3 dbs]$ ls -lrt init* -rw-r--r-- 1 oracle oinstall 2851 Jan 25 19:25 init.ora -rw-r--r-- 1 oracle oinstall 38 Feb 24 20:02 initprimt3.ora [/sourcode] Start the Database using SRVCTL: [oracle@10gnode1 ~]$ srvctl start database -d primt PRCR-1079 : Failed to start resource ora.primt.db CRS-5017: The resource action "ora.primt.db start" encountered the following error: ORA-01618: redo thread 3 is not enabled - cannot mount . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid3/log/10gnode3/agent/crsd/oraagent_oracle/oraagent_oracle.log". CRS-2674: Start of 'ora.primt.db' on '10gnode3' failed CRS-2632: There are no more servers to try to place resource 'ora.primt.db' on that would satisfy its placement policy
[oracle@10gnode1 ~]$ srvctl status database -d primt -v -f Instance primt1 is running on node 10gnode1. Instance status: Open. Instance primt2 is running on node 10gnode2. Instance status: Open. Instance primt3 is not running on node 10gnode3
From the above error, it could be noticed that THREAD 3 was not enabled and due to which the instance primt3 wasn’t started.
From any one of the running instance, enable THREAD 3 as shown below.
[oracle@10gnode1 ~]$ echo $ORACLE_SID primt1 [oracle@10gnode1 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 26 21:02:31 2015 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> alter database enable public thread 3; Database altered.
Now, lets’ start the instance primt3 on 10gnode3.
[oracle@10gnode1 ~]$ srvctl start instance -i primt3 -d primt [oracle@10gnode1 ~]$ [oracle@10gnode1 ~]$ srvctl status database -d primt -v -f Instance primt1 is running on node 10gnode1. Instance status: Open. Instance primt2 is running on node 10gnode2. Instance status: Open. Instance primt3 is running on node 10gnode3. Instance status: Open.
Finally add the local_listener value to the instance “primt3” based on the listener configuration running on the new node 10gnode3.
Here we go !!
COPYRIGHT
© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.
DISCLAIMER
The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.
Shivananda Rao 
Leave a Reply