Shivananda Rao

We have come across OEM agent installation from the Cloud Control tool. This article demonstrates on how to install 12c cloud control agent uisng “silent” mode which makes use of the “response” file.

Environment details is as follows:

OMS hostname              : ora1-2
Agent will be deployed on : ora1-1

Below is the details of the OMS status. This is checked from the OMS server.

[oracle@ora1-2 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : ora1-2.mydomain
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://ora1-2.mydomain:7802/em
Upload URL: https://ora1-2.mydomain:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : ora1-2.mydomain
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: ora1-2.mydomain
WebTier is Up
Oracle Management Server is Up

BI Publisher is not configured to run on this host.
[oracle@ora1-2 bin]$

Login to the “Enterprise Manager Command Line Interface” client from the OMS host with username as SYSMAN and it’s password.

[oracle@ora1-2 bin]$ pwd
/u01/app/oracle/oms12c/oms/bin
[oracle@ora1-2 bin]$
[oracle@ora1-2 bin]$ ./emcli login -username=sysman -password=Micromot10n
Login successful

Once logged in successfully, synchronize the EMCLI.

[oracle@ora1-2 bin]$ ./emcli sync
Synchronized successfully
[oracle@ora1-2 bin]$

Identify the list of Operating Systems for which the Agent software is available. This can be obtained by running the “get_supported_platforms” from the EMCLI interface.

[oracle@ora1-2 bin]$ ./emcli get_supported_platforms
-----------------------------------------------
Version = 12.1.0.4.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

Get the Agent software for the required Operating system by specifying the “platform” option and the relevant agent version by passing “version” option.
The image will be saved on to the OMS host at the location specifed under the “destination” option. Here, I’m getting the AgentImage for “Linux x86-64” platform and of version “12.1.0.4.0” as stated above and saving the AgentImage to “/u03/agent_software” location on the OMS host.

[oracle@ora1-2 bin]$ ./emcli get_agentimage -destination=/u03/agent_software -platform="Linux x86-64" -version="12.1.0.4.0"
 === Partition Detail ===
Space free : 15 GB
Space required : 1 GB
Check the logs at /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2016-04-08_18-49-33-PM.log
Downloading /u03/agent_software/12.1.0.4.0_AgentCore_226.zip
File saved as /u03/agent_software/12.1.0.4.0_AgentCore_226.zip
Downloading /u03/agent_software/12.1.0.4.0_PluginsOneoffs_226.zip
File saved as /u03/agent_software/12.1.0.4.0_PluginsOneoffs_226.zip
Downloading /u03/agent_software/unzip
File saved as /u03/agent_software/unzip
Agent Image Download completed successfully.
[oracle@ora1-2 bin]$

Copy this downloaded AgentImage on to the target server where the Agent needs to be installed. In my case, it’s host ora1-1 where the agent will be installed.

[oracle@ora1-2 bin]$ cd /u03/agent_software
[oracle@ora1-2 agent_software]$ ls -lrt
total 250000
-rw-r--r--. 1 oracle oinstall 255744050 Apr  8 18:52 12.1.0.4.0_AgentCore_226.zip
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$ scp 12.1.0.4.0_AgentCore_226.zip oracle@ora1-1:/u03/
oracle@192.168.56.101's password:
12.1.0.4.0_AgentCore_226.zip                                                                                                         100%  244MB   9.0MB/s   00:27
[oracle@ora1-2 agent_software]$

[oracle@ora1-1 ~]$ cd /u03
[oracle@ora1-1 u03]$ ls -lrt
total 1196388
drwx------. 2 oracle oinstall     16384 Jan  1 17:25 lost+found
-rw-r--r--. 1 oracle oinstall 915338678 Mar 27 11:25 p19954978_121020_Linux-x86-64.zip
drwxr-xr-x. 3 oracle oinstall      4096 Mar 27 11:43 121022_gi_psu
-rw-r--r--. 1 oracle oinstall  52774823 Mar 27 13:38 p6880880_121010_Linux-x86-64.zip
-rw-r--r--. 1 oracle oinstall 255744050 Apr  8 19:11 12.1.0.4.0_AgentCore_226.zip
[oracle@ora1-1 u03]$

On the target server, unzip the AgentImage software received from the OMS host.

[oracle@ora1-1 u03]$
[oracle@ora1-1 u03]$ mkdir 12104_agent
[oracle@ora1-1 u03]$
[oracle@ora1-1 u03]$ unzip -d 12104_agent 12.1.0.4.0_AgentCore_226.zip
Archive:  12.1.0.4.0_AgentCore_226.zip
  inflating: 12104_agent/unzip
  inflating: 12104_agent/agentDeploy.sh
  inflating: 12104_agent/agentimage.properties
  inflating: 12104_agent/agent.rsp
 extracting: 12104_agent/agentcoreimage.zip
 extracting: 12104_agent/12.1.0.4.0_PluginsOneoffs_226.zip
[oracle@ora1-1 u03]$

Edit the “agent.rsp” (AgentResponseFile) to add the details such as OMS Hostname, EM Upload Port (the HTTPS port number of OMS), Agent Registration Password (used to secure the agent), Agent Instance Home location, Agent Port number (port where Agent process will be started),b_startAgent (True if the agent needs to be started automatically after installed and configured, ORACLE_HOSTNAME (host name of the target server where the agent will be installed), s_agentHomeName (name for the Agent Home).

Below are the details suiting my environment.

OMS_HOST=ora1-2.mydomain
EM_UPLOAD_PORT=4903
AGENT_REGISTRATION_PASSWORD=oracle123
AGENT_INSTANCE_HOME=
AGENT_PORT=3873
b_startAgent=true
ORACLE_HOSTNAME=ora1-1.mydomain
s_agentHomeName=agent12cr1

[oracle@ora1-1 12104_agent]$ cat agent.rsp
<output trimmed>

OMS_HOST=ora1-2.mydomain
EM_UPLOAD_PORT=4903
AGENT_REGISTRATION_PASSWORD=oracle123
AGENT_INSTANCE_HOME=
AGENT_PORT=3873
b_startAgent=true
ORACLE_HOSTNAME=ora1-1.mydomain
s_agentHomeName=agent12cr1


<output trimmed>

[oracle@ora1-1 oracle]$ cd /u03/12104_agent/
[oracle@ora1-1 12104_agent]$ ls -lrt
total 250112
-rw-rw-r--. 1 oracle oinstall       178 May 24  2014 agentimage.properties
-rwxr-xr-x. 1 oracle oinstall    145976 May 24  2014 unzip
-rwxrwxr-x. 1 oracle oinstall     27905 May 24  2014 agentDeploy.sh
-rw-rw-r--. 1 oracle oinstall 248704435 May 24  2014 agentcoreimage.zip
-rw-r--r--. 1 oracle oinstall   6958959 Apr  8 18:52 12.1.0.4.0_PluginsOneoffs_226.zip
-rwxrwxr-x. 1 oracle oinstall      3726 Apr  8 19:26 agent.rsp

[oracle@ora1-1 12104_agent]$ ./agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/agent12c RESPONSE_FILE=/u03/12104_agent/agent.rsp

Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u01/app/oracle/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /u01/app/oracle/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/agent12c/core/12.1.0.4.0 /u03/12104_agent /u01/app/oracle/agent12c -prereq

Validating oms host & port with url: http://ora1-2.mydomain:4903/empbs/genwallet
Validating oms host & port with url: https://ora1-2.mydomain:4903/empbs/genwallet
Return status:3-oms https port is passed
Unzipping the agentcoreimage.zip to /u01/app/oracle/agent12c ....
12.1.0.4.0_PluginsOneoffs_226.zip
Executing command : /u03/12104_agent/unzip -o /u03/12104_agent/12.1.0.4.0_PluginsOneoffs_226.zip -d /u01/app/oracle/agent12c
Executing command : /u01/app/oracle/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/xmlparserv2.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/srvm.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/emCfg.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/agent12c/core/12.1.0.4.0 /u03/12104_agent /u01/app/oracle/agent12c /u01/app/oracle/agent12c/agent_inst AGENT_BASE_DIR=/u01/app/oracle/agent12c


Executing agent install prereqs...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion  -prereqlogloc /u01/app/oracle/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/u01/app/oracle/agent12c/core/12.1.0.4.0/prereqs  -J-DORACLE_HOSTNAME=ora1-1.mydomain -J-DAGENT_PORT=3873 -J-DAGENT_BASE_DIR=/u01/app/oracle/agent12c

Agent install prereqs completed successfully

Cloning the agent home...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/agent12c/core/12.1.0.4.0 -responseFile /u03/12104_agent/agent.rsp  AGENT_BASE_DIR=/u01/app/oracle/agent12c AGENT_BASE_DIR=/u01/app/oracle/agent12c RESPONSE_FILE=/u03/12104_agent/agent.rsp -noconfig  ORACLE_HOME_NAME=agent12cr1 -force b_noUpgrade=true

Cloning of agent home completed successfully

Attaching sbin home...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -attachHome -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/agent12c/sbin ORACLE_HOME_NAME=sbin12c1 -force

Attach home for sbin home completed successfully.

Updating home dependencies...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/u01/app/oracle/agent12c/sbin:/u01/app/oracle/agent12c/core/12.1.0.4.0} -invPtrLoc /u01/app/oracle/agent12c/core/12.1.0.4.0/oraInst.loc -force

Update home dependency completed successfully.
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/u01/app/oracle/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true

ERROR: Agent Configuration Failed SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1

Agent Deploy Log Location:/u01/app/oracle/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy/agentDeploy_2016-04-09_12-34-49-PM.log

The Agent configuration failed with error “ERROR: Agent Configuration Failed SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1”. This error signifies that the configuration failed while securing the agent.

Does that mean that the agent is not installed ? Let’s check the status and try securing the agent manually.

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

The agent is installed but seems to be not running. Expected.
Not sure why the “agentdeploy” script failed securing the agent even though the Agent_Registration_Password is right. Let me try securing it manually with the same password.

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl secure agent oracle123
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Securing agent...   Successful.
[oracle@ora1-1 12104_agent]$

Ok, good to see that it’s success 🙂
Let me check the status of the agent.

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

Agent seems to be not running. Again as expected. Let me start the agent and add the targets manually using the “addinternaltargets” option at the EMCTL prompt.

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting agent ........... started.

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Check the status of the agent.

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent12c/
Agent Log Directory    : /u01/app/oracle/agent12c//sysman/log
Agent Binaries         : /u01/app/oracle/agent12c/core/12.1.0.4.0
Agent Process ID       : 6703
Parent Process ID      : 6610
Agent URL              : https://ora1-1.mydomain:3873/emd/main/
Local Agent URL in NAT : https://ora1-1.mydomain:3873/emd/main/
Repository URL         : https://ora1-2.mydomain:4903/empbs/upload
Started at             : 2016-04-09 12:42:25
Started by user        : oracle
Operating System       : Linux version 2.6.32-71.el6.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : 2016-04-09 12:56:26
Last attempted upload                        : 2016-04-09 12:56:26
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 33
Size of XML files pending upload(MB)         : 0.02
Available disk space on upload filesystem    : 41.15%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-04-09 12:55:47
Last successful heartbeat to OMS             : 2016-04-09 12:55:47
Next scheduled heartbeat to OMS              : 2016-04-09 12:56:47

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ora1-1 12104_agent]$

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine

In this article, I’d like to demonstrate how to installing OEM 12c Release 4 on a linux machine. In the upcoming posts, I’d demonstrate on upgrading this OEM 12c to 13c.

Environment:

OS                          : OEL 6
OEM 12c Software            : 12c Release 4 (12.1.0.4)
Repository Database version : 11.2.0.3
Repositry Database Name     : omsdb
Hostname                    : ora1-2

This article assumes that oracle 11.2.0.3 binary is pre installed on the server and that the repository database “omsdb” is created on it. Do not have the Enterprise Manager DB control configured for this database. If already configured, then have it removed using the “emca” utility with “-deconfig” option.
Download the OEM 12c Release 4 software from here http://www.oracle.com/technetwork/oem/grid-control/downloads/index.html The software consists of 3 disks.
Unzip all the 3 disks into a single directory. Here, I have unzipped them into “/u03/oem12c” directory.

[oracle@ora1-2 ~]$ cd /u03/
[oracle@ora1-2 u03]$ ls -lrt
total 6647416
drwxrwxrwx.  2 oracle oinstall      16384 Feb 26 16:14 lost+found
-rw-r--r--.  1 oracle oinstall 2195693096 Mar 25 14:22 em12104_linux64_disk1.zip
-rw-r--r--.  1 oracle oinstall 1877449643 Mar 25 14:26 em12104_linux64_disk2.zip
-rw-r--r--.  1 oracle oinstall 2727123784 Mar 25 14:31 em12104_linux64_disk3.zip
drwxr-xr-x. 11 oracle oinstall       4096 Mar 25 14:46 oem12c
[oracle@ora1-2 u03]$ cd oem12c
[oracle@ora1-2 oem12c]$ ls -lrt
total 1496020
-rwxr-xr-x. 1 oracle oinstall       5375 Dec 26  2013 runInstaller
drwxr-xr-x. 4 oracle oinstall       4096 Jan 15  2014 bipruntime
-rwxr-xr-x. 1 oracle oinstall 1530333315 May 24  2014 WT.zip
drwxr-xr-x. 4 oracle oinstall       4096 May 24  2014 oms
drwxr-xr-x. 4 oracle oinstall       4096 May 24  2014 jdk
drwxrwxr-x. 2 oracle oinstall       4096 May 24  2014 response
drwxrwxr-x. 2 oracle oinstall       4096 May 24  2014 wls
drwxr-xr-x. 7 oracle oinstall       4096 May 24  2014 install
drwxrwxr-x. 4 oracle oinstall       4096 May 24  2014 libskgxn
drwxr-xr-x. 9 oracle oinstall       4096 May 24  2014 stage
drwxr-xr-x. 2 oracle oinstall       4096 May 24  2014 plugins
-rw-r--r--. 1 oracle oinstall      42623 May 26  2014 release_notes.pdf

Create the required directories for the OMS and the agent installation.

[oracle@ora1-2 ~]$ cd /u01/app/oracle/
[oracle@ora1-2 oracle]$ ls -lrt
total 28
drwxr-xr-x.  2 oracle oinstall 4096 Mar 25 13:32 Clusterware
drwxr-xr-x.  3 oracle oinstall 4096 Mar 25 13:32 ora1-2
drwxr-xr-x.  3 oracle oinstall 4096 Mar 25 13:40 product
drwxrwxr-x. 11 oracle oinstall 4096 Mar 25 13:54 diag
drwxr-x---.  4 oracle oinstall 4096 Mar 25 13:57 admin
drwxr-xr-x.  7 oracle oinstall 4096 Mar 25 14:07 cfgtoollogs
drwxr-xr-x.  2 oracle oinstall 4096 Mar 25 14:12 checkpoints
[oracle@ora1-2 oracle]$
[oracle@ora1-2 oracle]$

[oracle@ora1-2 oracle]$ mkdir oms12c
[oracle@ora1-2 oracle]$ mkdir 12cagent

Start the installation by running the “runInstaller” script.

[oracle@ora1-2 oem12c]$ ./runInstaller

I wish to not receive any updates as this is my own test machine and hence I un-check the “I wish to receive security updates via My Oracle Support”

Select the “Skip” option.

Run the “Prerequisite” checks in the next step. Look out for the Status section to know if the respective check Succeeded or Failed.

Select the “Create a new Enterprise Manager System” option with “Simple”. You can choose the Advanced option for more options.

Provide the Middleware Home Location (OMS Home), Agent Base Directory Location and the Host names on which it has to be installed.

Provide the configuration details such as “Administrative Password”, OMS repository database details such as host name of the OMS repository database, the Port number on which the listener on this host servicing OMS repository database is listening to and the SYS password.

You can opt the “Configure Oracle Software Library” if planning to have a multi-OMS setup, else you may skip this.

There would be some pre-requisite checks performed on the repository database and would be reported if any violations. If there are any other recommendations with changing of the repository database parameters, then necessary actions need to be taken upon accordingly.

Review the summary.

If all perfect, then proceed with the Install option.

Run the script “allroot.sh” as ROOT user when prompted.

[root@ora1-2 ~]# /u01/app/oracle/oms12c/oms/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/oms12c/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/oms12c/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist

Creating /etc/oragchomelist file...
/u01/app/oracle/oms12c/oms
Finished execution of  /u01/app/oracle/oms12c/oms/root.sh ......

Starting to execute /u01/app/oracle/12cagent/core/12.1.0.4.0/root.sh ......
Finished product-specific root actions.
/etc exist
Finished execution of  /u01/app/oracle/12cagent/core/12.1.0.4.0/root.sh ......
[root@ora1-2 ~]#

Review the final page consisting of the details of accessing the OEM 12c.

Login to the OEM 12c using the URL mentioned in the above page.

Accept the License Agreement.

The selected homepage (“Summary” page in my case) will be shown up.

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.

This article demonstrates on how to add a node to a RAC cluster in oracle 12c. The environment used here makes use of ASM as a storage area for the OCR/Voting disks and database physical files.
Environment:

OS              : OEL 6
Existing nodes  : 12cnode1, 12cnode2
Node to be added: 12cnode3
Database Name   : srprim
DB Version      : 12.1.0.2

It is assumed that the new node to be added is configured to access the ASM disks that are shared between the existing nodes. OS installation, oracle or grid user creation and network configuraiton steps are not outlined in this document.

Lets move on with the node addition.

Add Public and private interfaces IP (eth0 and eth1)on the new node. I have assigned the IPs as follows

Public Interface    : 192.168.0.120
Private interface   : 192.168.1.109
Virtual IP interface: 192.168.0.121

Update the the above IP addresses of the new node in the “/etc/hosts” file on all the nodes of the cluster.
Sample of “/etc/hosts” file on the new node. The same entries exists on the other nodes of the cluster.

[root@12cnode3 u02]# cat /etc/hosts

127.0.0.1       localhost.localdomain   localhost
::1           	localhost6.localdomain6 localhost6

################ PUBLIC ###################

192.168.0.115   12cnode1.mydomain       12cnode1
192.168.0.116   12cnode2.mydomain       12cnode2
192.168.0.120   12cnode3.mydomain       12cnode3

############### PRIVATE ###################

192.168.1.107   12cnode1-priv.mydomain  12cnode1-priv
192.168.1.108   12cnode2-priv.mydomain  12cnode2-priv
192.168.1.109   12cnode3-priv.mydomain  12cnode3-priv

############### VIP #######################

192.168.0.117   12cnode1-vip.mydomain   12cnode1-vip
192.168.0.118   12cnode2-vip.mydomain   12cnode2-vip
192.168.0.121   12cnode3-vip.mydomain   12cnode3-vip

############## SCAN #######################

192.168.0.119   node12c-scan.mydomain   node12c-scan

#################################################################

As a pre-requisite check, install the “cvuqdisk-1”, “nfs-utils” rpm on the new node.

[root@12cnode3 u02]# rpm -Uivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]

[root@12cnode3 u02]# rpm -Uivh nfs-utils-1.2.3-15.el6.x86_64.rpm --force --nodeps
warning: nfs-utils-1.2.3-15.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY
Preparing...                ########################################### [100%]
   1:nfs-utils              ########################################### [100%]
[root@12cnode3 u02]#

Update the /etc/sysctl.conf file on the new node with the required kernel parameters. The values can be obtained from the existing nodes.

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Update /etc/security/limits.conf file on the new node. The values can be obtained from the existing nodes.

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    2047
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768

oracle soft memlock unlimited
oracle hard memlock unlimited

Disable the firewall between the nodes.

[root@12cnode3 u02]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

[root@12cnode3 u02]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@12cnode3 u02]#
[root@12cnode3 u02]# chkconfig iptables off
[root@12cnode3 u02]# service iptables status
iptables: Firewall is not running.

Check if avahi-daemon is running and if so, then stop it.

[root@12cnode3 u02]# chkconfig --list | grep avahi
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off
[root@12cnode3 u02]#
[root@12cnode3 u02]# service avahi-daemon stop
Shutting down Avahi daemon:                                [  OK  ]
[root@12cnode3 u02]# chkconfig avahi-daemon off
[root@12cnode3 u02]# chkconfig --list | grep avahi
avahi-daemon    0:off   1:off   2:off   3:off   4:off   5:off   6:off

Set parameter NOZEROCONF to YES in the /etc/sysconfig/network file.

[root@12cnode3 u02]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=12cnode3.mydomain
NOZEROCONF=yes

I have my ASM disks configured using UDEV. Configure UDEV rules as defined on the other nodes of the cluster.
This step can be skipped if UDEV is not being used to configure the disks.

[root@12cnode3 dev]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB1cd37fc0-cc66bbc6", NAME="DSK1", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBc62708e8-ee54011f", NAME="DSK2", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB668ac5ba-619c34bb", NAME="DSK3", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB1dfcc4bc-fc73fb86", NAME="DSK4", OWNER="oracle", GROUP="oinstall", MODE="0660"

Create passwordless ssh between the existing nodes of the cluster and with the new node. This can be done by running the “sshUserSetup.sh” script available with the GI software.

[root@12cnode3 ~]# cd /u03
[root@12cnode3 u03]# ls -lrt
total 52
drwxrwxrwx. 2 oracle oinstall 16384 Mar  7 19:26 lost+found
-rwxr-xr-x. 1 oracle oinstall 32334 Mar 16 20:10 sshUserSetup.sh
[root@12cnode3 u03]#
[root@12cnode3 u03]#
[root@12cnode3 u03]#
[root@12cnode3 u03]# ./sshUserSetup.sh -user oracle -hosts "12cnode1 12cnode2 12cnode3" -noPromptPassphrase -confirm -advanced
The output of this script is also logged into /tmp/sshUserSetup_2016-03-16-20-13-42.log
Hosts are 12cnode1 12cnode2 12cnode3
user is oracle
Platform:- Linux
Checking if the remote hosts are reachable
PING 12cnode1.mydomain (192.168.0.115) 56(84) bytes of data.
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=1 ttl=64 time=1.33 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=2 ttl=64 time=0.338 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=3 ttl=64 time=0.237 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=4 ttl=64 time=0.360 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=5 ttl=64 time=0.243 ms

--- 12cnode1.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.237/0.502/1.332/0.417 ms
PING 12cnode2.mydomain (192.168.0.116) 56(84) bytes of data.
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=1 ttl=64 time=0.787 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=2 ttl=64 time=0.250 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=3 ttl=64 time=0.267 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=4 ttl=64 time=0.260 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=5 ttl=64 time=0.229 ms

--- 12cnode2.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.229/0.358/0.787/0.215 ms
PING 12cnode3.mydomain (192.168.56.113) 56(84) bytes of data.
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=2 ttl=64 time=0.018 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=3 ttl=64 time=0.068 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=4 ttl=64 time=0.058 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=5 ttl=64 time=0.069 ms

--- 12cnode3.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.010/0.044/0.069/0.026 ms
Remote host reachability check succeeded.
The following hosts are reachable: 12cnode1 12cnode2 12cnode3.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost 12cnode1
numhosts 3
The script will setup SSH connectivity from the host 12cnode3.mydomain to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host 12cnode3.mydomain
and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.

NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?
Confirmation provided on the command line

The user chose yes
User chose to skip passphrase related questions.
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /root/.ssh/config, it would be backed up to /root/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host with empty passphrase
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a4:83:da:75:8e:99:a2:57:63:a7:2b:b0:e2:df:4f:09 root@12cnode3.mydomain
The key's randomart image is:
+--[ RSA 1024]----+
|                 |
|                 |
|        .        |
|     . o         |
|    .E+ S        |
|  .o .=Bo        |
|  .ooo+*.        |
|. ..+.o          |
|.ooo ooo         |
+-----------------+
Creating .ssh directory and setting permissions on remote host 12cnode1
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode1. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode1.
Warning: Permanently added '12cnode1,192.168.0.115' (RSA) to the list of known hosts.
oracle@12cnode1's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode1.
Creating .ssh directory and setting permissions on remote host 12cnode2
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode2. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode2.
Warning: Permanently added '12cnode2,192.168.0.116' (RSA) to the list of known hosts.
oracle@12cnode2's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode2.
Creating .ssh directory and setting permissions on remote host 12cnode3
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode3. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode3.
Warning: Permanently added '12cnode3' (RSA) to the list of known hosts.
oracle@12cnode3's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode3.
Copying local host public key to the remote host 12cnode1
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode1.
oracle@12cnode1's password:
Permission denied, please try again.
oracle@12cnode1's password:
Done copying local host public key to the remote host 12cnode1
Copying local host public key to the remote host 12cnode2
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode2.
oracle@12cnode2's password:
Done copying local host public key to the remote host 12cnode2
Copying local host public key to the remote host 12cnode3
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode3.
oracle@12cnode3's password:
Done copying local host public key to the remote host 12cnode3
Creating keys on remote host 12cnode1 if they do not exist already. This is required to setup SSH on host 12cnode1.

Creating keys on remote host 12cnode2 if they do not exist already. This is required to setup SSH on host 12cnode2.

Creating keys on remote host 12cnode3 if they do not exist already. This is required to setup SSH on host 12cnode3.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
de:87:40:a9:83:7d:92:22:7e:01:cf:95:4d:25:7b:52 oracle@12cnode3.mydomain
The key's randomart image is:
+--[ RSA 1024]----+
|        o.E      |
|       + =       |
|  .   o * .      |
|   + + + o       |
|  . * * S        |
| . . o = o .     |
|  . .   . o .    |
|   .       .     |
|                 |
+-----------------+
Updating authorized_keys file on remote host 12cnode1
Updating known_hosts file on remote host 12cnode1
Updating authorized_keys file on remote host 12cnode2
Updating known_hosts file on remote host 12cnode2
Updating authorized_keys file on remote host 12cnode3
Updating known_hosts file on remote host 12cnode3
SSH setup is complete.

------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user oracle.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--12cnode1:--
Running /usr/bin/ssh -x -l oracle 12cnode1 date to verify SSH connectivity has been setup from local host to 12cnode1.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:26 IST 2016
------------------------------------------------------------------------
--12cnode2:--
Running /usr/bin/ssh -x -l oracle 12cnode2 date to verify SSH connectivity has been setup from local host to 12cnode2.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:26 IST 2016
------------------------------------------------------------------------
--12cnode3:--
Running /usr/bin/ssh -x -l oracle 12cnode3 date to verify SSH connectivity has been setup from local host to 12cnode3.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:27 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode1
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Wed Mar 16 20:14:28 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode2
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Wed Mar 16 20:14:27 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode3
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Warning: Permanently added the RSA host key for IP address '192.168.0.120' to the list of known hosts.
Wed Mar 16 20:14:28 IST 2016
------------------------------------------------------------------------
-Verification from complete-
SSH verification complete.
[root@12cnode3 u03]#

Though the script says that verification of passwordless ssh is verified, I would like to check this out with a small test.

[oracle@12cnode1 ~]$ ssh 12cnode2 date
Wed Mar 16 20:15:53 IST 2016
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ ssh 12cnode3 date
Wed Mar 16 20:15:59 IST 2016
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ ssh 12cnode1 date
Wed Mar 16 20:16:05 IST 2016
[oracle@12cnode1 ~]$

[oracle@12cnode2 ~]$ ssh 12cnode1 date
Wed Mar 16 20:16:15 IST 2016
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$ ssh 12cnode3 date
Warning: Permanently added the RSA host key for IP address '192.168.0.120' to the list of known hosts.
Wed Mar 16 20:16:19 IST 2016
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$ ssh 12cnode2 date
Wed Mar 16 20:16:25 IST 2016

[oracle@12cnode3 dev]$ ssh 12cnode1 date
Wed Mar 16 20:16:37 IST 2016
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$ ssh 12cnode2 date
Wed Mar 16 20:16:42 IST 2016
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$ ssh 12cnode3 date
Wed Mar 16 20:16:48 IST 2016

Run the CLUVFY utility with the options “pre nodeadd” to check the pre-requisites on the new node. This needs to be run from the node which is already part of the cluster.
Here, it’s being run from node “12cnode1”.

[oracle@12cnode1 bin]$ cd /u01/app/12.1.0.2/grid/bin
[oracle@12cnode1 bin]$ ./cluvfy stage -pre nodeadd -n 12cnode3 -verbose > /u02/12cnode3prenodeadd.txt
[oracle@12cnode1 bin]$

Let me check what all conditions are not met.

[oracle@12cnode1 bin]$ grep -i "failed" /u02/12cnode3prenodeadd.txt
  12cnode3      2.6.32-71.el6.x86_64      2.6.39                    failed
  12cnode1      2.6.32-71.el6.x86_64      2.6.39                    failed
Result: Kernel version check failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
  12cnode1                              failed
  12cnode3                              failed
Check for integrity of file "/etc/resolv.conf" failed

The above 2 conditions have failed. We can ignore the Kernel version if it’s greater than or equal to 2.6.32. The integrity check of file “/etc/resolv.conf” for DNS has failed and I would ignore this as I’m not using any DNS.
Let’s move on with the node addition. In 12c, the location of “addnode.sh” is quite different than that of in 11gR2. In 11gR2, it was located under GRID_HOME/oui/bin, but in 12c, it’s located under “GRID_HOME/addnode”. On the node which is already a part of the cluster, run the “addnode.sh” script. Here, it’s been run from 12cnode1 in silent mode (and not through GUI). Specify the new node name and new node VIP name. Since the above mentioned 2 conditions have failed, I’m running “addnode.sh” with the “ignorePrereq” option.

[oracle@12cnode1 ~]$ cd /u01/app/12.1.0.2/grid/addnode/
[oracle@12cnode1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={12cnode3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={12cnode3-vip}" -silent -ignorePrereq
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 44743 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8138 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2016-03-23_05-18-17PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.1.0.2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/oraInventory/orainstRoot.sh
        2. /u01/app/12.1.0.2/grid/root.sh

Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[12cnode3]
Execute /u01/app/12.1.0.2/grid/root.sh on the following nodes:
[12cnode3]

The scripts can be executed in parallel on all the nodes.

..........
Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.
[oracle@12cnode1 addnode]$

Run as ROOT user the scripts “/u01/app/oraInventory/orainstRoot.sh” and “/u01/app/12.1.0.2/grid/root.sh” on the new node 12cnode3.

Here is the execution of the script “orainstRoot.sh”

[root@12cnode3 grid]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Execution of “root.sh” is as follows:

[root@12cnode3 grid]# /u01/app/12.1.0.2/grid/root.sh
Check /u01/app/12.1.0.2/grid/install/root_12cnode3.mydomain_2016-03-23_18-49-49.log for the output of root script
[root@12cnode3 grid]#

[root@12cnode3 ~]# cat /u01/app/12.1.0.2/grid/install/root_12cnode3.mydomain_2016-03-23_18-49-49.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/03/23 18:49:51 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2016/03/23 18:49:52 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2016/03/23 18:49:56 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2016/03/23 18:52:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12cnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12cnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on '12cnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12cnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on '12cnode3'
CRS-2672: Attempting to start 'ora.evmd' on '12cnode3'
CRS-2676: Start of 'ora.mdnsd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.evmd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on '12cnode3'
CRS-2676: Start of 'ora.gpnpd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on '12cnode3'
CRS-2676: Start of 'ora.gipcd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode3'
CRS-2676: Start of 'ora.cssdmonitor' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '12cnode3'
CRS-2672: Attempting to start 'ora.diskmon' on '12cnode3'
CRS-2676: Start of 'ora.diskmon' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cssd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode3'
CRS-2672: Attempting to start 'ora.ctssd' on '12cnode3'
CRS-2676: Start of 'ora.ctssd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on '12cnode3'
CRS-2676: Start of 'ora.asm' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12cnode3'
CRS-2676: Start of 'ora.storage' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crf' on '12cnode3'
CRS-2676: Start of 'ora.crf' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '12cnode3'
CRS-2676: Start of 'ora.crsd' on '12cnode3' succeeded
CRS-6017: Processing resource auto-start for servers: 12cnode3
CRS-2672: Attempting to start 'ora.net1.network' on '12cnode3'
CRS-2672: Attempting to start 'ora.FRA.dg' on '12cnode3'
CRS-2676: Start of 'ora.net1.network' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.ons' on '12cnode3'
CRS-2676: Start of 'ora.ons' on '12cnode3' succeeded
CRS-2676: Start of 'ora.FRA.dg' on '12cnode3' succeeded
CRS-6016: Resource auto-start has completed for server 12cnode3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/03/23 19:01:12 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2016/03/23 19:02:21 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Let’s check the CRS status and the resources on the new node.

[root@12cnode3 ~]# cd /u01/app/12.1.0.2/grid/bin/
[root@12cnode3 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@12cnode3 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.asm
               ONLINE  ONLINE       12cnode1                 Started,STABLE
               ONLINE  ONLINE       12cnode2                 Started,STABLE
               ONLINE  ONLINE       12cnode3                 Started,STABLE
ora.net1.network
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.ons
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.12cnode1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.12cnode2.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.12cnode3.vip
      1        ONLINE  ONLINE       12cnode3                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       12cnode1                 169.254.3.189 192.16
                                                             8.1.107,STABLE
ora.cvu
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.srprim.db
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
      2        ONLINE  ONLINE       12cnode2                 Open,STABLE
--------------------------------------------------------------------------------
[root@12cnode3 bin]#

Cluster status on all the nodes:

[oracle@12cnode1 ~]$ cd /u01/app/12.1.0.2/grid/bin/
[oracle@12cnode1 bin]$ ./crsctl check cluster -all
**************************************************************
12cnode1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
12cnode2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
12cnode3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@12cnode1 bin]$

Now it’s time to add the RDBMS home on to the new node. This is done in the same way as above by running the “addNode.sh” script from the RDBMS home on the existing node and not from the new node. (Of-course, it cannot be run on the new node as the RDBMS Home is still not available in it 🙂 )
The script is located at $ORACLE_HOME/addnode and here it’s being run from 12cnode1. Here too, I’m ignoring the pre-requisite as the pre-requisite for “resolve.conf” is not met.

[oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/addnode/
[oracle@12cnode1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={12cnode3}" -silent -ignorePrereq
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 44736 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8189 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2016-03-24_11-18-10AM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.2/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0.2/db_1/root.sh

Execute /u01/app/oracle/product/12.1.0.2/db_1/root.sh on the following nodes:
[12cnode3]

..........
Update Inventory in progress.

..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.
[oracle@12cnode1 addnode]$

Now, run the “ROOT.SH” script from the new node as ROOT user. Here is the execution of the script.

[root@12cnode3 ~]# /u01/app/oracle/product/12.1.0.2/db_1/root.sh
Check /u01/app/oracle/product/12.1.0.2/db_1/install/root_12cnode3.mydomain_2016-03-24_12-01-16.log for the output of root script
[root@12cnode3 ~]# cat /u01/app/oracle/product/12.1.0.2/db_1/install/root_12cnode3.mydomain_2016-03-24_12-01-16.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0.2/db_1
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@12cnode3 ~]#

We now need to add a new instance “srprim3” on the cluster database that needs to be run on the new node 12cnode3. But before we do that, we need to have few of the cluster parameters modified according to the instance.

instance_number
Thread
Online Redo log groups for the new thread
UNDO_TABLESPACE

Let’s get the current log group and it’s members’ details based on which we can add new groups to the new thread.

SYS@srprim1>select group#,thread#,bytes/1024/1024,members from v$log;

    GROUP#    THREAD# BYTES/1024/1024    MEMBERS
---------- ---------- --------------- ----------
         1          1              50          2
         2          1              50          2
         3          2              50          2
         4          2              50          2

SYS@srprim1>select group#,member from v$logfile order by group#;

    GROUP# MEMBER
---------- -------------------------------------------------------
         1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571
         1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571
         2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577
         2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575
         3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373
         3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375
         4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377
         4 +FRA/SRPRIM/ONLINELOG/group_4.260.904396379

Let me add new groups to the new thread 3.

SYS@srprim1>alter database add logfile thread 3 group 5 size 50M;

Database altered.

SYS@srprim1>alter database add logfile thread 3 group 6 size 50M;

Database altered.

SYS@srprim1>select group#,thread#,bytes/1024/1024,members from v$log;

    GROUP#    THREAD# BYTES/1024/1024    MEMBERS
---------- ---------- --------------- ----------
         1          1              50          2
         2          1              50          2
         3          2              50          2
         4          2              50          2
         5          3              50          2
         6          3              50          2

6 rows selected.

SYS@srprim1>select group#,member from v$logfile order by group#;

    GROUP# MEMBER
---------- -------------------------------------------------------
         1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571
         1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571
         2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575
         2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577
         3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373
         3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375
         4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377
         4 +FRA/SRPRIM/ONLINELOG/group_4.260.904396379
         5 +DATA/SRPRIM/ONLINELOG/group_5.296.907330775
         5 +FRA/SRPRIM/ONLINELOG/group_5.261.907330777
         6 +DATA/SRPRIM/ONLINELOG/group_6.297.907330877

    GROUP# MEMBER
---------- -------------------------------------------------------
         6 +FRA/SRPRIM/ONLINELOG/group_6.262.907330879

12 rows selected.

Getting the UNDO tablespace details in order to know the undo tabelspaces assigned to the instances of the database.

SYS@srprim1>select inst_id,name,value from gv$parameter where name like 'undo_tablespa%';

   INST_ID NAME                                     VALUE
---------- ---------------------------------------- ----------------------------------------
         2 undo_tablespace                          UNDOTBS2
         1 undo_tablespace                          UNDOTBS1

Now, let me create a new UNDO tablespace called “undotbs3” and is assign it to the new instance “srprim3”.

SYS@srprim1>create undo tablespace UNDOTBS3 datafile size 400M;

Tablespace created.

SYS@srprim1>alter system set undo_tablespace=UNDOTBS3 scope=spfile sid='srprim3';

System altered.

Assign an instance number to the new instance.

SYS@srprim1>alter system set instance_number=3 SCOPE=SPFILE SID='srprim3';

System altered.

Assign the newly created Redo Log thread to the new instance.

SYS@srprim1>alter system set thread=3 scope=spfile sid='srprim3';

System altered.

The current configuration of the “srprim” database is as follows:

[oracle@12cnode1 admin]$ srvctl config database -db srprim
Database unique name: srprim
Database name: srprim
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381
Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: srprim1,srprim2
Configured nodes: 12cnode1,12cnode2
Database is administrator managed
[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$

It’s time to add the instance “srprim3” to the database “srprim” using srvctl utility.

[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$ srvctl add instance -db srprim -instance srprim3 -node 12cnode3

Now, check the updated database configuration.

[oracle@12cnode1 admin]$ srvctl config database -db srprim
Database unique name: srprim
Database name: srprim
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381
Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: srprim1,srprim2,srprim3
Configured nodes: 12cnode1,12cnode2,12cnode3
Database is administrator managed
[oracle@12cnode1 admin]$

Since the PFILE of instance srprim1 has nothing but only the location of the SPFILE, copy it to the new node and rename it accordingly to the new instance name (srprim3).

[oracle@12cnode1 ~]$ cd $ORACLE_HOME/dbs
[oracle@12cnode1 dbs]$ ls -lrt initsrprim*
-rw-r-----. 1 oracle oinstall 39 Feb 21 13:15 initsrprim1.ora
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$ cat initsrprim1.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$ scp initsrprim1.ora oracle@12cnode3:/u01/app/oracle/product/12.1.0.2/db_1/dbs/initsrprim3.ora
initsrprim1.ora                                                                                                                      100%   39     0.0KB/s   00:00
[oracle@12cnode1 dbs]$

[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/dbs
[oracle@12cnode3 dbs]$ ls -lrt initsrprim*
-rw-r-----. 1 oracle oinstall 39 Mar 24 12:38 initsrprim3.ora
[oracle@12cnode3 dbs]$
[oracle@12cnode3 dbs]$
[oracle@12cnode3 dbs]$ cat initsrprim3.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'
[oracle@12cnode3 dbs]$

Finally, before starting the instance, enable the newly created Redo Log thread 3 that was assigned to instance srprim3.

SYS@srprim1>alter database enable public thread 3;

Database altered.

SYS@srprim>

Start the instance srprim3 using SRVCTL utility.

[oracle@12cnode1 ~]$ srvctl status database -db srprim
Instance srprim1 is running on node 12cnode1
Instance srprim2 is running on node 12cnode2
Instance srprim3 is not running on node 12cnode3
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl start instance -db srprim -instance srprim3
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl status database -db srprim
Instance srprim1 is running on node 12cnode1
Instance srprim2 is running on node 12cnode2
Instance srprim3 is running on node 12cnode3

Below is just a list of the databases and the listeners running on the new node 12cnode3.

[oracle@12cnode3 ~]$ ps -ef | grep pmon
oracle    4266     1  0 10:24 ?        00:00:00 asm_pmon_+ASM3
oracle    7749     1  0 10:28 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   23632     1  0 12:42 ?        00:00:00 ora_pmon_srprim3
oracle   25350  3534  0 12:47 pts/0    00:00:00 grep pmon

[oracle@12cnode3 ~]$ ps -ef | grep tns
root         9     2  0 10:15 ?        00:00:00 [netns]
oracle    5774     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
oracle    5853     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
oracle    5856     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle   25406  3534  0 12:47 pts/0    00:00:00 grep tns
[oracle@12cnode3 ~]$

If the addnode.sh is run through GUI mode without using the “silent” option, then the instance addition too would be taken care by the GUI tool and there would be no requirement to go through the above manual steps.

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.

This post demonstrates on how to perform a GI upgrade from 11.2.0.3 version to 12.1.0.2 on a standalone machine. Usually the upgrade is performed via a GUI, but here I’m demonstrating the upgrade through silent mode using the response file.

Environment:

Hostname                                  :   ora1-4
Current GI version                        :   11.2.0.3
GI response file location                 :   /u02/grid_install.rsp
Existing 11.2.0.3 GI Home                 :   /u01/app/oracle/product/11.2.0.3/grid
To be installed 12.1.0.2 GI Home location :   /u01/app/oracle/product/12.1.0.2/grid/

As you know, GI on a standalone machine makes use of High Availability Service (HAS) and not Cluster Ready Service (CRS). Let’s query the current HAS version.

[oracle@ora1-4 bin]$ ./crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.3.0]
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$ ./crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.3.0]
[oracle@ora1-4 bin]$

Unzip the 12.1.0.2 GI software pack and take a copy of the response file grid_install.rsp located at “<unzipped location>/grid/response”. Here I have copied it to “/u02”.

Make necessary changes to the copied response file. The changes for a standalone GI upgrade include:

1. ORACLE_HOSTNAME
     - Specify the hostname on which the GI upgrade is being performed.
2. INVENTORY_LOCATION
     - Specify the Inventory location.
3. oracle.install.option
     - Choose the option as UPGRADE as this is what we are performing
4. ORACLE_BASE
     - Specify the ORACLE_BASE location.
5. ORACLE_HOME
     - Specify the location where GI 12.1.0.2 needs to be installed.
6. Specify the OS groups for OSDBA, OSOPER and OSASM

7. Since this is a standalone, SCAN and Cluster has no role to play. So skipping these.

8. ASM part is also skipped because the GI on 11.2.0.3 is already having an ASM instance using DATA_NEW and FRA_NEW as diskgroups. Since I do not want a fresh ASM instance or a fresh diskgroup to be created, this has been skipped.

9. Under the UPGRADE section, if you would want the GI to be managed by OEM Grid Control, then select oracle.install.config.managementOption to CLOUD_CONTROL, else choose NONE. If CLOUD_CONTROL is chosen, then provide more details that has been asked for like OMS host name, OMS Port. In this demo, I do not want the GI to be managed by Grid Control and hence chosen as NONE.

Here is the response file with the updated values used to perform the upgrade to 12.1.0.2.

GI respone file.
Run the “runInstaller” with the silent option and providing the complete path of the response file. Make sure that ORACLE_HOME and ORA_CRS_HOME are unset else you may face the below error message.

[oracle@ora1-4 grid]$ ./runInstaller -silent -responseFile /u02/grid_install.rsp -showProgress
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB. Actual 8756 MB Passed
Checking swap space: must be greater than 150 MB. Actual 10236 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-01-10_06-29-38PM. Please wait ...[oracle@ora1-4 grid]$ [FATAL] [INS-30131] Initial setup required for the execution of installer validations failed.
CAUSE: Failed to access the temporary location.
ACTION: Ensure that the current user has required permissions to access the temporary location.
*ADDITIONAL INFORMATION:*
Exception details
 - PRVG-1561 : Setting ORA_CRS_HOME variable is not supported

Unset the ORACLE_HOME and ORA_CRS_HOME and run the runInstaller with silent option. If the pre-requisite checks fail, then the installation throws error.
If so, then take necessary actions accordingly.

[oracle@ora1-4 grid]$ unset ORACLE_HOME
[oracle@ora1-4 grid]$ unset ORA_CRS_HOME
[oracle@ora1-4 grid]$
[oracle@ora1-4 grid]$ ./runInstaller -silent -responseFile /u02/grid_install.rsp -showProgress
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB. Actual 8756 MB Passed
Checking swap space: must be greater than 150 MB. Actual 10236 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-01-10_06-30-53PM. Please wait ...
You can find the log of this install session at:
/u01/app/oraInventory/logs/installActions2016-01-10_06-30-53PM.log

Prepare in progress.
.................................................. 7% Done.

Prepare successful.

Copy files in progress.
.................................................. 12% Done.
.................................................. 22% Done.
.................................................. 27% Done.
.................................................. 32% Done.
.................................................. 40% Done.
.................................................. 48% Done.
.................................................. 53% Done.
.................................................. 58% Done.
.................................................. 63% Done.
....................
Copy files successful.

Link binaries in progress.
..........
Link binaries successful.

Setup files in progress.
.................................................. 68% Done.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.
.................................................. 73% Done.

Finish Setup successful.
The installation of Oracle Grid Infrastructure 12c was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2016-01-10_06-30-53PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
.................................................. 82% Done.

Prepare for configuration steps in progress.

Prepare for configuration steps successful.
..........
Update Inventory in progress.

Update Inventory successful.
.................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/grid/rootupgrade.sh
Run the script on the local node.

.................................................. 100% Done.
Successfully Setup Software.

Run the rootupgrade.sh script as ROOT user from the new 12.1.0.2 GI home. Here is outcome of the rootupgrade.sh script.

[root@ora1-4 ~]# /u01/app/oracle/product/12.1.0.2/grid/rootupgrade.sh
Check /u01/app/oracle/product/12.1.0.2/grid/install/root_ora1-4.mydomain_2016-01-10_18-50-05.log for the output of root script

[root@ora1-4 ~]#

Let’s check the output for the rootupgrade script.

[oracle@ora1-4 ~]$ cat /u01/app/oracle/product/12.1.0.2/grid/install/root_ora1-4.mydomain_2016-01-10_18-50-05.log
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/12.1.0.2/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/oracle/product/12.1.0.2/grid/crs/install/crsconfig_params

ASM Configuration upgraded successfully.

Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ora1-4 successfully pinned.
2016/01/10 18:51:37 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

2016/01/10 18:52:40 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

2016/01/10 18:54:10 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p first'

2016/01/10 18:54:18 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p last'

ora1-4 2016/01/10 18:54:19 /u01/app/oracle/product/12.1.0.2/grid/cdata/ora1-4/backup_20160110_185419.olr 0

ora1-4 2016/01/09 13:02:23 /u01/app/oracle/product/11.2.0.3/grid/cdata/ora1-4/backup_20160109_130223.olr -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora1-4'
CRS-2673: Attempting to stop 'ora.evmd' on 'ora1-4'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ora1-4'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ora1-4' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ora1-4' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ora1-4'
CRS-2677: Stop of 'ora.cssd' on 'ora1-4' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora1-4' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/01/10 18:56:03 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

[oracle@ora1-4 ~]$

ASM too has been upgraded. Let’s check the HAS version from the 12.1.0.2 GI home.

[oracle@ora1-4 grid]$ cd /u01/app/oracle/product/12.1.0.2/grid/bin/
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$ ./crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$
[oracle@ora1-4 bin]$ ./crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.1.0.2.0]

The ASM home too would be updated in the ORATAB file automatically during the upgrade.

[oracle@ora1-4 bin]$ cat /etc/oratab | grep ASM
+ASM:/u01/app/oracle/product/12.1.0.2/grid:N # line added by Agent

Let’s connect to ASM instance and verify ther version and the state of it’s diskgroups.

[oracle@ora1-4 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Sun Jan 10 19:33:28 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA_NEW                       MOUNTED
FRA_NEW                        MOUNTED

SQL> select banner from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
PL/SQL Release 12.1.0.2.0 - Production
CORE 12.1.0.2.0 Production
TNS for Linux: Version 12.1.0.2.0 - Production
NLSRTL Version 12.1.0.2.0 - Production

As seen above, the GI has been upgraded successfully from 11.2.0.3 to 12.1.0.2.

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 
DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.