Advertisements
Skip to content
January 16, 2017 / Shivananda Rao P

Cache Fusion – Internals of Block Transfer within RAC DB instances

We all know that each instance of a RAC database has it’s own buffer cache in its SGA and all these caches put together forms the cache fusion. The blocks within the RAC instances are transferred using a high speed private interconnect. In order to make sure that no two or more instances are updating the same block at the same time and also to track what each instance is performing with the block, cache coherency and consistency needs to be maintained. Oracle RAC maitains this cache coherency and consistency using two main services GCS (Global Cache Service)and GES (Global Enqueue Service).

 

GCS with the help of LMS processes, co-ordinates within the RAC instances and records the status of the cached data blocks in the GRD (Global Resource Directory). The GRD is distributed across all the active instances and contains the information of the cached block such as the block number, which instance is owning the current version of the block, mode of the block, role of the block.

 

GES maintains the coherency of enqueues on the dictionary and library cache. It keeps track of all global enqueues of the resources in the RAC environment.

 

Based on what a resource holder is requesting to perform, a data block can be of any of the 3 modes.
 
1. NULL (N): A null mode indicates just a resource holder and has no access rights.

2. Exclusive (X): An exclusive mode signifies exclusive access of the block. This means that the resource holder needs to perform a write operation on the block and no other resource can write over it. However, other resources can perform read operation on the block.

3. Shared (S): A shared mode indicates that the resource holder has a shared lock on the block and is performing a read operation. As the mode name, since the lock is shared, any other resource can also read the block.

 

In addition to the modes, GCS also has roles for the resources.
 
1. Local role: When a data block is first read from the disk into the cache, it’s role is said to be LOCAL. This also means that no other modified/dirty copy of the block exists in the cache.

2. Global Role: When a data block is acquired from a remote instance and if this data block is already modified in the remote instance, then the role of the block is said to be GLOBAL. If multiple copies of modified/dirtied data block exists across multiple instances, then the role of this data block is considered to be GLOBAL.

 

Another important concept that we need to know of is the “Past Image” (PI). As the name says, an image copy of the modified/updated data block is saved by the modifying instance before passing on this data block to the other requesting instance that would like to perform either read or write operation. A PI block is considered to be the current version of the block and in case of a node failure, GCS would start the recovery from this PI block thereby reducing the recovery time. Once the latest version of the data block is written to the disk (checkpoint), GCS informs all the instances having PI to discard those images.

 

All the above details speak about the data blocks, but when a block is read into the memory, it would be stored in the buffer. The state of a buffer is determined based on the 3 characters.
 

1. Lock mode – N (NULL), S (Shared), X (Exclusive)

2. Role – L(Local), G(Global)

3. Past Image(PI): Number indicating the number of past images

 

GV$BH view can be used to check the status of the block against an instance of the database. The values represent:

1. “cr“: This represents a NULL lock mode on the block.

2. “SCUR“: This represents that a SHARED lock is held on the block by that particular instance.

3. “XCUR“: This represents an EXCLUSIVE lock is held on the block by that particular instance.
 
Lets consider a 3 node RAC as an example to see how the block transfer works with user A connected to instance 1, user B connected to instance 2 and user C connected to instance 3.
 
Assuming that the the table EMP under schema BTTEST is created freshly and no connections from any of the 3 instances have accessed this table, the following scenarios have been defined. But before moving on to the scenarios, let’s capture the block relevant information for a row of this EMP table. This can be done with the help of dbms_rowid.
 


SQL> select * from bttest.emp;

CODE       NAME
---------- --------------------------------------------------
100        JAMES
200        SCOTT
300        SMITH
400        JOHN

 


SQL>select owner,object_name,data_object_id from dba_objects where object_name='EMP';

OWNER     OBJECT_NAME  DATA_OBJECT_ID
--------- ------------ ----------------
BTTEST    EMP          91791

 


SQL>select dbms_rowid.rowid_relative_fno(rowid) FILE#,dbms_rowid.rowid_block_number(rowid) BLOCK# from bttest.emp where code=300;

FILE#      BLOCK#
---------- ----------
11         135

 
Considering the block 135 which holds a row of the table EMP whose value for column CODE is 300, we are explaining the following scenarios.
 
Scenario 1:

User B on instance 2 performs a SELECT operation on this table which access the contents of the block 135.

SQL statement run: select * from bttest.emp where code=300;

1. Since no other connections from other instances have accessed this table previously, the data block 135 needs to be read from the disk and written on to the buffer.
 
2. User B will now hold a Shared (S) lock on this block as it’s performing only a READ and not WRITE operation. Since the block is being read for the first time from the disk and that there exists no dirty copy of this block in the buffer cache, the role of the block is LOCAL (L). The third consideration is that has been no modification done to this block by USER B and hence there exists no PAST IMAGES (0). With this, the mode of the block on instance 2 would be SL0 and nothing with respect to instance 1 and 3.
 
Querying the GV$BH view to capture the details of the block, we see that the status of the block for instance 2 is SCUR which indicates that the block is held in SHARED lock by instance 2. There is no information for instance 1 and 3 as they haven’t yet accessed this block.
 


SQL> select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        scur       N 91791      2

 
===========================================================================
 
Scenario 2:

Now user C on instance 3 performs an UPDATE operation on the row in the block 135. The SQL statement run is:

update bttest.emp set name=’UPDATE3′ where code=300;

Since the block is already available in the cache of instance B, there would be no requirement to read the block from the disk and thereby avoiding physical read.
 
1. Instance 3 would send the request to GCS which knows who currently owns the block (using the information from GRD).
 
2. GCS forwards the request to instance B who is currently holding the SHARED lock.
 
3. Instance 2 would downgrade it’s lock on the block from SHARED to NULL and since there is no modification done on this block by instance 2, the role of the block remains LOCAL and Past Image (PI) count is 0. Thereby, the mode of the block on instance 2 would be NL0.
 
4. Instance 2 would then send the requested block to instance 3 and updates this to the GCS.
 
5. Instance 3 will now go for an EXCLUSIVE lock (X) on the block and the role of the block still remains LOCAL as the instance previously holding this block (instance 2) hasn’t made any modification to this block and so would the PI count be 0. The mode of the block on instance 3 would be XL0.
 
6. Instance 1 has not come into picture yet with respect to this block.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        cr         N 91791      2
11         135        xcur       Y 91791      3

 
From the above result, we can see that the buffer status on instance 2 is CR (NULL) and no modifications being done to the block. While on instance 3, the buffer status is XCUR (EXCLUSIVE) and instance 3 has dirtied (modified) (column DIRTY – Value “Y”) the block.
 
===========================================================================
 
Scenario 3:
 
User A on instance 1 runs a “select” statement to access the row in block 135.
 
1. Instance 1 requests GCS for the data block. GCS knows that instance 3 owns the block and has an EXCLUSIVE lock on it.
 
2. GCS forwards the request to instance 3.
 
3. Instance 3 would now bring down it’s lock from Exclusive to Shared on the requested block. The lock would have been lowered to NULL if instance 1 would have requested the block to be modified (WRITE operation), instead, it has requested for a READ operation. Since instance 3 has modified the block in the previous scenario, the role of the block would be GLOBAL and GCS be informed that the requesting instance needs to have this block in the GLBOAL role.
 
4. Since instance 3 has modified the block in scenario 2, it retains a copy (PI) of the modified block and sends the requested block. Instance 3 will now have SG1 mode on the block.
 
5. Instance 1 receives the block and will hold a Shared (S) lock with Global Role (as informed previously) with 0 Past Images (there exists no past images for this block on instance 1). So, instance 1 will now have SG0 mode on the block.
 
6. The mode of the block on instance 2 will be NG0. The role of the block will be changed to GLOBAL as instance 3 has modified the block in the previous scenario.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        cr         N 91791      2
11         135        scur       N 91791      1
11         135        scur       Y 91791      3

 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        scur       N 91791      1
11         135        scur       Y 91791      3

 
When I query the GV$BH view, I could see that both instance 1 and 3 have SHARED lock on the block 135 whereas instance 2 with just CR mode (NULL). Since instance 3 has dirtied this block previously, the value for column DIRTY against instance 3 remains as Y (YES).
 
===========================================================================
 
Scenario 4:
 
Now, I run an “Update” statement on the same block through User B from instance 2.
 
1. Instance 2 requests GCS for the data block. The block was last accessed by instance 1 and GCS would forward the request to instance 1.
 
2. Instance 1 would send the block to the requesting instance 2 through GCS. But, before passing on the block, instance 1 would downgrade it’s lock on the block from Shared to NULL with the role of the block being “GLOBAL” and not holding any Past Images. The mode of the block on instance 1 would now be NG0.
 
3. Since instance 2 has requested for WRITE operation, instance 3 will also downgrade it’s lock on the block from SHARED to NULL. The role of the block remains GLOBAL and since it holds 1 PAST IMAGE (as per Scenario 3), the mode of the block on instance 3 will be NG1.
 
4. Instance 2 will acquire an EXCLUSIVE (X) lock on the block with the role of the block retained as GLOBAL and the PI count remains 0. The PI count remains 0 for instance 2 is because this instance has never performed any modifications to this block and thereby not retaining any copies of the modifications. The mode of the block on instance 2 would be XG0.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        xcur       Y 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        pi         Y 91791      3
11         135        cr         N 91791      3

 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      3
11         135        xcur       Y 91791      2

 
Querying GV$BH, one can see that the status of the buffer on instance 2 is XCUR with the value for the column DIRTY being “Y”. Also, it shows a clear image of the status of the buffer being shown as “PI” with the value for the column DIRTY being “Y” against instance 3. This is related to the scenario 2 where instance 3 performed an UPDATE operation on the block 135. The values “CR” for the column “STATUS” against instance 2 or instance 3 represent the status of the block in the previous scenarios and not to the current one.
 
===========================================================================
 
Scenario 5:
 
User C on instance 3 runs a “SELECT” statement to access row in block 135.
 
1. Instance 3 requests GCS for the block 135. GCS then forwards the request to instance 2 which is holding the block with an EXCLUSIVE lock.
 
2. Instance 2 downgrades its lock from “EXCLUSIVE” to “SHARED”, adds a flag that it modified the block as stated in scenario 4 and thereby declaring that it has 1 PAST IMAGE. However the role of the block still remains GLBOAL and the mode of this block would be SG1 on instance 2.
 
3. The block is then received by instance 3 and since it requested for a READ operation, it would acquire a SHARED lock on the block. The role of the block would be GLBOAL (as multiple changes have occurred to this block by remoted instances too). The PAST IMAGE count for this bock on instance 3 would be 1 (Reason – there has been only 1 change made to this block by instance 3 (as explained in scenario 2)). Finally, the mode of the block on instance 3 will be SG1.
 
4. Since instance 1 hasn’t come into picture in this scenario, it would retain it’s mode as it had in the previous scenario i.e, NG0.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        scur       Y 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        pi         Y 91791      3
11         135        scur       N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3

11 rows selected.

 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      3
11         135        scur       N 91791      3
11         135        scur       Y 91791      2

 
From the above result, we can see that the status of the block against instance 2 and instance 3 is SCUR (SHARED lock) and that NULL (cr) for instance 1.
 
===========================================================================
 
Scenario 6:
 
User A on instance 1 performs an UPDATE operation on the same block 135.
 
SQL statement run: “update bttest.emp set name=’UPDATE2′ where code=300;
 
1. Instance 1 requests for the block with GCS. GCS knows that the block was last modified by instance 2 and is the latest one. Hence, it would request instance 2 to transfer the block to instance 1.
 
2. Since instance 1 wants to perform a WRITE operation on the block, it would acquire an EXCLUSIVE lock. This in turn means that all other instances should downgrade their locks on this block to NULL. The role of the block would be GLBOAL and the PAST IMAGE count of this block on instance 1 would be 0 as this is the first time that this instance has requested for an UPDATE operation on this block. So no PAST IMAGES exist. As a result, the mode of the block on instance 1 would be XG0.
 
3. As explained in previous step, instance 2 would have NULL lock on the block with GLBOAL role. The PAST IMAGE count would still be 1 because of the update operation it did as per scenario 4. The mode of the block 135 with respect to instance 2 would be NG1.
 
4. Similarly the mode of the block on instance 3 would be NG1.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        pi         Y 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        xcur       Y 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1

13 rows selected.

 

SYS@srprim1&gt;select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      2
11         135        pi         Y 91791      3
11         135        xcur       Y 91791      1

 
We now have the status as “PI” which is in dirty state for instance 2 (scenario 4) and instance 3 (scenario 2) each and that for instance 1, the block status is XCUR (EXCLUSIVE).
 
===========================================================================
 
Scenario 7:
 
User A on instance 1 performs another UPDATE operation on the same block.
 
SQL statement run: “update bttest.emp set name=’UPDATE11′ where code=300;
 
1. GCS knows that the last update operation performed on this block was instance 1 itself. Now, again instance 1 has requested for an EXCLUSIVE lock.
 
2. Mode of the block on instance 1 would be XG1 (The Past image count is 1 for instance 1 as it previously performed as UPDATE operation as per scenario 6).
 
3. The mode of the block with respect to instance 2 and 3 would remain as in the previous stage (NG1).
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        xcur       Y 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        pi         Y 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3

14 rows selected.

 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        pi         Y 91791      2 
11         135        xcur       Y 91791      1
11         135        pi         Y 91791      3

 
From the above result, instance 1 holds an XCUR status on the block 135 which indicates an EXCLUSIVE lock is held. While the other 2 instances (2 and 3) hold NULL lock which is why the value is “cr”.
 
===========================================================================
 
Scenario 8:
 
User B on instance 2 performs a CHECKPOINT. (This signifies that the dirty blocks in the buffer needs to be written to the disk.)
 
1. Instance 2 requests GCS for a checkpoint.
 
2. GCS forwards the request to instance 1 who held the block in EXCLUSIVE mode in scenario 7, to write the block to disk.
 
3. Instance 1 does the write operation of the block to disk and informs GCS of the completion of the operation, but would still retain it’s lock as EXCLUSIVE on the block.
 
4. GCS then informs all the instances holding PIs to discard or flush those PIs and requests the instances to change the role of the block from GLOBAL to LOCAL.
 
5. Thus, instance 1 will now have the block with mode XL0, instance 2 and instance 3 with NULL.
 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      2
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        cr         N 91791      3
11         135        xcur       N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1
11         135        cr         N 91791      1

14 rows selected.

 

SQL>select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status <> 'cr';

FILE#      BLOCK#     STATUS     D OBJD       INST_ID
---------- ---------- ---------- - ---------- ----------
11         135        xcur       N 91791      1

 
Here, one can see that the block is currently held by instance 1 in EXCLUSIVE mode while all other instances have NULL lock on this block. All the past images that instance 2 and 3 had have been flushed.
 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

Advertisements
July 25, 2016 / Shivananda Rao P

Overview of Global Data Services in oracle 12c

This article should give you an overview of the new feature in Oracle 12c – Global Services. We are aware of database services in oracle. These services provide a workload management by ensuring that clients connect to the optimal instance which is offering the service. It also serves the purpose of high availability by failing over the client connections to the other survival instances that are offering the same service. This is referred to as local data services.

 

Oracle 12c came up with a new feature of Global Data Services which extends the above feature with automated and centralized workload management across the set of replicated databases like dataguard, golden gate. In addition to this, GDS provides replication lag based workload routing, role based global services and region based global services. Note that the global services can be configured only for databases of version 12c. For databases prior 12c, only local data services can be configured.

 

One of the major question that would be raised is the difference between local and global data services – A global data service is created across a set of multiple databases, whereas local data services is created only across a set of instances of a single database. With local data services, there is more of manual intervention needed such as a local data service for a standby needs to be first created on the primary database and then on the standby database. Whereas, with GDS, it’s managed globally with a single service.

 

A global data service configuration consists of some of the major components.

 

GDS Pool: A GDS Pool is a set of databases containing replicated data in the GDS configuration that provide unique Global Data Services and are administered by different administrator.
A database can belong to only one GDS pool. All databases in a GDS pool need not provide the same global service, but all databases that provide the same global service must belong to the same GDS pool.

 

GDS Catalog: Just as we have a catalog database in RMAN which acts as a repository for the backup configuration of the registered database, likewise, a GDS Catalog is a repository that is used to store the GDS configuration. This catalog must reside in an oracle database of version 12c.
This database may reside inside or outside the GDS configuration. Please note that a catalog is associated with only one GDS configuration.

 

GSM (Global Service Manager): As the name says, GSM serves the role of a manager in managing the Global services, failover and load balancing of global services.
It acts as a middle layer between the clients and the databases, just as a remote listener does in RAC databases. In addition to this, it measures the network latency between regions of the configuration by collecting the performance metrics from the databases in the configuration, monitors the databases and global services in the GDS configuration and notifies the clients when they fail.

 

Client first connects to the GSM and requests for a connection to the global service. The GSM, then, forwards the connection request to the database instance in the GDS configuration that is offering the requested Global Service.

 

GDS Region: A GDS region is a set of databases within the GDS configuration and the clients that share a very little network latency. A region can contain multiple GDS pools.

 

You can configure, modify, start or stop global services using GDSCTL utility. To use this, you need to download and install the Oracle Global Service Manager software from the Oracle site.

 

In the coming posts, we shall see on the installation of GSM software, configuring global services for the databases, how Global Services work with Dataguard and how effective it serves when there is a replication lag.

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

July 9, 2016 / Shivananda Rao P

Upgrade OEM 12c to 13c

Here, in this article, I’m demonstrating an upgrade from cloud control 12c to 13c on a linux machine. It is required to take a look at the https://docs.oracle.com/cd/E63000_01/EMUPG/preface.htm#EMUPG101 upgrade document before proceeding further and make sure that all the pre-requisites are met.

 

Few things to consider before you proceed with the upgrade.

 

1. OMS 13c repository database needs to be on 12.1.0.2 version. So, if the repository database is on a pre-12c release, then you need to upgrade it before upgrading the Cloud control.

 

2. Cloud control 13c is not supported on OEL/RHEL 5 and so are the 13c agents. You cannot upgrade cloud control 12c on linux machine less than version 6 to 13c.

 

Environment details is as below:

 

The environment used here has the Cloud control 12c installed on OEL 6 with the repository database of version 12.1.0.2 which is a non-CDB and has the 12c agents deployed on linux machine of release 6 (OEL 6).

 

Cloud Control 12c OMS hostname: ora1-2
Repository database version: 12.1.0.2

 

Make sure that the “COMPATIBLE” parameter on the OMS respository database is set to 12.1.0.2.0.

 

SYS@omsdb> show parameter compatible

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
compatible                           string      12.1.0.2.0
noncdb_compatible                    boolean     FALSE
SYS@omsdb&gt;
SYS@omsdb> show parameter adaptive

 

If the adaptive optimizer feature (optimizer_adaptive_features) is enabled, then it needs to be disabled on the OMS repository database before upgrading it.

 

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features          boolean     TRUE
optimizer_adaptive_reporting_only    boolean     FALSE
parallel_adaptive_multi_user         boolean     TRUE
SYS@omsdb>
SYS@omsdb> alter system set optimizer_adaptive_features=false;

System altered.

SYS@omsdb> show parameter adaptive

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features          boolean     FALSE
optimizer_adaptive_reporting_only    boolean     FALSE
parallel_adaptive_multi_user         boolean     TRUE

 

If there are any invalid objects in the OMS Repository database, then those need to be validated.

 

Take a backup of the current OMS. Refer the documentation “http://docs.oracle.com/cd/E24628_01/install.121/e24089/ha_backup_recover.htm#EMADV9639&#8221; on how to backup the OMS.

 

Copy the EMKEY from the exsting OMS to the existing management repository. EMKEY is the encryption key which is used by Enterprise Manager to encrypt/decrypt sensitive data such as passwords, preferred credentials.

 

[oracle@ora1-2 ~]$ echo $OMS_HOME
/u02/oms12c/oms

[oracle@ora1-2 ~]$ $OMS_HOME/bin/emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".


[oracle@ora1-2 ~]$ $OMS_HOME/bin/emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey  is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos".
[oracle@ora1-2 ~]$

 

Stop all the components of OMS which includes WebTier, OMS and AdminServer.

 

[oracle@ora1-2 u03]$ /u02/oms12c/oms/bin/emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
[oracle@ora1-2 u03]$

 

If there is an agent configured for the OMS server, then stop that too.

 

[oracle@ora1-2 ~]$ /u02/12cagent/core/12.1.0.4.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

 

Copy all the EM13c software parts into a staging directory. I have them copied into directory “/u03/em13csoftware”.

 

[oracle@ora1-2 u03]$ ls -lrt /u03/em13csoftware | grep -i "em"
-rw-r--r--. 1 oracle oinstall 1315250699 Jun 11 15:54 em13100_linux64-2.zip
-rw-r--r--. 1 oracle oinstall 2145473190 Jun 11 16:05 em13100_linux64-3.zip
-rw-r--r--. 1 oracle oinstall 2141357330 Jun 11 16:15 em13100_linux64-4.zip
-rw-r--r--. 1 oracle oinstall  331590923 Jun 11 16:16 em13100_linux64-5.zip
-rw-r--r--. 1 oracle oinstall  841114290 Jun 11 16:20 em13100_linux64.bin

 

Set the executable permission on the file “em13100_linux64.bin” which is the executable binary file.
Execute the em13100_linux64.bin file to invoke the 13c GUI component.

 

[oracle@ora1-2 em13csoftware]$ chmod +x em13100_linux64.bin
[oracle@ora1-2 em13csoftware]$ ls -lrt em13100_linux64.bin
-rwxr-xr-x. 1 oracle oinstall 841114290 Jun 11 16:20 em13100_linux64.bin

 

[oracle@ora1-2 em13csoftware]$ ./em13100_linux64.bin
0%...............................................................100%
Launcher log file is /tmp/OraInstall2016-06-12_11-28-00AM/launcher2016-06-12_11-28-00AM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 3180.718 MHz    Passed
Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed
Checking swap space: must be greater than 512 MB.   Actual 10236 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 22122 MB    Passed


Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2016-06-12_11-28-00AM
====Prereq Config Location main===
/tmp/OraInstall2016-06-12_11-28-00AM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /tmp/OraInstall2016-06-12_11-28-00AM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2016-06-12_11-28-00AM
EMGCInstaller args -paramFile
EMGCInstaller args /tmp/sfx_Wr9ToQ/Disk1/install/linux64/oraparam.ini
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/u03/em13csoftware
EMFileLoc:/tmp/OraInstall2016-06-12_11-28-00AM/oui/em/
ScratchPathValue :/tmp/OraInstall2016-06-12_11-28-00AM

 

Provide the mail ID followed by password if you wish to receive the support updates and click NEXT.

 

1

 

I would not like to search for the software update and hence chosing the SKIP option and clicking NEXT.

 

2

 

Run the pre-requisite checks and if any discrepancy, then fix them before proceeding further.

 

3

 

Since we are upgrading to 13c EM, select the “Upgrade an existing Enterprise Manager system” option and under this select the “One-System Upgrade” by selecting the 12c EM home.

 

4

 

Enter the middleware home location. I’m installing 13c under the location “/u02/13c”.

 

5

 

Oracle automatically captures the connect description details for the repository database. Provide the SYS and SYSMAN password.
Also confirm that the Management Repository is backed up by checking the box. If you wish to stop the Enterprise Manager for post-upgradation maintenance, then check the “Disable DDMP jobs” and click NEXT button. In that case, the post-upgradation maintenance activity needs to be carried out manually. DDMP is Deferred Data Migration which is to migrate the format of the data from the previous release Enterprise Manager to the format used in 13c.

 

6

 

If there are any pre-requisite checks at OMS repository database level failed, then review the recommendations provided. Click YES, if you would like the installer to take necessary actions else NO for a manual fix.

 

7

 

Review the information on the plugins that will be upgraded and click Next.

 

8

 

Select the additional plugins that you want to deploy and click Next.

 

9

 

Fill in the WebLogic server details by specifying the password for the Weblogic User Name and also providing the OMS Instance Base Location.

 

10

 

If you want to configure and enable BI publisher, you can select it here, else click Next.

 

11

 

Review the default ports listed and click Next.

 

12

 

Review the list of information provided and click on Upgrade to begin the upgrade.

 

13

 

14

 

Run the “allroot.sh” script as ROOT user when prompted.

 

[root@ora1-2 ~]# /u02/13c/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u02/13c/root.sh ......
/etc exist
/u02/13c
Finished execution of  /u02/13c/root.sh ......

 

15

 

Review and note down the URLs

 

16

 

Login to the EM 13c through the browser with SYSMAN as the user and the password which was being used previously.

 

17

 

Accept the License Agreement to proceed further.

 

18

 

Review the OEM 13c summary page.

 

19

 

If you had disabled DDMP jobs while upgrading EM, you should consider to carry out the post upgrade tasks.

 

Login to the EM 13c through the browser. On the summary page, click SETUP. Under this, select “Manage Cloud Control” and then the “Post Upgrade Tasks” option.

 

24

 

Click the “Start” option to start the post upgrade tasks.

 

25

 

Review the status of the post upgrade tasks.

 

26

 

Upgrading a 12c agent to 13c:

 

As said previously, 13c agents can be configured on a Linux target machine whose version is not below 6. So, if you already have the 12c agent installed on a Linux machine which is less than version 6, then this cannot be considered to upgrade to 13c agent. But, you can still use the old agent (12c agent) with EM 13c.

 

For the purpose of this demo, I had a 12c agent installed on my OMS host which is of version 6. So, I’m considering to upgrade this agent to 13c.

 

Agent Host name: ora1-2
Agent Version: 12c
Agent Host Flavor: OEL 6

 

Login to the EM 13c through the browser. On the summary page, click SETUP. Under this, select “Manage Cloud Control” and then the “Upgrade Agents” option.

 

Click on the Add option under the “Agents for upgrade” option to view the list of agents that can be upgraded to 13c.

 

20

 

Select the agent you desire to upgrade and click OK.

 

21

 

Review the information and click on Submit to upgrade the agent.

 

22

 

After the agent upgrade, review the status manually.

 

12c agent under location:

 

[oracle@ora1-2 bin]$ cd /u02/12cagent/
[oracle@ora1-2 12cagent]$ ls -lrt
total 32
drwxr-xr-x.  3 oracle oinstall 4096 May 24  2014 core
drwxr-xr-x. 12 oracle oinstall 4096 Jun  7 13:24 plugins
-rw-rw-r--.  1 oracle oinstall  179 Jun 13 20:30 agentimage.properties
-rw-r--r--.  1 oracle oinstall  693 Jun 13 20:35 agentInstall.rsp
drwxr-xr-x.  3 oracle oinstall 4096 Jun 13 20:35 backup_agtup
drwxr-xr-x.  5 oracle oinstall 4096 Jun 13 20:40 sbin
drwxr-xr-x.  9 oracle oinstall 4096 Jun 13 20:49 agent_inst
drwxr-xr-x. 28 oracle oinstall 4096 Jun 13 20:50 agent_13.1.0.0.0

 

Agent upgrade installer creates a new directory called “agent_13.1.0.0.0” under the previous 12c agent directory.

 

[oracle@ora1-2 12cagent]$ cd agent_13.1.0.0.0/bin/
[oracle@ora1-2 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : 13.1.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u02/12cagent/agent_inst
Agent Log Directory    : /u02/12cagent/agent_inst/sysman/log
Agent Binaries         : /u02/12cagent/agent_13.1.0.0.0
Core JAR Location      : /u02/12cagent/agent_13.1.0.0.0/jlib
Agent Process ID       : 28044
Parent Process ID      : 27922
Agent URL              : https://ora1-2.mydomain:3872/emd/main/
Local Agent URL in NAT : https://ora1-2.mydomain:3872/emd/main/
Repository URL         : https://ora1-2.mydomain:4903/empbs/upload
Started at             : 2016-06-13 20:40:14
Started by user        : oracle
Operating System       : Linux version 2.6.32-71.el6.x86_64 (amd64)
Number of Targets      : 36
Last Reload            : 2016-06-13 20:43:33
Last successful upload                       : 2016-06-13 20:50:53
Last attempted upload                        : 2016-06-13 20:50:53
Total Megabytes of XML files uploaded so far : 0.29
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 30.18%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-06-13 20:50:38
Last successful heartbeat to OMS             : 2016-06-13 20:50:38
Next scheduled heartbeat to OMS              : 2016-06-13 20:51:39

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ora1-2 bin]$

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

June 17, 2016 / Shivananda Rao P

OEM 12c Cloud Control – Agent Silent installation using RPM

In the previous article we have seen how to install OEM agent in silent mode using a response file. In the present articl, we shall look at an alternative way of silent installation of agent using RPM.

 

Environment details:

 

OMS Hostname: ora1-2
Agent to be deployed on : ora1-5

 

Below is the detail of the OMS status. This is checked from the OMS server.

 

[oracle@ora1-2 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
SYSMAN password provided is invalid
[oracle@ora1-2 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : ora1-2.mydomain
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://ora1-2.mydomain:7802/em
Upload URL: https://ora1-2.mydomain:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : ora1-2.mydomain
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: ora1-2.mydomain
WebTier is Up
Oracle Management Server is Up

BI Publisher is not configured to run on this host.
[oracle@ora1-2 bin]$

 

Login to the “Enterprise Manager Command Line Interface” client from the OMS host with username as SYSMAN and it’s password.

 

[oracle@ora1-2 ~]$ cd /u01/app/oracle/oms12c/oms/bin
[oracle@ora1-2 bin]$ ./emcli login -username=sysman -password=Micromot10n
Login successful
[oracle@ora1-2 bin]$

 

Once logged in successfully, synchronize the EMCLI.

 

[oracle@ora1-2 bin]$ ./emcli sync
Synchronized successfully
[oracle@ora1-2 bin]$

 

Identify the list of Operating Systems for which the Agent software is available. This can be obtained by running the “get_supported_platforms” from the EMCLI interface.

 

[oracle@ora1-2 bin]$ ./emcli get_supported_platforms
-----------------------------------------------
Version = 12.1.0.4.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.
[oracle@ora1-2 bin]$

 

Get the Agent “.rpm” for the required Operating system by specifying the “platform” option and the relevant agent version by passing “version” option. This will be saved on to the OMS host at the location specifed under the “destination” option.

Here, I’m getting the “.rpm” for “Linux x86-64” platform and of version “12.1.0.4.0” as stated above and saving it to “/u03/agent_software” location on the OMS host.

As a pre-requisite, make sure that the directory “/usr/lib/oracle” exists and has write permissions on the OMS server. Else the “get_agentimage_rpm” command might fail as shown below.

 

[oracle@ora1-2 bin]$ ./emcli get_agentimage_rpm -destination=/u03/agent_software -platform="Linux x86-64" -version="12.1.0.4.0"
Platform:Linux x86-64
Destination:/u03/agent_software
Exalogic:false
 Checking for disk space requirements...
 === Partition Detail ===
Space free : 15 GB
Space required : 1 GB
Space check pereq has failed. Make sure that you have 2 GB of space on /usr/lib/oracle directory
[oracle@ora1-2 bin]$

 

[oracle@ora1-2 bin]$ su -
Password:
[root@ora1-2 ~]# mkdir -p /usr/lib/oracle
[root@ora1-2 ~]#
[root@ora1-2 ~]# chmod 777 /usr/lib/oracle

 

Retry to get the agent image rpm after creating “/usr/lib/oracle” directory with write permission.

 

[oracle@ora1-2 bin]$ ./emcli get_agentimage_rpm -destination=/u03/agent_software -platform="Linux x86-64" -version="12.1.0.4.0"
Platform:Linux x86-64
Destination:/u03/agent_software
Exalogic:false
 Checking for disk space requirements...
 === Partition Detail ===
Space free : 15 GB
Space required : 1 GB
RPM Creation in progress ...
Check the logs at /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_rpm_2016-05-12_12-29-54-PM.log
This operation may take few minutes, please wait ...
Agent image to rpm conversion completed successfully

 

Copy this downloaded AgentImage RPM on to the target server where the Agent needs to be installed. In my case, it’s host ora1-5 where the agent will be installed.

 

[oracle@ora1-2 agent_software]$ pwd
/u03/agent_software
[oracle@ora1-2 agent_software]$ scp oracle-agt-12.1.0.4.0-1.0.x86_64.rpm oracle@ora1-5:/u03/agent_software/
The authenticity of host 'ora1-5 (192.168.56.107)' can't be established.
RSA key fingerprint is cb:5b:3b:bd:51:e9:86:77:e8:f5:8f:be:59:f9:fa:73.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ora1-5,192.168.56.107' (RSA) to the list of known hosts.
oracle@ora1-5's password:
oracle-agt-12.1.0.4.0-1.0.x86_64.rpm                                                                                                 100%  224MB  11.2MB/s   00:20
[oracle@ora1-2 agent_software]$

 

On the target server, as ROOT user, install the copied agent-image rpm

 

[oracle@ora1-5 ~]$ cd /u03/agent_software/
[oracle@ora1-5 agent_software]$ ls -lrt
total 229736
-rw-r-----. 1 oracle oinstall 235009376 May 13 10:23 oracle-agt-12.1.0.4.0-1.0.x86_64.rpm
[oracle@ora1-5 agent_software]$

 

[root@ora1-5 ~]# rpm -Uivh /u03/agent_software/oracle-agt-12.1.0.4.0-1.0.x86_64.rpm
Preparing...                ########################################### [100%]
Running the prereq
   1:oracle-agt             ########################################### [100%]
Agent RPM installation is completed successfully. Now to configure the agent follow the below steps:
1. Edit the properties file: /usr/lib/oracle/agent/agent.properties with the correct values
2. Execute the script /etc/init.d/oracle-agt RESPONSE_FILE=/usr/lib/oracle/agent/agent.properties
[root@ora1-5 ~]#

 

Please note that when a “.rpm” file is used to install the agent, the default agent base directory location is “/usr/lib/oracle/agent”. So before installing the rpm, make sure the directory “/usr/lib/oracle” exists with write permissions on the target server where the agent will be installed.

If the agent needs to be installed on to a custom agent base directory location, then specify the “–relocate” option while installing the copied “.rpm” file.

 

For example:

 

rpm -Uivh --relocate /usr/lib/oracle/agent=<your_custom_agent_base_directory_location> <copied_rpm_file_with_complete_path>

rpm -Uivh --relocate /usr/lib/oracle/agent=/u01/app/oracle/agent /u03/agent_software/oracle-agt-12.1.0.4.0-1.0.x86_64.rpm

 

Also note that, when an agent is installed using rpm file, the inventory location is the AGENT_BASE_DIR/oraInventory. So, by default the inventory location will be “/usr/lib/oracle/agent/oraInventory”.

Once the agent rpm is installed successfully, we would be provided with sequence of steps to be followed to complete the cnfiguration.

 

Edit the “/usr/lib/oracle/agent/agent.properties” file with appropriate values such as OMS Hostname, OMS Port number (upload port number to communicate with OMS), Agent registration password (used to secure the agent), Agent username (username with which the agent needs to be installed), agent group (group to which the agent user belongs), agent port (optional – port where agent process will be started), hostname where you want the agent to be installed.

 

[root@ora1-5 ~]# cat /usr/lib/oracle/agent/agent.properties
#-------------------------------------------------------------------------------
#OMS_HOST:<String> OMS host info required to connect to OMS
#OMS_PORT:<String> OMS port info required to connect to OMS
#AGENT_REGISTRATION_PASSWORD:<String> Agent Registration Password needed to
#     establish a secure connection to the OMS.
#-------------------------------------------------------------------------------
OMS_HOST=ora1-2.mydomain
OMS_PORT=4903
AGENT_REGISTRATION_PASSWORD=oracle123
#-------------------------------------------------------------------------------
#AGENT_USERNAME:<String> User name with which the agent should be installed.
#AGENT_GROUP:<String> Group to which the agent user belogs.
#AGENT_PORT:<String> Port in which the agent process will come up.
#-------------------------------------------------------------------------------
AGENT_USERNAME=oracle
AGENT_GROUP=oinstall
#AGENT_PORT=3873
#-------------------------------------------------------------------------------
#ORACLE_HOSTNAME:<String> Virtual hostname where the agent is deployed.
#Example: ORACLE_HOSTNAME=hostname.domain
#-------------------------------------------------------------------------------
ORACLE_HOSTNAME=ora1-5.mydomain

 

As ROOT user using the above edited properties file, run the below command to cofigure the agent as directed after the agent rpm installation.

 

[root@ora1-5 ~]# /etc/init.d/oracle-agt RESPONSE_FILE=/usr/lib/oracle/agent/agent.properties
/usr/lib/oracle/agent/parameter.lst
Response File:/usr/lib/oracle/agent/agent.properties


Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10239 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-05-13_10-29-42AM. Please wait ...
LD_LIBRARY_PATH environment variable :
-------------------------------------------------------
Total args: 31
Command line argument array elements ...
Arg:0:/tmp/OraInstall2016-05-13_10-29-42AM/jre/bin/java:
Arg:1:-Doracle.installer.library_loc=/tmp/OraInstall2016-05-13_10-29-42AM/oui/lib/linux64:
Arg:2:-Doracle.installer.oui_loc=/tmp/OraInstall2016-05-13_10-29-42AM/oui:
Arg:3:-Doracle.installer.bootstrap=TRUE:
Arg:4:-Doracle.installer.startup_location=/usr/lib/oracle/agent/core/12.1.0.4.0/oui/bin:
Arg:5:-Doracle.installer.jre_loc=../../jre:
Arg:6:-Doracle.installer.nlsEnabled="TRUE":
Arg:7:-Doracle.installer.prereqConfigLoc= :
Arg:8:-Doracle.installer.unixVersion=2.6.32-71.el6.x86_64:
Arg:9:-mx160m:
Arg:10:-cp:
Arg:11:/tmp/OraInstall2016-05-13_10-29-42AM::/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/OraInstaller.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/oneclick.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/xmlparserv2.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/share.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/OraInstallerNet.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/emocmutl.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/emCfg.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/OraPrereq.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/jsch.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/ssh.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/remoteinterfaces.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/http_client.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/OraSuiteInstaller.jar:/tmp/OraInstall2016-05-13_10-29-42AM/OPatch/jlib/opatch.jar:/tmp/OraInstall2016-05-13_10-29-42AM/OPatch/jlib/opatchactions.jar:/tmp/OraInstall2016-05-13_10-29-42AM/OPatch/jlib/opatchprereq.jar:/tmp/OraInstall2016-05-13_10-29-42AM/OPatch/jlib/opatchutil.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstImages.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_de.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_es.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_fr.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_it.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_ja.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_ko.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_pt_BR.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_zh_CN.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/InstHelp_zh_TW.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/oracle_ice.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/help4.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/help4-nls.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/ewt3.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/ewt3-swingaccess.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/ewt3-nls.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/swingaccess.jar::/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/jewt4.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/jewt4-nls.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/orai18n-collation.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/orai18n-mapping.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/ojmisc.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/xml.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/srvm.jar:/tmp/OraInstall2016-05-13_10-29-42AM/oui/jlib/classes12.jar:
Arg:12:oracle.sysman.oii.oiic.OiicInstaller:
Arg:13:-scratchPath:
Arg:14:/tmp/OraInstall2016-05-13_10-29-42AM:
Arg:15:-sourceType:
Arg:16:network:
Arg:17:-timestamp:
Arg:18:2016-05-13_10-29-42AM:
Arg:19:-clone:
Arg:20:-forceClone:
Arg:21:-force:
Arg:22:-silent:
Arg:23:ORACLE_HOME=/usr/lib/oracle/agent/core/12.1.0.4.0:
Arg:24:ORACLE_HOME_NAME=agent12c0:
Arg:25:-waitForCompletion:
Arg:26:-debug:
Arg:27:OMS_HOST=ora1-2.mydomain:
Arg:28:EM_UPLOAD_PORT=4903:
Arg:29:AGENT_BASE_DIR=/usr/lib/oracle/agent:
Arg:30:EM_PROTOCOL=https:
-------------------------------------------------------
Initializing Java Virtual Machine from /tmp/OraInstall2016-05-13_10-29-42AM/jre/bin/java. Please wait...
Oracle Universal Installer, Version 11.1.0.12.0 Production
Copyright (C) 1999, 2014, Oracle. All rights reserved.

You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2016-05-13_10-29-42AM.log
.................................................................................................... 100% Done.



Installation in progress (Friday, May 13, 2016 10:30:06 AM IST)
...................................................              51% Done.
Install successful

Linking in progress (Friday, May 13, 2016 10:30:15 AM IST)
Link successful

Setup in progress (Friday, May 13, 2016 10:30:15 AM IST)
...............................                                 100% Done.
Setup successful

End of install phases.(Friday, May 13, 2016 10:30:29 AM IST)
The cloning of agent12c0 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2016-05-13_10-29-42AM.log' for more details.
copying /u01/app/oraInventory/logs/silentInstall2016-05-13_10-29-42AM.log to /usr/lib/oracle/agent/core/12.1.0.4.0/cfgtoollogs/oui/silentInstall2016-05-13_10-29-42AM.log
copying /u01/app/oraInventory/logs/cloneActions2016-05-13_10-29-42AM.log to /usr/lib/oracle/agent/core/12.1.0.4.0/cfgtoollogs/oui/cloneActions2016-05-13_10-29-42AM.log
copying /u01/app/oraInventory/logs/oraInstall2016-05-13_10-29-42AM.err to /usr/lib/oracle/agent/core/12.1.0.4.0/cfgtoollogs/oui/oraInstall2016-05-13_10-29-42AM.err
copying /u01/app/oraInventory/logs/oraInstall2016-05-13_10-29-42AM.out to /usr/lib/oracle/agent/core/12.1.0.4.0/cfgtoollogs/oui/oraInstall2016-05-13_10-29-42AM.out
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10239 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-05-13_10-30-30AM. Please wait ...
LD_LIBRARY_PATH environment variable :
-------------------------------------------------------
Total args: 26
Command line argument array elements ...
Arg:0:/tmp/OraInstall2016-05-13_10-30-30AM/jre/bin/java:
Arg:1:-Doracle.installer.library_loc=/tmp/OraInstall2016-05-13_10-30-30AM/oui/lib/linux64:
Arg:2:-Doracle.installer.oui_loc=/tmp/OraInstall2016-05-13_10-30-30AM/oui:
Arg:3:-Doracle.installer.bootstrap=TRUE:
Arg:4:-Doracle.installer.startup_location=/usr/lib/oracle/agent/core/12.1.0.4.0/oui/bin:
Arg:5:-Doracle.installer.jre_loc=../../jre:
Arg:6:-Doracle.installer.nlsEnabled="TRUE":
Arg:7:-Doracle.installer.prereqConfigLoc= :
Arg:8:-Doracle.installer.unixVersion=2.6.32-71.el6.x86_64:
Arg:9:-mx160m:
Arg:10:-cp:
Arg:11:/tmp/OraInstall2016-05-13_10-30-30AM::/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/OraInstaller.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/oneclick.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/xmlparserv2.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/share.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/OraInstallerNet.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/emocmutl.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/emCfg.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/OraPrereq.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/jsch.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/ssh.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/remoteinterfaces.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/http_client.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/OraSuiteInstaller.jar:/tmp/OraInstall2016-05-13_10-30-30AM/OPatch/jlib/opatch.jar:/tmp/OraInstall2016-05-13_10-30-30AM/OPatch/jlib/opatchactions.jar:/tmp/OraInstall2016-05-13_10-30-30AM/OPatch/jlib/opatchprereq.jar:/tmp/OraInstall2016-05-13_10-30-30AM/OPatch/jlib/opatchutil.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstImages.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_de.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_es.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_fr.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_it.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_ja.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_ko.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_pt_BR.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_zh_CN.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/InstHelp_zh_TW.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/oracle_ice.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/help4.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/help4-nls.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/ewt3.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/ewt3-swingaccess.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/ewt3-nls.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/swingaccess.jar::/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/jewt4.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/jewt4-nls.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/orai18n-collation.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/orai18n-mapping.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/ojmisc.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/xml.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/srvm.jar:/tmp/OraInstall2016-05-13_10-30-30AM/oui/jlib/classes12.jar:
Arg:12:oracle.sysman.oii.oiic.OiicAttachHome:
Arg:13:-scratchPath:
Arg:14:/tmp/OraInstall2016-05-13_10-30-30AM:
Arg:15:-sourceType:
Arg:16:network:
Arg:17:-timestamp:
Arg:18:2016-05-13_10-30-30AM:
Arg:19:-attachHome:
Arg:20:ORACLE_HOME=/usr/lib/oracle/agent/sbin:
Arg:21:ORACLE_HOME_NAME=sbin12c0:
Arg:22:-waitForCompletion:
Arg:23:-debug:
Arg:24:-invPtrLoc:
Arg:25:/usr/lib/oracle/agent/core/12.1.0.4.0/oraInst.loc:
-------------------------------------------------------
Initializing Java Virtual Machine from /tmp/OraInstall2016-05-13_10-30-30AM/jre/bin/java. Please wait...
The inventory pointer is located at /usr/lib/oracle/agent/core/12.1.0.4.0/oraInst.loc
'AttachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10239 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-05-13_10-30-34AM. Please wait ...
LD_LIBRARY_PATH environment variable :
-------------------------------------------------------
Total args: 25
Command line argument array elements ...
Arg:0:/tmp/OraInstall2016-05-13_10-30-34AM/jre/bin/java:
Arg:1:-Doracle.installer.library_loc=/tmp/OraInstall2016-05-13_10-30-34AM/oui/lib/linux64:
Arg:2:-Doracle.installer.oui_loc=/tmp/OraInstall2016-05-13_10-30-34AM/oui:
Arg:3:-Doracle.installer.bootstrap=TRUE:
Arg:4:-Doracle.installer.startup_location=/usr/lib/oracle/agent/core/12.1.0.4.0/oui/bin:
Arg:5:-Doracle.installer.jre_loc=../../jre:
Arg:6:-Doracle.installer.nlsEnabled="TRUE":
Arg:7:-Doracle.installer.prereqConfigLoc= :
Arg:8:-Doracle.installer.unixVersion=2.6.32-71.el6.x86_64:
Arg:9:-mx160m:
Arg:10:-cp:
Arg:11:/tmp/OraInstall2016-05-13_10-30-34AM::/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/OraInstaller.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/oneclick.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/xmlparserv2.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/share.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/OraInstallerNet.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/emocmutl.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/emCfg.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/OraPrereq.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/jsch.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/ssh.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/remoteinterfaces.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/http_client.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/OraSuiteInstaller.jar:/tmp/OraInstall2016-05-13_10-30-34AM/OPatch/jlib/opatch.jar:/tmp/OraInstall2016-05-13_10-30-34AM/OPatch/jlib/opatchactions.jar:/tmp/OraInstall2016-05-13_10-30-34AM/OPatch/jlib/opatchprereq.jar:/tmp/OraInstall2016-05-13_10-30-34AM/OPatch/jlib/opatchutil.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstImages.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_de.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_es.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_fr.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_it.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_ja.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_ko.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_pt_BR.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_zh_CN.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/InstHelp_zh_TW.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/oracle_ice.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/help4.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/help4-nls.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/ewt3.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/ewt3-swingaccess.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/ewt3-nls.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/swingaccess.jar::/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/jewt4.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/jewt4-nls.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/orai18n-collation.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/orai18n-mapping.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/ojmisc.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/xml.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/srvm.jar:/tmp/OraInstall2016-05-13_10-30-34AM/oui/jlib/classes12.jar:
Arg:12:oracle.sysman.oii.oiic.OiicUpdateHomeDeps:
Arg:13:-scratchPath:
Arg:14:/tmp/OraInstall2016-05-13_10-30-34AM:
Arg:15:-sourceType:
Arg:16:network:
Arg:17:-timestamp:
Arg:18:2016-05-13_10-30-34AM:
Arg:19:-updateHomeDeps:
Arg:20:HOME_DEPENDENCY_LIST={/usr/lib/oracle/agent/sbin:/usr/lib/oracle/agent/core/12.1.0.4.0}:
Arg:21:-waitForCompletion:
Arg:22:-debug:
Arg:23:-invPtrLoc:
Arg:24:/usr/lib/oracle/agent/core/12.1.0.4.0/oraInst.loc:
-------------------------------------------------------
Initializing Java Virtual Machine from /tmp/OraInstall2016-05-13_10-30-34AM/jre/bin/java. Please wait...
The inventory pointer is located at /usr/lib/oracle/agent/core/12.1.0.4.0/oraInst.loc
'UpdateHomeDeps' was successful.
Finished product-specific root actions.
/etc exist

Creating /etc/oragchomelist file...
/usr/lib/oracle/agent/core/12.1.0.4.0
Setting the invPtrLoc to /usr/lib/oracle/agent/core/12.1.0.4.0/oraInst.loc

perform - mode is starting for action: configure

** Agent Port Check completed successfully.**

perform - mode finished for action: configure

You can see the log file: /usr/lib/oracle/agent/core/12.1.0.4.0/cfgtoollogs/oui/configActions2016-05-13_10-30-39-AM.log
Agent Configuration completed successfully
[root@ora1-5 ~]#

 

Once the configuration is a success, check the agent status.

 

[oracle@ora1-5 ~]$
[oracle@ora1-5 ~]$ cd /usr/lib/oracle/agent/agent_inst/bin
[oracle@ora1-5 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /usr/lib/oracle/agent/agent_inst
Agent Log Directory    : /usr/lib/oracle/agent/agent_inst/sysman/log
Agent Binaries         : /usr/lib/oracle/agent/core/12.1.0.4.0
Agent Process ID       : 20280
Parent Process ID      : 20238
Agent URL              : https://ora1-5.mydomain:3872/emd/main/
Local Agent URL in NAT : https://ora1-5.mydomain:3872/emd/main/
Repository URL         : https://ora1-2.mydomain:4903/empbs/upload
Started at             : 2016-05-13 10:32:33
Started by user        : oracle
Operating System       : Linux version 2.6.32-71.el6.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : 2016-05-13 10:34:25
Last attempted upload                        : 2016-05-13 10:34:33
Total Megabytes of XML files uploaded so far : 0.33
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 86.76%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-05-13 10:35:44
Last successful heartbeat to OMS             : 2016-05-13 10:35:44
Next scheduled heartbeat to OMS              : 2016-05-13 10:36:44

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ora1-5 bin]$

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

May 10, 2016 / Shivananda Rao P

OEM 12c Cloud Control – Agent Silent installation using Response File

We have come across OEM agent installation from the Cloud Control tool. This article demonstrates on how to install 12c cloud control agent uisng “silent” mode which makes use of the “response” file.

 

Environment details is as follows:

 

OMS hostname              : ora1-2
Agent will be deployed on : ora1-1

 

Below is the details of the OMS status. This is checked from the OMS server.

 

[oracle@ora1-2 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : ora1-2.mydomain
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://ora1-2.mydomain:7802/em
Upload URL: https://ora1-2.mydomain:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : ora1-2.mydomain
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: ora1-2.mydomain
WebTier is Up
Oracle Management Server is Up

BI Publisher is not configured to run on this host.
[oracle@ora1-2 bin]$

 

Login to the “Enterprise Manager Command Line Interface” client from the OMS host with username as SYSMAN and it’s password.

 

[oracle@ora1-2 bin]$ pwd
/u01/app/oracle/oms12c/oms/bin
[oracle@ora1-2 bin]$
[oracle@ora1-2 bin]$ ./emcli login -username=sysman -password=Micromot10n
Login successful

 

Once logged in successfully, synchronize the EMCLI.

 

[oracle@ora1-2 bin]$ ./emcli sync
Synchronized successfully
[oracle@ora1-2 bin]$

 

Identify the list of Operating Systems for which the Agent software is available. This can be obtained by running the “get_supported_platforms” from the EMCLI interface.

 

[oracle@ora1-2 bin]$ ./emcli get_supported_platforms
-----------------------------------------------
Version = 12.1.0.4.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

 

Get the Agent software for the required Operating system by specifying the “platform” option and the relevant agent version by passing “version” option.
The image will be saved on to the OMS host at the location specifed under the “destination” option. Here, I’m getting the AgentImage for “Linux x86-64” platform and of version “12.1.0.4.0” as stated above and saving the AgentImage to “/u03/agent_software” location on the OMS host.

 

[oracle@ora1-2 bin]$ ./emcli get_agentimage -destination=/u03/agent_software -platform="Linux x86-64" -version="12.1.0.4.0"
 === Partition Detail ===
Space free : 15 GB
Space required : 1 GB
Check the logs at /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2016-04-08_18-49-33-PM.log
Downloading /u03/agent_software/12.1.0.4.0_AgentCore_226.zip
File saved as /u03/agent_software/12.1.0.4.0_AgentCore_226.zip
Downloading /u03/agent_software/12.1.0.4.0_PluginsOneoffs_226.zip
File saved as /u03/agent_software/12.1.0.4.0_PluginsOneoffs_226.zip
Downloading /u03/agent_software/unzip
File saved as /u03/agent_software/unzip
Agent Image Download completed successfully.
[oracle@ora1-2 bin]$

 

Copy this downloaded AgentImage on to the target server where the Agent needs to be installed. In my case, it’s host ora1-1 where the agent will be installed.

 

[oracle@ora1-2 bin]$ cd /u03/agent_software
[oracle@ora1-2 agent_software]$ ls -lrt
total 250000
-rw-r--r--. 1 oracle oinstall 255744050 Apr  8 18:52 12.1.0.4.0_AgentCore_226.zip
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$
[oracle@ora1-2 agent_software]$ scp 12.1.0.4.0_AgentCore_226.zip oracle@ora1-1:/u03/
oracle@192.168.56.101's password:
12.1.0.4.0_AgentCore_226.zip                                                                                                         100%  244MB   9.0MB/s   00:27
[oracle@ora1-2 agent_software]$

 

[oracle@ora1-1 ~]$ cd /u03
[oracle@ora1-1 u03]$ ls -lrt
total 1196388
drwx------. 2 oracle oinstall     16384 Jan  1 17:25 lost+found
-rw-r--r--. 1 oracle oinstall 915338678 Mar 27 11:25 p19954978_121020_Linux-x86-64.zip
drwxr-xr-x. 3 oracle oinstall      4096 Mar 27 11:43 121022_gi_psu
-rw-r--r--. 1 oracle oinstall  52774823 Mar 27 13:38 p6880880_121010_Linux-x86-64.zip
-rw-r--r--. 1 oracle oinstall 255744050 Apr  8 19:11 12.1.0.4.0_AgentCore_226.zip
[oracle@ora1-1 u03]$

 

On the target server, unzip the AgentImage software received from the OMS host.

 

[oracle@ora1-1 u03]$
[oracle@ora1-1 u03]$ mkdir 12104_agent
[oracle@ora1-1 u03]$
[oracle@ora1-1 u03]$ unzip -d 12104_agent 12.1.0.4.0_AgentCore_226.zip
Archive:  12.1.0.4.0_AgentCore_226.zip
  inflating: 12104_agent/unzip
  inflating: 12104_agent/agentDeploy.sh
  inflating: 12104_agent/agentimage.properties
  inflating: 12104_agent/agent.rsp
 extracting: 12104_agent/agentcoreimage.zip
 extracting: 12104_agent/12.1.0.4.0_PluginsOneoffs_226.zip
[oracle@ora1-1 u03]$

 

Edit the “agent.rsp” (AgentResponseFile) to add the details such as OMS Hostname, EM Upload Port (the HTTPS port number of OMS), Agent Registration Password (used to secure the agent), Agent Instance Home location, Agent Port number (port where Agent process will be started),b_startAgent (True if the agent needs to be started automatically after installed and configured, ORACLE_HOSTNAME (host name of the target server where the agent will be installed), s_agentHomeName (name for the Agent Home).

 

Below are the details suiting my environment.

 

OMS_HOST=ora1-2.mydomain
EM_UPLOAD_PORT=4903
AGENT_REGISTRATION_PASSWORD=oracle123
AGENT_INSTANCE_HOME=
AGENT_PORT=3873
b_startAgent=true
ORACLE_HOSTNAME=ora1-1.mydomain
s_agentHomeName=agent12cr1

 

[oracle@ora1-1 12104_agent]$ cat agent.rsp
<output trimmed>

OMS_HOST=ora1-2.mydomain
EM_UPLOAD_PORT=4903
AGENT_REGISTRATION_PASSWORD=oracle123
AGENT_INSTANCE_HOME=
AGENT_PORT=3873
b_startAgent=true
ORACLE_HOSTNAME=ora1-1.mydomain
s_agentHomeName=agent12cr1


<output trimmed>

 

[oracle@ora1-1 oracle]$ cd /u03/12104_agent/
[oracle@ora1-1 12104_agent]$ ls -lrt
total 250112
-rw-rw-r--. 1 oracle oinstall       178 May 24  2014 agentimage.properties
-rwxr-xr-x. 1 oracle oinstall    145976 May 24  2014 unzip
-rwxrwxr-x. 1 oracle oinstall     27905 May 24  2014 agentDeploy.sh
-rw-rw-r--. 1 oracle oinstall 248704435 May 24  2014 agentcoreimage.zip
-rw-r--r--. 1 oracle oinstall   6958959 Apr  8 18:52 12.1.0.4.0_PluginsOneoffs_226.zip
-rwxrwxr-x. 1 oracle oinstall      3726 Apr  8 19:26 agent.rsp

 

[oracle@ora1-1 12104_agent]$ ./agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/agent12c RESPONSE_FILE=/u03/12104_agent/agent.rsp

Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u01/app/oracle/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /u01/app/oracle/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/agent12c/core/12.1.0.4.0 /u03/12104_agent /u01/app/oracle/agent12c -prereq

Validating oms host & port with url: http://ora1-2.mydomain:4903/empbs/genwallet
Validating oms host & port with url: https://ora1-2.mydomain:4903/empbs/genwallet
Return status:3-oms https port is passed
Unzipping the agentcoreimage.zip to /u01/app/oracle/agent12c ....
12.1.0.4.0_PluginsOneoffs_226.zip
Executing command : /u03/12104_agent/unzip -o /u03/12104_agent/12.1.0.4.0_PluginsOneoffs_226.zip -d /u01/app/oracle/agent12c
Executing command : /u01/app/oracle/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/xmlparserv2.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/srvm.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/emCfg.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/u01/app/oracle/agent12c/core/12.1.0.4.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/agent12c/core/12.1.0.4.0 /u03/12104_agent /u01/app/oracle/agent12c /u01/app/oracle/agent12c/agent_inst AGENT_BASE_DIR=/u01/app/oracle/agent12c


Executing agent install prereqs...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion  -prereqlogloc /u01/app/oracle/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/u01/app/oracle/agent12c/core/12.1.0.4.0/prereqs  -J-DORACLE_HOSTNAME=ora1-1.mydomain -J-DAGENT_PORT=3873 -J-DAGENT_BASE_DIR=/u01/app/oracle/agent12c

Agent install prereqs completed successfully

Cloning the agent home...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/agent12c/core/12.1.0.4.0 -responseFile /u03/12104_agent/agent.rsp  AGENT_BASE_DIR=/u01/app/oracle/agent12c AGENT_BASE_DIR=/u01/app/oracle/agent12c RESPONSE_FILE=/u03/12104_agent/agent.rsp -noconfig  ORACLE_HOME_NAME=agent12cr1 -force b_noUpgrade=true

Cloning of agent home completed successfully

Attaching sbin home...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -attachHome -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/agent12c/sbin ORACLE_HOME_NAME=sbin12c1 -force

Attach home for sbin home completed successfully.

Updating home dependencies...
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/u01/app/oracle/agent12c/sbin:/u01/app/oracle/agent12c/core/12.1.0.4.0} -invPtrLoc /u01/app/oracle/agent12c/core/12.1.0.4.0/oraInst.loc -force

Update home dependency completed successfully.
Executing command: /u01/app/oracle/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/u01/app/oracle/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true

ERROR: Agent Configuration Failed SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1

Agent Deploy Log Location:/u01/app/oracle/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy/agentDeploy_2016-04-09_12-34-49-PM.log

 

The Agent configuration failed with error “ERROR: Agent Configuration Failed SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1SEVERE:emctl secure agent command has failed with status=1”. This error signifies that the configuration failed while securing the agent.

Does that mean that the agent is not installed ? Let’s check the status and try securing the agent manually.

 

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

 

The agent is installed but seems to be not running. Expected.
Not sure why the “agentdeploy” script failed securing the agent even though the Agent_Registration_Password is right. Let me try securing it manually with the same password.

 

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl secure agent oracle123
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Securing agent...   Successful.
[oracle@ora1-1 12104_agent]$

 

Ok, good to see that it’s success 🙂
Let me check the status of the agent.

 

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

 

Agent seems to be not running. Again as expected. Let me start the agent and add the targets manually using the “addinternaltargets” option at the EMCTL prompt.

 

[oracle@ora1-1 12104_agent]$
[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting agent ........... started.

 

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

 

Check the status of the agent.

 

[oracle@ora1-1 12104_agent]$ /u01/app/oracle/agent12c/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent12c/
Agent Log Directory    : /u01/app/oracle/agent12c//sysman/log
Agent Binaries         : /u01/app/oracle/agent12c/core/12.1.0.4.0
Agent Process ID       : 6703
Parent Process ID      : 6610
Agent URL              : https://ora1-1.mydomain:3873/emd/main/
Local Agent URL in NAT : https://ora1-1.mydomain:3873/emd/main/
Repository URL         : https://ora1-2.mydomain:4903/empbs/upload
Started at             : 2016-04-09 12:42:25
Started by user        : oracle
Operating System       : Linux version 2.6.32-71.el6.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : 2016-04-09 12:56:26
Last attempted upload                        : 2016-04-09 12:56:26
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 33
Size of XML files pending upload(MB)         : 0.02
Available disk space on upload filesystem    : 41.15%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-04-09 12:55:47
Last successful heartbeat to OMS             : 2016-04-09 12:55:47
Next scheduled heartbeat to OMS              : 2016-04-09 12:56:47

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ora1-1 12104_agent]$

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine

May 1, 2016 / Shivananda Rao P

OEM 12c (Cloud Control) installation on Linux

In this article, I’d like to demonstrate how to installing OEM 12c Release 4 on a linux machine. In the upcoming posts, I’d demonstrate on upgrading this OEM 12c to 13c.

Environment:

OS                          : OEL 6
OEM 12c Software            : 12c Release 4 (12.1.0.4)
Repository Database version : 11.2.0.3
Repositry Database Name     : omsdb
Hostname                    : ora1-2

 

This article assumes that oracle 11.2.0.3 binary is pre installed on the server and that the repository database “omsdb” is created on it. Do not have the Enterprise Manager DB control configured for this database. If already configured, then have it removed using the “emca” utility with “-deconfig” option.
Download the OEM 12c Release 4 software from here http://www.oracle.com/technetwork/oem/grid-control/downloads/index.html The software consists of 3 disks.
Unzip all the 3 disks into a single directory. Here, I have unzipped them into “/u03/oem12c” directory.

 

 

[oracle@ora1-2 ~]$ cd /u03/
[oracle@ora1-2 u03]$ ls -lrt
total 6647416
drwxrwxrwx.  2 oracle oinstall      16384 Feb 26 16:14 lost+found
-rw-r--r--.  1 oracle oinstall 2195693096 Mar 25 14:22 em12104_linux64_disk1.zip
-rw-r--r--.  1 oracle oinstall 1877449643 Mar 25 14:26 em12104_linux64_disk2.zip
-rw-r--r--.  1 oracle oinstall 2727123784 Mar 25 14:31 em12104_linux64_disk3.zip
drwxr-xr-x. 11 oracle oinstall       4096 Mar 25 14:46 oem12c
[oracle@ora1-2 u03]$ cd oem12c
[oracle@ora1-2 oem12c]$ ls -lrt
total 1496020
-rwxr-xr-x. 1 oracle oinstall       5375 Dec 26  2013 runInstaller
drwxr-xr-x. 4 oracle oinstall       4096 Jan 15  2014 bipruntime
-rwxr-xr-x. 1 oracle oinstall 1530333315 May 24  2014 WT.zip
drwxr-xr-x. 4 oracle oinstall       4096 May 24  2014 oms
drwxr-xr-x. 4 oracle oinstall       4096 May 24  2014 jdk
drwxrwxr-x. 2 oracle oinstall       4096 May 24  2014 response
drwxrwxr-x. 2 oracle oinstall       4096 May 24  2014 wls
drwxr-xr-x. 7 oracle oinstall       4096 May 24  2014 install
drwxrwxr-x. 4 oracle oinstall       4096 May 24  2014 libskgxn
drwxr-xr-x. 9 oracle oinstall       4096 May 24  2014 stage
drwxr-xr-x. 2 oracle oinstall       4096 May 24  2014 plugins
-rw-r--r--. 1 oracle oinstall      42623 May 26  2014 release_notes.pdf

 

Create the required directories for the OMS and the agent installation.

 

[oracle@ora1-2 ~]$ cd /u01/app/oracle/
[oracle@ora1-2 oracle]$ ls -lrt
total 28
drwxr-xr-x.  2 oracle oinstall 4096 Mar 25 13:32 Clusterware
drwxr-xr-x.  3 oracle oinstall 4096 Mar 25 13:32 ora1-2
drwxr-xr-x.  3 oracle oinstall 4096 Mar 25 13:40 product
drwxrwxr-x. 11 oracle oinstall 4096 Mar 25 13:54 diag
drwxr-x---.  4 oracle oinstall 4096 Mar 25 13:57 admin
drwxr-xr-x.  7 oracle oinstall 4096 Mar 25 14:07 cfgtoollogs
drwxr-xr-x.  2 oracle oinstall 4096 Mar 25 14:12 checkpoints
[oracle@ora1-2 oracle]$
[oracle@ora1-2 oracle]$

[oracle@ora1-2 oracle]$ mkdir oms12c
[oracle@ora1-2 oracle]$ mkdir 12cagent

 

Start the installation by running the “runInstaller” script.

 

[oracle@ora1-2 oem12c]$ ./runInstaller

 

I wish to not receive any updates as this is my own test machine and hence I un-check the “I wish to receive security updates via My Oracle Support”

1

 

Select the “Skip” option.

 

2

 

Run the “Prerequisite” checks in the next step. Look out for the Status section to know if the respective check Succeeded or Failed.

 

3

 

Select the “Create a new Enterprise Manager System” option with “Simple”. You can choose the Advanced option for more options.

 

4

 

Provide the Middleware Home Location (OMS Home), Agent Base Directory Location and the Host names on which it has to be installed.

 

5

 

Provide the configuration details such as “Administrative Password”, OMS repository database details such as host name of the OMS repository database, the Port number on which the listener on this host servicing OMS repository database is listening to and the SYS password.

 

You can opt the “Configure Oracle Software Library” if planning to have a multi-OMS setup, else you may skip this.

6

 

There would be some pre-requisite checks performed on the repository database and would be reported if any violations. If there are any other recommendations with changing of the repository database parameters, then necessary actions need to be taken upon accordingly.

 

7

 

Review the summary.

 

8

 

If all perfect, then proceed with the Install option.

 

9

 

Run the script “allroot.sh” as ROOT user when prompted.

 

10

 

[root@ora1-2 ~]# /u01/app/oracle/oms12c/oms/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/oms12c/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/oms12c/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist

Creating /etc/oragchomelist file...
/u01/app/oracle/oms12c/oms
Finished execution of  /u01/app/oracle/oms12c/oms/root.sh ......

Starting to execute /u01/app/oracle/12cagent/core/12.1.0.4.0/root.sh ......
Finished product-specific root actions.
/etc exist
Finished execution of  /u01/app/oracle/12cagent/core/12.1.0.4.0/root.sh ......
[root@ora1-2 ~]#

 

Review the final page consisting of the details of accessing the OEM 12c.

 

11

 

Login to the OEM 12c using the URL mentioned in the above page.

 

12

 

Accept the License Agreement.

 

14

 

The selected homepage (“Summary” page in my case) will be shown up.

 

15

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.

April 5, 2016 / Shivananda Rao P

RAC Node addition in Oracle 12c

This article demonstrates on how to add a node to a RAC cluster in oracle 12c. The environment used here makes use of ASM as a storage area for the OCR/Voting disks and database physical files.
Environment:

OS              : OEL 6
Existing nodes  : 12cnode1, 12cnode2
Node to be added: 12cnode3
Database Name   : srprim
DB Version      : 12.1.0.2

It is assumed that the new node to be added is configured to access the ASM disks that are shared between the existing nodes. OS installation, oracle or grid user creation and network configuraiton steps are not outlined in this document.

Lets move on with the node addition.

Add Public and private interfaces IP (eth0 and eth1)on the new node. I have assigned the IPs as follows

Public Interface    : 192.168.0.120
Private interface   : 192.168.1.109
Virtual IP interface: 192.168.0.121

Update the the above IP addresses of the new node in the “/etc/hosts” file on all the nodes of the cluster.
Sample of “/etc/hosts” file on the new node. The same entries exists on the other nodes of the cluster.

[root@12cnode3 u02]# cat /etc/hosts

127.0.0.1       localhost.localdomain   localhost
::1           	localhost6.localdomain6 localhost6
################ PUBLIC ###################

192.168.0.115   12cnode1.mydomain       12cnode1
192.168.0.116   12cnode2.mydomain       12cnode2
192.168.0.120   12cnode3.mydomain       12cnode3

############### PRIVATE ###################

192.168.1.107   12cnode1-priv.mydomain  12cnode1-priv
192.168.1.108   12cnode2-priv.mydomain  12cnode2-priv
192.168.1.109   12cnode3-priv.mydomain  12cnode3-priv

############### VIP #######################

192.168.0.117   12cnode1-vip.mydomain   12cnode1-vip
192.168.0.118   12cnode2-vip.mydomain   12cnode2-vip
192.168.0.121   12cnode3-vip.mydomain   12cnode3-vip

############## SCAN #######################

192.168.0.119   node12c-scan.mydomain   node12c-scan

#################################################################

As a pre-requisite check, install the “cvuqdisk-1”, “nfs-utils” rpm on the new node.

[root@12cnode3 u02]# rpm -Uivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]

[root@12cnode3 u02]# rpm -Uivh nfs-utils-1.2.3-15.el6.x86_64.rpm --force --nodeps
warning: nfs-utils-1.2.3-15.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY
Preparing...                ########################################### [100%]
   1:nfs-utils              ########################################### [100%]
[root@12cnode3 u02]#

Update the /etc/sysctl.conf file on the new node with the required kernel parameters. The values can be obtained from the existing nodes.

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Update /etc/security/limits.conf file on the new node. The values can be obtained from the existing nodes.

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    2047
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768

oracle soft memlock unlimited
oracle hard memlock unlimited

Disable the firewall between the nodes.

[root@12cnode3 u02]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
[root@12cnode3 u02]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@12cnode3 u02]#
[root@12cnode3 u02]# chkconfig iptables off
[root@12cnode3 u02]# service iptables status
iptables: Firewall is not running.

Check if avahi-daemon is running and if so, then stop it.

[root@12cnode3 u02]# chkconfig --list | grep avahi
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off
[root@12cnode3 u02]#
[root@12cnode3 u02]# service avahi-daemon stop
Shutting down Avahi daemon:                                [  OK  ]
[root@12cnode3 u02]# chkconfig avahi-daemon off
[root@12cnode3 u02]# chkconfig --list | grep avahi
avahi-daemon    0:off   1:off   2:off   3:off   4:off   5:off   6:off

Set parameter NOZEROCONF to YES in the /etc/sysconfig/network file.

[root@12cnode3 u02]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=12cnode3.mydomain
NOZEROCONF=yes

I have my ASM disks configured using UDEV. Configure UDEV rules as defined on the other nodes of the cluster.
This step can be skipped if UDEV is not being used to configure the disks.

[root@12cnode3 dev]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB1cd37fc0-cc66bbc6", NAME="DSK1", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBc62708e8-ee54011f", NAME="DSK2", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB668ac5ba-619c34bb", NAME="DSK3", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL =="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB1dfcc4bc-fc73fb86", NAME="DSK4", OWNER="oracle", GROUP="oinstall", MODE="0660"

Create passwordless ssh between the existing nodes of the cluster and with the new node. This can be done by running the “sshUserSetup.sh” script available with the GI software.

[root@12cnode3 ~]# cd /u03
[root@12cnode3 u03]# ls -lrt
total 52
drwxrwxrwx. 2 oracle oinstall 16384 Mar  7 19:26 lost+found
-rwxr-xr-x. 1 oracle oinstall 32334 Mar 16 20:10 sshUserSetup.sh
[root@12cnode3 u03]#
[root@12cnode3 u03]#
[root@12cnode3 u03]#
[root@12cnode3 u03]# ./sshUserSetup.sh -user oracle -hosts "12cnode1 12cnode2 12cnode3" -noPromptPassphrase -confirm -advanced
The output of this script is also logged into /tmp/sshUserSetup_2016-03-16-20-13-42.log
Hosts are 12cnode1 12cnode2 12cnode3
user is oracle
Platform:- Linux
Checking if the remote hosts are reachable
PING 12cnode1.mydomain (192.168.0.115) 56(84) bytes of data.
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=1 ttl=64 time=1.33 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=2 ttl=64 time=0.338 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=3 ttl=64 time=0.237 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=4 ttl=64 time=0.360 ms
64 bytes from 12cnode1.mydomain (192.168.0.115): icmp_seq=5 ttl=64 time=0.243 ms

--- 12cnode1.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.237/0.502/1.332/0.417 ms
PING 12cnode2.mydomain (192.168.0.116) 56(84) bytes of data.
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=1 ttl=64 time=0.787 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=2 ttl=64 time=0.250 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=3 ttl=64 time=0.267 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=4 ttl=64 time=0.260 ms
64 bytes from 12cnode2.mydomain (192.168.0.116): icmp_seq=5 ttl=64 time=0.229 ms

--- 12cnode2.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.229/0.358/0.787/0.215 ms
PING 12cnode3.mydomain (192.168.56.113) 56(84) bytes of data.
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=2 ttl=64 time=0.018 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=3 ttl=64 time=0.068 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=4 ttl=64 time=0.058 ms
64 bytes from 12cnode3.mydomain (192.168.56.113): icmp_seq=5 ttl=64 time=0.069 ms

--- 12cnode3.mydomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.010/0.044/0.069/0.026 ms
Remote host reachability check succeeded.
The following hosts are reachable: 12cnode1 12cnode2 12cnode3.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost 12cnode1
numhosts 3
The script will setup SSH connectivity from the host 12cnode3.mydomain to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host 12cnode3.mydomain
and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.

NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?
Confirmation provided on the command line

The user chose yes
User chose to skip passphrase related questions.
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /root/.ssh/config, it would be backed up to /root/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host with empty passphrase
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a4:83:da:75:8e:99:a2:57:63:a7:2b:b0:e2:df:4f:09 root@12cnode3.mydomain
The key's randomart image is:
+--[ RSA 1024]----+
|                 |
|                 |
|        .        |
|     . o         |
|    .E+ S        |
|  .o .=Bo        |
|  .ooo+*.        |
|. ..+.o          |
|.ooo ooo         |
+-----------------+
Creating .ssh directory and setting permissions on remote host 12cnode1
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode1. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode1.
Warning: Permanently added '12cnode1,192.168.0.115' (RSA) to the list of known hosts.
oracle@12cnode1's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode1.
Creating .ssh directory and setting permissions on remote host 12cnode2
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode2. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode2.
Warning: Permanently added '12cnode2,192.168.0.116' (RSA) to the list of known hosts.
oracle@12cnode2's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode2.
Creating .ssh directory and setting permissions on remote host 12cnode3
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host 12cnode3. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host 12cnode3.
Warning: Permanently added '12cnode3' (RSA) to the list of known hosts.
oracle@12cnode3's password:
Done with creating .ssh directory and setting permissions on remote host 12cnode3.
Copying local host public key to the remote host 12cnode1
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode1.
oracle@12cnode1's password:
Permission denied, please try again.
oracle@12cnode1's password:
Done copying local host public key to the remote host 12cnode1
Copying local host public key to the remote host 12cnode2
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode2.
oracle@12cnode2's password:
Done copying local host public key to the remote host 12cnode2
Copying local host public key to the remote host 12cnode3
The user may be prompted for a password or passphrase here since the script would be using SCP for host 12cnode3.
oracle@12cnode3's password:
Done copying local host public key to the remote host 12cnode3
Creating keys on remote host 12cnode1 if they do not exist already. This is required to setup SSH on host 12cnode1.

Creating keys on remote host 12cnode2 if they do not exist already. This is required to setup SSH on host 12cnode2.

Creating keys on remote host 12cnode3 if they do not exist already. This is required to setup SSH on host 12cnode3.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
de:87:40:a9:83:7d:92:22:7e:01:cf:95:4d:25:7b:52 oracle@12cnode3.mydomain
The key's randomart image is:
+--[ RSA 1024]----+
|        o.E      |
|       + =       |
|  .   o * .      |
|   + + + o       |
|  . * * S        |
| . . o = o .     |
|  . .   . o .    |
|   .       .     |
|                 |
+-----------------+
Updating authorized_keys file on remote host 12cnode1
Updating known_hosts file on remote host 12cnode1
Updating authorized_keys file on remote host 12cnode2
Updating known_hosts file on remote host 12cnode2
Updating authorized_keys file on remote host 12cnode3
Updating known_hosts file on remote host 12cnode3
SSH setup is complete.

------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user oracle.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--12cnode1:--
Running /usr/bin/ssh -x -l oracle 12cnode1 date to verify SSH connectivity has been setup from local host to 12cnode1.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:26 IST 2016
------------------------------------------------------------------------
--12cnode2:--
Running /usr/bin/ssh -x -l oracle 12cnode2 date to verify SSH connectivity has been setup from local host to 12cnode2.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:26 IST 2016
------------------------------------------------------------------------
--12cnode3:--
Running /usr/bin/ssh -x -l oracle 12cnode3 date to verify SSH connectivity has been setup from local host to 12cnode3.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Wed Mar 16 20:14:27 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode1
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Wed Mar 16 20:14:28 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode2
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Wed Mar 16 20:14:27 IST 2016
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from 12cnode1 to 12cnode3
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Warning: Permanently added the RSA host key for IP address '192.168.0.120' to the list of known hosts.
Wed Mar 16 20:14:28 IST 2016
------------------------------------------------------------------------
-Verification from complete-
SSH verification complete.
[root@12cnode3 u03]#

Though the script says that verification of passwordless ssh is verified, I would like to check this out with a small test.

[oracle@12cnode1 ~]$ ssh 12cnode2 date
Wed Mar 16 20:15:53 IST 2016
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ ssh 12cnode3 date
Wed Mar 16 20:15:59 IST 2016
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ ssh 12cnode1 date
Wed Mar 16 20:16:05 IST 2016
[oracle@12cnode1 ~]$
[oracle@12cnode2 ~]$ ssh 12cnode1 date
Wed Mar 16 20:16:15 IST 2016
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$ ssh 12cnode3 date
Warning: Permanently added the RSA host key for IP address '192.168.0.120' to the list of known hosts.
Wed Mar 16 20:16:19 IST 2016
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$
[oracle@12cnode2 ~]$ ssh 12cnode2 date
Wed Mar 16 20:16:25 IST 2016
[oracle@12cnode3 dev]$ ssh 12cnode1 date
Wed Mar 16 20:16:37 IST 2016
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$ ssh 12cnode2 date
Wed Mar 16 20:16:42 IST 2016
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$
[oracle@12cnode3 dev]$ ssh 12cnode3 date
Wed Mar 16 20:16:48 IST 2016

Run the CLUVFY utility with the options “pre nodeadd” to check the pre-requisites on the new node. This needs to be run from the node which is already part of the cluster.
Here, it’s being run from node “12cnode1”.

[oracle@12cnode1 bin]$ cd /u01/app/12.1.0.2/grid/bin
[oracle@12cnode1 bin]$ ./cluvfy stage -pre nodeadd -n 12cnode3 -verbose > /u02/12cnode3prenodeadd.txt
[oracle@12cnode1 bin]$

Let me check what all conditions are not met.

[oracle@12cnode1 bin]$ grep -i "failed" /u02/12cnode3prenodeadd.txt
  12cnode3      2.6.32-71.el6.x86_64      2.6.39                    failed
  12cnode1      2.6.32-71.el6.x86_64      2.6.39                    failed
Result: Kernel version check failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
  12cnode1                              failed
  12cnode3                              failed
Check for integrity of file "/etc/resolv.conf" failed

The above 2 conditions have failed. We can ignore the Kernel version if it’s greater than or equal to 2.6.32. The integrity check of file “/etc/resolv.conf” for DNS has failed and I would ignore this as I’m not using any DNS.
Let’s move on with the node addition. In 12c, the location of “addnode.sh” is quite different than that of in 11gR2. In 11gR2, it was located under GRID_HOME/oui/bin, but in 12c, it’s located under “GRID_HOME/addnode”. On the node which is already a part of the cluster, run the “addnode.sh” script. Here, it’s been run from 12cnode1 in silent mode (and not through GUI). Specify the new node name and new node VIP name. Since the above mentioned 2 conditions have failed, I’m running “addnode.sh” with the “ignorePrereq” option.

[oracle@12cnode1 ~]$ cd /u01/app/12.1.0.2/grid/addnode/
[oracle@12cnode1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={12cnode3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={12cnode3-vip}" -silent -ignorePrereq
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 44743 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8138 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2016-03-23_05-18-17PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.1.0.2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/oraInventory/orainstRoot.sh
        2. /u01/app/12.1.0.2/grid/root.sh

Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[12cnode3]
Execute /u01/app/12.1.0.2/grid/root.sh on the following nodes:
[12cnode3]

The scripts can be executed in parallel on all the nodes.

..........
Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.
[oracle@12cnode1 addnode]$

Run as ROOT user the scripts “/u01/app/oraInventory/orainstRoot.sh” and “/u01/app/12.1.0.2/grid/root.sh” on the new node 12cnode3.

Here is the execution of the script “orainstRoot.sh”

[root@12cnode3 grid]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Execution of “root.sh” is as follows:

[root@12cnode3 grid]# /u01/app/12.1.0.2/grid/root.sh
Check /u01/app/12.1.0.2/grid/install/root_12cnode3.mydomain_2016-03-23_18-49-49.log for the output of root script
[root@12cnode3 grid]#
[root@12cnode3 ~]# cat /u01/app/12.1.0.2/grid/install/root_12cnode3.mydomain_2016-03-23_18-49-49.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/03/23 18:49:51 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2016/03/23 18:49:52 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2016/03/23 18:49:56 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2016/03/23 18:52:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12cnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12cnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on '12cnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12cnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on '12cnode3'
CRS-2672: Attempting to start 'ora.evmd' on '12cnode3'
CRS-2676: Start of 'ora.mdnsd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.evmd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on '12cnode3'
CRS-2676: Start of 'ora.gpnpd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on '12cnode3'
CRS-2676: Start of 'ora.gipcd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode3'
CRS-2676: Start of 'ora.cssdmonitor' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '12cnode3'
CRS-2672: Attempting to start 'ora.diskmon' on '12cnode3'
CRS-2676: Start of 'ora.diskmon' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cssd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode3'
CRS-2672: Attempting to start 'ora.ctssd' on '12cnode3'
CRS-2676: Start of 'ora.ctssd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on '12cnode3'
CRS-2676: Start of 'ora.asm' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12cnode3'
CRS-2676: Start of 'ora.storage' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crf' on '12cnode3'
CRS-2676: Start of 'ora.crf' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '12cnode3'
CRS-2676: Start of 'ora.crsd' on '12cnode3' succeeded
CRS-6017: Processing resource auto-start for servers: 12cnode3
CRS-2672: Attempting to start 'ora.net1.network' on '12cnode3'
CRS-2672: Attempting to start 'ora.FRA.dg' on '12cnode3'
CRS-2676: Start of 'ora.net1.network' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.ons' on '12cnode3'
CRS-2676: Start of 'ora.ons' on '12cnode3' succeeded
CRS-2676: Start of 'ora.FRA.dg' on '12cnode3' succeeded
CRS-6016: Resource auto-start has completed for server 12cnode3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/03/23 19:01:12 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2016/03/23 19:02:21 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Let’s check the CRS status and the resources on the new node.

[root@12cnode3 ~]# cd /u01/app/12.1.0.2/grid/bin/
[root@12cnode3 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@12cnode3 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.asm
               ONLINE  ONLINE       12cnode1                 Started,STABLE
               ONLINE  ONLINE       12cnode2                 Started,STABLE
               ONLINE  ONLINE       12cnode3                 Started,STABLE
ora.net1.network
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.ons
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.12cnode1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.12cnode2.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.12cnode3.vip
      1        ONLINE  ONLINE       12cnode3                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       12cnode1                 169.254.3.189 192.16
                                                             8.1.107,STABLE
ora.cvu
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.srprim.db
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
      2        ONLINE  ONLINE       12cnode2                 Open,STABLE
--------------------------------------------------------------------------------
[root@12cnode3 bin]#

Cluster status on all the nodes:

[oracle@12cnode1 ~]$ cd /u01/app/12.1.0.2/grid/bin/
[oracle@12cnode1 bin]$ ./crsctl check cluster -all
**************************************************************
12cnode1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
12cnode2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
12cnode3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@12cnode1 bin]$

Now it’s time to add the RDBMS home on to the new node. This is done in the same way as above by running the “addNode.sh” script from the RDBMS home on the existing node and not from the new node. (Of-course, it cannot be run on the new node as the RDBMS Home is still not available in it 🙂 )
The script is located at $ORACLE_HOME/addnode and here it’s being run from 12cnode1. Here too, I’m ignoring the pre-requisite as the pre-requisite for “resolve.conf” is not met.

[oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/addnode/
[oracle@12cnode1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={12cnode3}" -silent -ignorePrereq
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 44736 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8189 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/addNodeActions2016-03-24_11-18-10AM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.2/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0.2/db_1/root.sh

Execute /u01/app/oracle/product/12.1.0.2/db_1/root.sh on the following nodes:
[12cnode3]

..........
Update Inventory in progress.

..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.
[oracle@12cnode1 addnode]$

Now, run the “ROOT.SH” script from the new node as ROOT user. Here is the execution of the script.

[root@12cnode3 ~]# /u01/app/oracle/product/12.1.0.2/db_1/root.sh
Check /u01/app/oracle/product/12.1.0.2/db_1/install/root_12cnode3.mydomain_2016-03-24_12-01-16.log for the output of root script
[root@12cnode3 ~]# cat /u01/app/oracle/product/12.1.0.2/db_1/install/root_12cnode3.mydomain_2016-03-24_12-01-16.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0.2/db_1
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@12cnode3 ~]#

We now need to add a new instance “srprim3” on the cluster database that needs to be run on the new node 12cnode3. But before we do that, we need to have few of the cluster parameters modified according to the instance.

instance_number
Thread
Online Redo log groups for the new thread
UNDO_TABLESPACE

Let’s get the current log group and it’s members’ details based on which we can add new groups to the new thread.

SYS@srprim1&gt;select group#,thread#,bytes/1024/1024,members from v$log;

    GROUP#    THREAD# BYTES/1024/1024    MEMBERS
---------- ---------- --------------- ----------
         1          1              50          2
         2          1              50          2
         3          2              50          2
         4          2              50          2

SYS@srprim1&gt;select group#,member from v$logfile order by group#;

    GROUP# MEMBER
---------- -------------------------------------------------------
         1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571
         1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571
         2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577
         2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575
         3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373
         3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375
         4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377
         4 +FRA/SRPRIM/ONLINELOG/group_4.260.904396379

Let me add new groups to the new thread 3.

SYS@srprim1&gt;alter database add logfile thread 3 group 5 size 50M;

Database altered.

SYS@srprim1&gt;alter database add logfile thread 3 group 6 size 50M;

Database altered.

SYS@srprim1&gt;select group#,thread#,bytes/1024/1024,members from v$log;

    GROUP#    THREAD# BYTES/1024/1024    MEMBERS
---------- ---------- --------------- ----------
         1          1              50          2
         2          1              50          2
         3          2              50          2
         4          2              50          2
         5          3              50          2
         6          3              50          2

6 rows selected.

SYS@srprim1&gt;select group#,member from v$logfile order by group#;

    GROUP# MEMBER
---------- -------------------------------------------------------
         1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571
         1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571
         2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575
         2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577
         3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373
         3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375
         4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377
         4 +FRA/SRPRIM/ONLINELOG/group_4.260.904396379
         5 +DATA/SRPRIM/ONLINELOG/group_5.296.907330775
         5 +FRA/SRPRIM/ONLINELOG/group_5.261.907330777
         6 +DATA/SRPRIM/ONLINELOG/group_6.297.907330877

    GROUP# MEMBER
---------- -------------------------------------------------------
         6 +FRA/SRPRIM/ONLINELOG/group_6.262.907330879

12 rows selected.

Getting the UNDO tablespace details in order to know the undo tabelspaces assigned to the instances of the database.

SYS@srprim1&gt;select inst_id,name,value from gv$parameter where name like 'undo_tablespa%';

   INST_ID NAME                                     VALUE
---------- ---------------------------------------- ----------------------------------------
         2 undo_tablespace                          UNDOTBS2
         1 undo_tablespace                          UNDOTBS1

Now, let me create a new UNDO tablespace called “undotbs3” and is assign it to the new instance “srprim3”.

SYS@srprim1&gt;create undo tablespace UNDOTBS3 datafile size 400M;

Tablespace created.

SYS@srprim1&gt;alter system set undo_tablespace=UNDOTBS3 scope=spfile sid='srprim3';

System altered.

Assign an instance number to the new instance.

SYS@srprim1&gt;alter system set instance_number=3 SCOPE=SPFILE SID='srprim3';

System altered.

Assign the newly created Redo Log thread to the new instance.

SYS@srprim1&gt;alter system set thread=3 scope=spfile sid='srprim3';

System altered.

The current configuration of the “srprim” database is as follows:

[oracle@12cnode1 admin]$ srvctl config database -db srprim
Database unique name: srprim
Database name: srprim
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381
Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: srprim1,srprim2
Configured nodes: 12cnode1,12cnode2
Database is administrator managed
[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$

It’s time to add the instance “srprim3” to the database “srprim” using srvctl utility.

[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$
[oracle@12cnode1 admin]$ srvctl add instance -db srprim -instance srprim3 -node 12cnode3

Now, check the updated database configuration.

[oracle@12cnode1 admin]$ srvctl config database -db srprim
Database unique name: srprim
Database name: srprim
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381
Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: srprim1,srprim2,srprim3
Configured nodes: 12cnode1,12cnode2,12cnode3
Database is administrator managed
[oracle@12cnode1 admin]$

Since the PFILE of instance srprim1 has nothing but only the location of the SPFILE, copy it to the new node and rename it accordingly to the new instance name (srprim3).

[oracle@12cnode1 ~]$ cd $ORACLE_HOME/dbs
[oracle@12cnode1 dbs]$ ls -lrt initsrprim*
-rw-r-----. 1 oracle oinstall 39 Feb 21 13:15 initsrprim1.ora
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$ cat initsrprim1.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$
[oracle@12cnode1 dbs]$ scp initsrprim1.ora oracle@12cnode3:/u01/app/oracle/product/12.1.0.2/db_1/dbs/initsrprim3.ora
initsrprim1.ora                                                                                                                      100%   39     0.0KB/s   00:00
[oracle@12cnode1 dbs]$
[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/dbs
[oracle@12cnode3 dbs]$ ls -lrt initsrprim*
-rw-r-----. 1 oracle oinstall 39 Mar 24 12:38 initsrprim3.ora
[oracle@12cnode3 dbs]$
[oracle@12cnode3 dbs]$
[oracle@12cnode3 dbs]$ cat initsrprim3.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'
[oracle@12cnode3 dbs]$

Finally, before starting the instance, enable the newly created Redo Log thread 3 that was assigned to instance srprim3.

SYS@srprim1>alter database enable public thread 3;

Database altered.

SYS@srprim>

Start the instance srprim3 using SRVCTL utility.

[oracle@12cnode1 ~]$ srvctl status database -db srprim
Instance srprim1 is running on node 12cnode1
Instance srprim2 is running on node 12cnode2
Instance srprim3 is not running on node 12cnode3
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl start instance -db srprim -instance srprim3
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl status database -db srprim
Instance srprim1 is running on node 12cnode1
Instance srprim2 is running on node 12cnode2
Instance srprim3 is running on node 12cnode3

Below is just a list of the databases and the listeners running on the new node 12cnode3.

[oracle@12cnode3 ~]$ ps -ef | grep pmon
oracle    4266     1  0 10:24 ?        00:00:00 asm_pmon_+ASM3
oracle    7749     1  0 10:28 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   23632     1  0 12:42 ?        00:00:00 ora_pmon_srprim3
oracle   25350  3534  0 12:47 pts/0    00:00:00 grep pmon
[oracle@12cnode3 ~]$ ps -ef | grep tns
root         9     2  0 10:15 ?        00:00:00 [netns]
oracle    5774     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
oracle    5853     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
oracle    5856     1  0 10:26 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle   25406  3534  0 12:47 pts/0    00:00:00 grep tns
[oracle@12cnode3 ~]$

If the addnode.sh is run through GUI mode without using the “silent” option, then the instance addition too would be taken care by the GUI tool and there would be no requirement to go through the above manual steps.

COPYRIGHT

© Shivananda Rao P, 2012 to 2017. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organization. The views expressed by visitors on this blog are theirs solely and may not reflect mine.