Advertisements
Skip to content
November 29, 2018 / Shivananda Rao P

Configuring TDE in 12.1 RAC database with dataguard enabled

This article demonstrates implementing TDE in a RAC environment with standby database configured. The method used is same as what was done in my previous post, but uses a new feature of 12c called “administer key management”.

 

Environment:
2 node primary RAC: 12cnode1, 12cnode2
2 node standby RAC: 1212drnode1, 1212drnode2
Primary database name: srprim
Standby database name: srpstb

 

Just ensuring before implementing TDE that the standby database is in sync with the primary database.

 

DGMGRL> show configuration

Configuration - dgconfig

  Protection Mode: MaxPerformance
  Members:
  srprim - Primary database
    srpstb - Physical standby database 

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 6 seconds ago)

 

Create the directory structure where you need to place the wallet files and add the following entry in the SQLNET.ORA file on all the nodes of the primary database. This entry is needed to determine the wallet location.
In my case, I have decided to place the wallet files in the location “/u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet” and so have I created the same.

 

[oracle@12cnode1 ~]$ mkdir -p /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet
[oracle@12cnode2 ~]$ mkdir -p /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet

 

ENCRYPTION_WALLET_LOCATION = 
 (SOURCE = 
   (METHOD = FILE)
   (METHOD_DATA =
     (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet)
    )
  )

 

The instances show the default location of the wallet when queried the GV$ENCRYPTION_WALLET view. So, in order to get the instances reflect the right wallet location as per the SQLNET.ORA file, we need to bounce the instances.

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                STATUS          WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ------------------------------------------------------------ --------------- -------------------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet    NOT_AVAILABLE   UNKNOWN              SINGLE    UNDEFINED          0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet    NOT_AVAILABLE   UNKNOWN              SINGLE    UNDEFINED          0
		 

 

[oracle@12cnode1 ~]$ srvctl stop instance -instance srprim1 -database srprim
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ srvctl start instance -instance srprim1 -database srprim

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                        STATUS          WALLET_TYP WALLET_OR FULLY_BAC     CON_ID
---------- ---------- -------------------------------------------------------------------- --------------- ---------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/  NOT_AVAILABLE   UNKNOWN    SINGLE    UNDEFINED          0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet            NOT_AVAILABLE   UNKNOWN    SINGLE    UNDEFINED          0
		 

 

Similarly, restart the second instance as well.

 

[oracle@12cnode1 ~]$ srvctl stop instance -instance srprim2 -database srprim
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ srvctl start instance -instance srprim2 -database srprim

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYPE  WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    NOT_AVAILABLE   UNKNOWN      SINGLE    UNDEFINED          0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    NOT_AVAILABLE   UNKNOWN      SINGLE    UNDEFINED          0
		 

 

Now since the WRL_PARAMETER is reflecting the right directory where the wallet needs to be stored, lets move in creating the wallet keystore. Use the “administer key management” clause to create the keystore in the wallet directory location. This needs to be done only on one instance. (In my case, I’m running it on the first instance).

 

SYS@srprim1>administer key management create keystore '/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/' identified by "oracle123";

keystore altered.

 

Now when you query the gv$encryption_wallet view, you should see that status as CLOSED for the first instance where as the second instance should reflect it as “NOT_AVAILABLE”.

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYPE  WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    CLOSED          UNKNOWN      SINGLE    UNDEFINED          0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    NOT_AVAILABLE   UNKNOWN      SINGLE    UNDEFINED          0

 

Now, open the keystore that was created previously on the instance where it was created.

 

SYS@srprim1>administer key management set keystore open identified by "oracle123"; 

keystore altered.

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYPE  WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    OPEN_NO_MASTER_ PASSWORD     SINGLE    UNDEFINED          0
                                                                                             KEY

         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    NOT_AVAILABLE   UNKNOWN      SINGLE    UNDEFINED          0

 

As we have opened the keystore, the status should reflect as OPEN, but in 12c, you need to have a MASTER KEY as well which is why the STATUS column reflects as “OPEN_NO_MASTER_KEY” where gv$encryption_wallet view is queried.

 

So, create a MASTER KEY using the “administer key management” clause as stated below. The MASTER KEY would be stored in the same location where the wallet file location was specified in the sqlnet.ora file.

 

		 
SYS@srprim1>administer key management set key identified by "oracle123" with backup;

keystore altered.

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYPE  WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    OPEN            PASSWORD     SINGLE    NO                 0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    NOT_AVAILABLE   UNKNOWN      SINGLE    UNDEFINED          0

 

Once the MASTER KEY is created, the status for the corresponding instance turns out to be OPEN when “gv$encryption_wallet” is queried.

 

As seen, the wallet file “ewallet.p12” does also contains the master key. This needs to be copied on to the remaining nodes of the cluster.

 

		 
[oracle@12cnode1 ~]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/
total 8
-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12
-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$ 
[oracle@12cnode1 ~]$

 

Copying the wallet file to node 12cnode2.

 

 
[oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/
[oracle@12cnode1 wallet]$ scp ewallet.p12 oracle@12cnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/
ewallet.p12                                                                                                           100% 3848     3.8KB/s   00:00    
[oracle@12cnode1 wallet]$ 

 

Now query gv$encryption_wallet to check the status with respect to each instance. We can see that the status now reflects as “OPEN” for the second instance on 12cnode2 as well.

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS     WALLET_TYPE     WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- ---------- --------------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    OPEN       PASSWORD        SINGLE    NO                 0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    OPEN       PASSWORD        SINGLE    NO                 0

 

Please note that at this point of time, we have setup only manual wallet management which means that when the database/instance is started, we need to open the wallet manually.

 

		 
[oracle@12cnode1 wallet]$ srvctl stop database -database srprim
[oracle@12cnode1 wallet]$ 
[oracle@12cnode1 wallet]$ 
[oracle@12cnode1 wallet]$ srvctl start database -database srprim

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE        WRL_PARAMETER                                                          STATUS     WALLET_TYP WALLET_OR FULLY_BAC     CON_ID
---------- --------------- ---------------------------------------------------------------------- ---------- ---------- --------- --------- ----------
         1 FILE            /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    CLOSED     UNKNOWN    SINGLE    UNDEFINED          0
         2 FILE            /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    CLOSED     UNKNOWN    SINGLE    UNDEFINED          0

SYS@srprim1>

 

Now, let me create auto login setup which automatically opens the wallet upon the instance/database retart. This creates a file with name “cwallet.sso” under the same location that is specified in the sqlnet.ora file.

 

SYS@srprim1>administer key management create auto_login keystore from keystore '/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/' identified by "oracle123";

keystore altered.

 

SYS@srprim1>host
[oracle@12cnode1 wallet]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/
total 12
-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12
-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12
-rw-r--r--. 1 oracle oinstall 3893 Aug 29 12:29 cwallet.sso

 

[oracle@12cnode1 wallet]$ srvctl stop database -d srprim

 

Copy this auto login wallet file (cwallet.sso) on to the second node 12cnode2 as well and then start the database. Upon this, the wallets should be opened automatically for both the instances.

 

[oracle@12cnode1 wallet]$ scp cwallet.sso oracle@12cnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/
cwallet.sso                                                                                                           100% 3893     3.8KB/s   00:00    
[oracle@12cnode1 wallet]$ srvctl start database -d srprim

 

SYS@srprim1>select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS     WALLET_TYPE     WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- ---------- --------------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/    OPEN       AUTOLOGIN       SINGLE    NO                 0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/    OPEN       AUTOLOGIN       SINGLE    NO                 0

 

With the primary setup, the standby nodes should not be a problem. Perform the same initial steps as did on primary nodes of adding the below entry (speaking about the wallet location) in the sqlnet.ora file on all the nodes of the standby cluster.
Please note that if the directory structure is different, act accordingly in specifiying the appropriate available location.

 

ENCRYPTION_WALLET_LOCATION = 
 (SOURCE = 
   (METHOD = FILE)
   (METHOD_DATA =
     (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet)
    )
  )

 

  
[oracle@1212drnode1 admin]$ pwd
/u01/app/oracle/product/12.1.0.2/db_1/network/admin
[oracle@1212drnode1 admin]$ mkdir -p srpstb1/wallet

 

[oracle@1212drnode2 admin]$ pwd
/u01/app/oracle/product/12.1.0.2/db_1/network/admin
[oracle@1212drnode2 admin]$ mkdir -p srpstb2/wallet

 

Since we haven’t bounced the standby instances after adding the above entry in sqlnet.ora file, the wrl_parameter yet points to the inital location. Let’s copy the wallet files from the primary node and then bounce the standby instances at one go.

 

SQL> select * from gv$encryption_wallet order by 1;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYP WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ---------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/admin/srpstb/wallet              NOT_AVAILABLE   UNKNOWN    SINGLE    UNDEFINED          0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/admin/srpstb/wallet              NOT_AVAILABLE   UNKNOWN    SINGLE    UNDEFINED          0

 

Scp all the wallet files (master key file and autologin file) from any one of the primary node to the first standby node.

 

[oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/
[oracle@12cnode1 wallet]$ ls -lrt
total 12
-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12
-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12
-rw-r--r--. 1 oracle oinstall 3893 Aug 29 12:29 cwallet.sso

 

[oracle@12cnode1 wallet]$ scp *wallet.* oracle@1212drnode1:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb1/wallet/
Warning: the RSA host key for '1212drnode1' differs from the key for the IP address '192.168.0.120'
Offending key for IP in /home/oracle/.ssh/known_hosts:14
Matching host key in /home/oracle/.ssh/known_hosts:16
Are you sure you want to continue connecting (yes/no)? yes
oracle@1212drnode1's password: 
cwallet.sso                                                                                                          100% 3893     3.8KB/s   00:00    
ewallet.p12                                                                                                          100% 3848     3.8KB/s   00:00    
[oracle@12cnode1 wallet]$ 

 

Similarly scp the same wallet files from the primary node to the second node of the standby cluster as well.

 

[oracle@12cnode1 wallet]$ scp *wallet.* oracle@1212drnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb2/wallet/
The authenticity of host '1212drnode2 (192.168.0.127)' can't be established.
RSA key fingerprint is 0a:f2:b7:22:3b:2b:fa:32:ea:dc:17:c1:22:7c:4e:e3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '1212drnode2,192.168.0.127' (RSA) to the list of known hosts.
oracle@1212drnode2's password: 
cwallet.sso                                                                                                          100% 3893     3.8KB/s   00:00    
ewallet.p12                                                                                                          100% 3848     3.8KB/s   00:00    
[oracle@12cnode1 wallet]$ 

 

[oracle@1212drnode1 ~]$ srvctl status database -d srpstb -v -f
Instance srpstb1 is running on node 1212drnode1. Instance status: Mounted (Closed).
Instance srpstb2 is running on node 1212drnode2. Instance status: Mounted (Closed).
[oracle@1212drnode1 ~]$ 
[oracle@1212drnode1 ~]$ 

 

Now its time to bounce the standby database and check the wallet status.

 

[oracle@1212drnode1 ~]$ srvctl stop database -d srpstb
[oracle@1212drnode1 ~]$ 
[oracle@1212drnode1 ~]$ 
[oracle@1212drnode1 ~]$ srvctl start database -d srpstb
[oracle@1212drnode1 ~]$ srvctl status database -d srpstb -v -f
Instance srpstb1 is running on node 1212drnode1. Instance status: Mounted (Closed).
Instance srpstb2 is running on node 1212drnode2. Instance status: Mounted (Closed).
[oracle@1212drnode1 ~]$ 

 

Glad to see that the wallets are opened on both the standby nodes and that the wallet type represents as “AUTOLOGIN”.

 

[oracle@1212drnode1 ~]$ sql

SQL*Plus: Release 12.1.0.2.0 Production on Sat Sep 8 12:14:18 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> set linesize 300
SQL> col wrl_type for a10
SQL> col wallet_type for a10
SQL> col status for a15
SQL> col wrl_parameter for a70
SQL> select * from gv$encryption_wallet order by 1;

   INST_ID WRL_TYPE   WRL_PARAMETER                                                          STATUS          WALLET_TYP WALLET_OR FULLY_BAC     CON_ID
---------- ---------- ---------------------------------------------------------------------- --------------- ---------- --------- --------- ----------
         1 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb1/wallet/    OPEN            AUTOLOGIN  SINGLE    NO                 0
         2 FILE       /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb2/wallet/    OPEN            AUTOLOGIN  SINGLE    NO                 0

SQL> 

 

Finally, ensure that the standby database is in sync with the primary database.

 

[oracle@12cnode1 u02]$ dgmgrl /
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL> show configuration

Configuration - dgconfig

  Protection Mode: MaxPerformance
  Members:
  srprim - Primary database
    srpstb - Physical standby database 

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

 

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

 

Advertisements
September 19, 2018 / Shivananda Rao P

Configuring TDE in 11.2 RAC database with dataguard enabled

In this post, I’m writing on how to implement TDE in a 11g RAC environment with Dataguard

 

Environment:
Oracle RDBMS version: 11.2.0.3
Primary nodes: node1, node2
Standby nodes: drnode1, drnode2
Primary Databaase: crmsdb
Standby Database: crmdb

 

Firstly, let me check if the standby database is in sync with the primary database.

 

[oracle@node1]$ dgmgrl /
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - crmdb_dg

  Protection Mode: MaxPerformance
  Databases:
    crmsdb - Primary database
    crmdb  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

 

Now, add the below entry in the SQLNET.ORA file on all the nodes of the Primary database. The SQLNET.ORA file needs to be present in the location where the TNS_ADMIN points.

 

The value for the DIRECTORY parameter in the below entry specifies the location where the wallet needs to be stored. In my case, I’m placing it ata “$ORACLE_HOME/network/admin//wallet”. If the specified directory structure does not exist, then you need to create one.

 

ENCRYPTION_WALLET_LOCATION=
(SOURCE=
  (METHOD=FILE)
  (METHOD_DATA=(DIRECTORY=/u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet))
)

 

Please note that in a RAC environment, the wallet can be placed in a shared location where all the nodes of the cluster database can access it (Ideally, it can be placed on ASM storage which I shall expalin in my coming posts). In this environment, I’m having a separate wallet for each node.

 

Once you add the entry in SQLNET.ORA file, you need to bounce the instances so that wallet location is upated correctly when you query the gv$encryption_wallet view.

 

As of now, I have not bounced the 2nd instance and due to which the wallet location is pointing to wrong location when I query the gv$encryption_wallet view.
Hence a restart of the 2nd instance is needed.

 

SQL> select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE        WRL_PARAMETER                                                          STATUS
---------- --------------- ---------------------------------------------------------------------- ------------------
         1 file            /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED
         2 file            /u01/app/oracle/product/11.2.0.3/db_1/admin/crmsdb/wallet              CLOSED
		 

 

[oracle@node1 admin]$ srvctl stop instance -i crmsdb2 -d crmsdb
[oracle@node1 admin]$ 
[oracle@node1 admin]$ srvctl start instance -i crmsdb2 -d crmsdb
[oracle@node1 admin]$ 

 

Now, querying the gv$encryption_wallet provides the expected result.

 

SQL> select * from gv$encryption_wallet order by inst_id;

   INST_ID WRL_TYPE        WRL_PARAMETER                                                          STATUS
---------- --------------- ---------------------------------------------------------------------- ------------------
         1 file            /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED
         2 file            /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED
		 

 

On any one instance of the primary database, create the wallet. Here, I’m running it on the first node of the primary database.

 

SQL> alter system set encryption key identified by "oracle123";

System altered.

 

Just crosscheck the wallet file presence under the wallet location.

 

[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet
[oracle@node1 wallet]$ ls -lrt
total 4
-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12

 

Now, SCP the wallet file

 

[oracle@node1 wallet]$ scp ewallet.p12 oracle@node2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb2/wallet/
ewallet.p12                                                                                                           100% 1573     1.5KB/s   00:00    
[oracle@node1 wallet]$ 

 

You now manually need to open the wallet

 

SQL> alter system set encryption wallet open identified by "oracle123";

System altered.

 


SQL> select * from gv$encryption_wallet order by 1;

   INST_ID WRL_TYPE             WRL_PARAMETER                                                          STATUS
---------- -------------------- ---------------------------------------------------------------------- ------------------
         1 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN
         2 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN

 

In order to close the wallet manually, you can use the below command.

 

SQL> alter system set encryption wallet close identified by "oracle123";

System altered.

 

It is recommended to enable the wallet for auto login. This will ensure that there is no need to open or close the wallet manually. Oracle will automatically open the wallet when an encrypted object is accessed or created. This can be done by using the “orapki” utility.

 

[oracle@node1 wallet]$ orapki wallet create -wallet /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet -auto_login
Oracle PKI Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:            
   
[oracle@node1 wallet]$

 

When prompted for the wallet password, provide the password that was used previously to create the key.

 

[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet/
[oracle@node1 wallet]$ ls -lrt
total 8
-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12
-rw-------. 1 oracle oinstall 1651 Aug 25 13:47 cwallet.sso

 

[oracle@node1 wallet]$ scp cwallet.sso oracle@node2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb2/wallet/
cwallet.sso                                                                                                           100% 1651     1.6KB/s   00:00   

 

Now, bouncing the database will not stop you for opening the wallet as the above autologin option opens it automatically.

 

[oracle@node1 ~]$ srvctl stop database -d crmsdb
[oracle@node1 ~]$ 
[oracle@node1 ~]$ 
[oracle@node1 ~]$ srvctl start database -d crmsdb

 

[oracle@node1 ~]$ sql

SQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 8 10:55:31 2018

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> set linesize 300
SQL> col wrl_parameter for a70
SQL> select * from gv$encryption_wallet order by 1;

   INST_ID WRL_TYPE             WRL_PARAMETER                                                          STATUS
---------- -------------------- ---------------------------------------------------------------------- ------------------
         1 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN
         2 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN

 

Now on the DR nodes, follow the same steps as done on the primary nodes.
Add same entry as above in sqlnet.ora file on all nodes of the standby and then restart the instances to get the wallet_location updated.

 

[oracle@drnode1 admin]$ srvctl stop instance -i crmdb1 -d crmdb
[oracle@drnode1 admin]$ srvctl start instance -i crmdb1 -d crmdb

 

[oracle@drnode1 admin]$ srvctl stop instance -i crmdb2 -d crmdb
[oracle@drnode1 admin]$ srvctl start instance -i crmdb2 -d crmdb
[oracle@drnode1 admin]$ 

 

Once restarted, query the gv$encryption_wallet view to verify if WRL_PARAMETER value corresponds to that mentioned in the WALLET_LOCATION value in the SQLNET.ORA file.

 

SQL> select * from gv$encryption_wallet;

   INST_ID WRL_TYPE        WRL_PARAMETER                                                          STATUS
---------- --------------- ---------------------------------------------------------------------- ------------------
         1 file            /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED
         2 file            /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED

 

On the standby, MRP may terminate

 

Errors with log +FRA/crmdb/archivelog/2018_08_25/thread_1_seq_64.376.985096395
MRP0: Background Media Recovery terminated with error 28365
Errors in file /u01/app/oracle/diag/rdbms/crmdb/crmdb1/trace/crmdb1_mrp0_12241.trc:
ORA-28365: wallet is not open
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 1361738
MRP0: Background Media Recovery process shutdown (crmdb1)

 

Now, do not create any key on the standby database.

 

		 
[oracle@node1 wallet]$ scp ewallet.p12 oracle@drnode1:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb1/wallet/
oracle@drnode1's password: 
ewallet.p12                                                                                                           100% 1573     1.5KB/s   00:00    
[oracle@node1 wallet]$ 
[oracle@node1 wallet]$ 
[oracle@node1 wallet]$ scp ewallet.p12 oracle@drnode2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb2/wallet/
The authenticity of host 'drnode2 (192.168.0.123)' can't be established.
RSA key fingerprint is 48:83:c6:32:28:0d:25:a8:5a:25:03:c7:21:96:6e:64.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'drnode2,192.168.0.123' (RSA) to the list of known hosts.
oracle@drnode2's password: 
ewallet.p12                                                                                                           100% 1573     1.5KB/s   00:00    
[oracle@node1 wallet]$ 

 

You can now open the wallet manually on the standby database using the below command:

 

SQL> alter system set encryption wallet open identified by "oracle123";

System altered.

 

But since I have enabled autologin on the primary database, I’d like the same to be enabled on the standby as well. So shall copy the cwallet.sso file from primary to both the nodes of the standby.

 

[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet/
[oracle@node1 wallet]$ ls -lrt
total 8
-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12
-rw-------. 1 oracle oinstall 1651 Aug 25 13:47 cwallet.sso
[oracle@node1 wallet]$ 
[oracle@node1 wallet]$ scp cwallet.sso oracle@drnode1:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb1/wallet/
oracle@drnode1's password: 
cwallet.sso                                                                                                           100% 1651     1.6KB/s   00:00    
[oracle@node1 wallet]$ 
[oracle@node1 wallet]$ scp cwallet.sso oracle@drnode2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb2/wallet/
oracle@drnode2's password: 
cwallet.sso                                                                                                           100% 1651     1.6KB/s   00:00    
[oracle@node1 wallet]$ 

 

[oracle@drnode1 ~]$ srvctl stop database -d crmdb
[oracle@drnode1 ~]$ 
[oracle@drnode1 ~]$ srvctl start database -d crmdb

 

[oracle@drnode1 ~]$ sql

SQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 8 11:17:50 2018

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> col wrl_parameter for a70
SQL> set linesize 300
SQL> select * from gv$encryption_wallet order by 1;

   INST_ID WRL_TYPE             WRL_PARAMETER                                                          STATUS
---------- -------------------- ---------------------------------------------------------------------- ------------------
         1 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN
         2 file                 /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN

 

	 
SQL> select status,instance_name,database_role from v$database,v$instance;

STATUS       INSTANCE_NAME    DATABASE_ROLE
------------ ---------------- ----------------
MOUNTED      crmdb1           PHYSICAL STANDBY

 

SQL> select process,status,sequence#,inst_id,thread# from gV$managed_standby;

PROCESS   STATUS        SEQUENCE#    INST_ID    THREAD#
--------- ------------ ---------- ---------- ----------
ARCH      CONNECTED             0          1          0
ARCH      CONNECTED             0          1          0
ARCH      CONNECTED             0          1          0
ARCH      CONNECTED             0          1          0
MRP0      APPLYING_LOG         71          1          2
ARCH      CLOSING              87          2          1
ARCH      CLOSING              70          2          2
ARCH      CONNECTED             0          2          0
ARCH      CONNECTED             0          2          0
RFS       IDLE                  0          2          0
RFS       IDLE                  0          2          0

PROCESS   STATUS        SEQUENCE#    INST_ID    THREAD#
--------- ------------ ---------- ---------- ----------
RFS       IDLE                 88          2          1
RFS       IDLE                  0          2          0
RFS       IDLE                 71          2          2
RFS       IDLE                  0          2          0

15 rows selected.

 

[oracle@drnode1 ~]$ dgmgrl /
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - crmdb_dg

  Protection Mode: MaxPerformance
  Databases:
    crmsdb - Primary database
    crmdb  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

 

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

 

April 28, 2018 / Shivananda Rao P

PDB Cloning in a Dataguard Environment

In this article, I’m demonstrating on how or what impact would be seen on the Physical standby database when you clone a Pluggable Database on the Primary.

 

Here is the environment detail.

 

Primary database name: targetdb
Standby database name: tarstdb
PDB name already plugged into CDB: targetpdb
PDB name to be created: targetpdb1

 

Please note that both TARGETPDB and TARGETPDB1 would be under the same container.

 

Before I start, let me give a brief outcome of the setup.
I have DG broker configured for the setup and I could see below that the standby database is in sync with the primary database.

 

[oracle@ora1-1 ~]$ dgmgrl /
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL> show configuration

Configuration - dgtest

  Protection Mode: MaxPerformance
  Members:
  targetdb - Primary database
    tarstdb  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 49 seconds ago)

 

TARGETPDB1 will be created from TARGETPDB and would be plugged into the same ROOT container as TARGETPDB
Below shows the list of datafiles that TARGETPDB has.

 

SYS@targetdb> select status,instance_Name,database_role from v$database,v$Instance;

STATUS       INSTANCE_NAME    DATABASE_ROLE
------------ ---------------- ----------------
OPEN         targetdb         PRIMARY

 

SYS@targetdb> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 TARGETPDB                      READ WRITE NO

		 
SYS@targetdb> alter session set container=targetpdb;

Session altered.

 

SYS@targetdb> col file_name for a99
SYS@targetdb> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/system.272.906204413
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/sysaux.290.906204413
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/test.271.907334287
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/myobjects.291.930751499
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/migtbs1.279.930751523
+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/users.273.913039191

6 rows selected.

 

Let me clone TARGETPDB to create TARGETPDB1. In order to clone, I need the TARGETPDB to be opened in READ ONLY mode which is done in the below steps.
Logging into the CDB$ROOT container, I OPEN TARGETPDB in READ ONLY mode followed by which I issue the CREATE PLUGGABLE DATABASE command. Please note that while creating the PDB, I’m also using the clause “STANDBY=NONE”.

 

In case if I was cloning from PDB$SEED, then the datafiles of the newly created PDB would automatically get replicated on the physical standby side too without using the “STANDBY=NONE” clause. But, when a PDB is cloned from another PDB under the same container in a dataguard environment, then, the datafiles may not get replicated to the Physical standby side if the physical standby database is in MOUNT mode. In such a case, use the “STANDBYS=NONE” clause which would defer the recover of the newly created PDB on the physical standby database. The datafiles for the new PDB would be marked as OFFLINE/RECOVER in physical standby database and any additional redo for that PDB will be ignored. This is interpreted as at some point in the future, the PDBs files will be copied to the physical standby database and recovery for the PDB will be enabled.

 

Here are the cloning and creation steps:
On Primary, place the “targetpdb” in read only mode.

 

SYS@targetdb> alter pluggable database targetpdb close immediate;

Pluggable database altered.

SYS@targetdb> alter pluggable database targetpdb open read only;

Pluggable database altered.

SYS@targetdb>

 

Now, create pluggable database “targetpdb1” with the clause “standbys=none”.

 

SYS@targetdb> create pluggable database targetpdb1 from targetpdb standbys=none;

Pluggable database created.

SYS@targetdb>		 

SYS@targetdb> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 TARGETPDB                      READ ONLY  NO
         4 TARGETPDB1                     MOUNTED
SYS@targetdb>

 

As seen above, since “targetpdb” is still in READ ONLY mode, we need to revert it back to “READ WRITE” mode. Also, we need to open the newly created pdb “targetpdb1”.

 

SYS@targetdb> alter pluggable database targetpdb close immediate;

Pluggable database altered.

SYS@targetdb> alter pluggable database targetpdb open;

Pluggable database altered.

SYS@targetdb> alter pluggable database targetpdb1 open;

Pluggable database altered.

SYS@targetdb> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 TARGETPDB                      READ WRITE NO
         4 TARGETPDB1                     READ WRITE NO
SYS@targetdb>

 

Moving on to the physical standby database side, let’s check the status of the PDBs.

 

SYS@tarstdb> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 TARGETPDB                      MOUNTED
         4 TARGETPDB1                     MOUNTED
SYS@tarstdb>

 

On checking the recovery status of each of the PDBs that are plugged into the physical standby container database, we see that the recovery of the newly created PDB “targetpdb1” is “DISABLED” and this is because we had used the clause “standbys=none” while creating the pdb.

 

SYS@tarstdb> select name, recovery_status from v$pdbs;

NAME                           RECOVERY
------------------------------ --------
PDB$SEED                       ENABLED
TARGETPDB                      ENABLED
TARGETPDB1                     DISABLED

SYS@tarstdb> alter session set container=targetpdb1;

Session altered.

SYS@tarstdb> select name, status from v$datafile;

NAME                                                    STATUS
------------------------------------------------------- -------
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021  SYSOFF
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022  RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023  RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024  RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025  RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026  RECOVER

6 rows selected.

 

When I query the v$recovery_file, I see that all the files of the newly created PDB seem to be errored out as “Missing” on the standby.

 

SYS@tarstdb> select * from v$recover_file;

     FILE# ONLINE  ONLINE_ ERROR                             CHANGE# TIME          CON_ID
---------- ------- ------- ------------------------------ ---------- --------- ----------
        21 OFFLINE OFFLINE FILE MISSING                            0                    4
        22 OFFLINE OFFLINE FILE MISSING                            0                    4
        23 OFFLINE OFFLINE FILE MISSING                            0                    4
        24 OFFLINE OFFLINE FILE MISSING                            0                    4
        25 OFFLINE OFFLINE FILE MISSING                            0                    4
        26 OFFLINE OFFLINE FILE MISSING                            0                    4

6 rows selected.

 

Ok, so how do we get these files on to the standby ? Simple option would be to restore them from the Primary PDB (Targetpdb) through RMAN using the “net service”.

 

I have DG Broker configured for my setup, so just making sure that the standby database is in sync with the primary database.

 

[oracle@ora1-1 ~]$ dgmgrl /
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL> show configuration

Configuration - dgtest

  Protection Mode: MaxPerformance
  Members:
  targetdb - Primary database
    tarstdb  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 9 seconds ago)

 

Now, it’s time for me to restore the files of the newly created PDB (targetpdb1) from primary to the standby site. I restore it using the “service” clause connecting to the primary.

 

I use the “set newname for pluggable database targetpdb1 to new” to get the OMF format for the datafiles to be restored which would be done to the value specified in the “db_create_file_dest” parameter. Followed by this, is the “restore pluggable database targetpdb1 from service targetdb” suggesting to restore that particular PDB using the NET SERVICE to connect to primary. And finally switching the datafiles restored.

 

[oracle@ora1-4 dbs]$ rman target /

Recovery Manager: Release 12.1.0.2.0 - Production on Sat Mar 17 12:22:59 2018

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TARGETDB (DBID=1297732342, not open)

RMAN> run
2> {
3> set newname for pluggable database targetpdb1 to new;
4> restore pluggable database targetpdb1 from service targetdb;
5> switch datafile all;
6> }

executing command: SET NEWNAME

Starting restore at 17-MAR-18
Starting implicit crosscheck backup at 17-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=51 device type=DISK
Crosschecked 14 objects
Finished implicit crosscheck backup at 17-MAR-18

Starting implicit crosscheck copy at 17-MAR-18
using channel ORA_DISK_1
Crosschecked 11 objects
Finished implicit crosscheck copy at 17-MAR-18

searching for all files in the recovery area
cataloging files...
no files cataloged

using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00021 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:26
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00022 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00023 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00024 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00025 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service targetdb
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00026 to +DATA_NEW
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
Finished restore at 17-MAR-18

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of switch command at 03/17/2018 12:26:38
ORA-19563: datafile copy header validation failed for file +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125

RMAN>

 

Quite strange !! The datafiles of “targetpdb1” are restored but errors out while performing the switch operation. This datafile was plugged in as part of transportable tablespace migration. Suspecting that could be the case but unsure why it failed during the execution of the switch command. I shall investigate on that and post in the upcoming days.

 

As a workaround, I thought to rename those files manually. In order to do that, I set the “StandbyFileManagement” parameter to “Manual” on standby database and stop the RECOVERY on the standby database (tarstdb).

 

DGMGRL> edit database tarstdb set property StandbyFileManagement =Manual;
Property "standbyfilemanagement" updated
DGMGRL> edit database tarstdb set state='apply-off';
Succeeded.

 

The datafile status and it’s name for the newly created PDB (targetpdb1) on standby site needs to be captured.

 

SYS@tarstdb> select name,status from v$datafile;

NAME                                                                             STATUS
-------------------------------------------------------------------------------- -------
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021                           SYSOFF
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022                           RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023                           RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024                           RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025                           RECOVER
/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026                           RECOVER

6 rows selected.

 

From the alert log of the standby database, I get the OMF format of the files restored in the above step for the new PDB (targetpdb1).

 

Sat Mar 17 13:34:38 2018
Full restore complete of datafile 21 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063.  Elapsed time: 0:00:13
  checkpoint is 10105360
  last deallocation scn is 8421271
  Undo Optimization current scn is 10090649
Sat Mar 17 13:35:19 2018
Full restore complete of datafile 22 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089.  Elapsed time: 0:00:30
  checkpoint is 10105405
  last deallocation scn is 4478083
Full restore complete of datafile 23 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125.  Elapsed time: 0:00:00
  checkpoint is 10105471
  last deallocation scn is 1594145
Full restore complete of datafile 24 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125.  Elapsed time: 0:00:01
  checkpoint is 10105481
  last deallocation scn is 907776
Full restore complete of datafile 25 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127.  Elapsed time: 0:00:00
  checkpoint is 10105484
  last deallocation scn is 907776
Full restore complete of datafile 26 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129.  Elapsed time: 0:00:01
  checkpoint is 10105489
  last deallocation scn is 3
Sat Mar 17 13:37:45 2018
Checker run found 13 new persistent data failures

 

I then rename each of these datafiles manually by connecting to the CDB$ROOT of the standby database. I understand that this is a tedious process, but for time being I had to go with this.

 

SYS@tarstdb>  alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063';

Database altered.

SYS@tarstdb>  alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089';

Database altered.

SYS@tarstdb> alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125';

Database altered.

SYS@tarstdb> alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125';

Database altered.

SYS@tarstdb> alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127';

Database altered.

SYS@tarstdb> alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129';

Database altered.

 

Now, let me check the datafile names after renaming them. All looks good as below.

 

SYS@tarstdb>  select name,status from v$datafile;

NAME                                                                                   	STATUS
-------------------------------------------------------------------------------------  	-------
+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063       	SYSOFF
+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089       	RECOVER
+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125         	RECOVER
+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125		RECOVER                                              +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127		 RECOVER
+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129         RECOVER

6 rows selected.

 

We now need to enable the recovery for the newly created PDB targetpdb1 on standby database. For this, connect to the targetpdb1 container and enable the recovery.

 

SYS@tarstdb> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 TARGETPDB                      MOUNTED
         4 TARGETPDB1                     MOUNTED
SYS@tarstdb>
SYS@tarstdb> alter session set container=targetpdb1;

Session altered.

SYS@tarstdb> alter pluggable database enable recovery;

Pluggable database altered.

 

The changes made above to the parameter “StandbyFileManagement” needs to be reverted back to the value “AUTO” and start MRP on the standby database.

 

DGMGRL> edit database tarstdb set property StandbyFileManagement =AUTO;
Property "standbyfilemanagement" updated
DGMGRL> edit database tarstdb set state='apply-on';
Succeeded.

 

Crosscheck if all looks fine.

 

SYS@tarstdb> select name, recovery_status from v$pdbs;

NAME                           RECOVERY
------------------------------ --------
PDB$SEED                       ENABLED
TARGETPDB                      ENABLED
TARGETPDB1                     ENABLED

 

SYS@tarstdb> alter session set container=targetpdb1;

Session altered.

SYS@tarstdb> col name for a90
SYS@tarstdb>
SYS@tarstdb> select * from v$recovery_status;

no rows selected

 

Verfiy if the standby database is in sync with the primary database.

 

On Primary:

 

SYS@targetdb> select max(sequence#) from v$archived_log;

MAX(SEQUENCE#)
--------------
           335

SYS@targetdb> alter system switch logfile;

System altered.

SYS@targetdb> /

System altered.

SYS@targetdb> /

System altered.

SYS@targetdb> select max(sequence#) from v$archived_log;

MAX(SEQUENCE#)
--------------
           338

 

On standby:

 

SYS@tarstdb> select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CLOSING             337
ARCH      CONNECTED             0
ARCH      CLOSING             336
ARCH      CLOSING             338
RFS       IDLE                  0
RFS       IDLE                339
MRP0      APPLYING_LOG        339

7 rows selected.

 

SYS@tarstdb> select max(sequence#) from v$archived_log where applied='YES';

MAX(SEQUENCE#)
--------------
           337

 

All looks good !!

 

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

 

February 12, 2018 / Shivananda Rao P

RAC Node Deletion in 12c

In this post, I shall be discussing on the steps involved in removing a node from RAC cluster in oracle 12c.

 

Environment:

 

Cluster Nodes: 12cnode1, 12cnode2, 12cnode3
Database instances on the above nodes:
srprim1 on 12cnode1
srprim2 on 12cnode2
srprim3 on 12cnode3

 

Grid Infrastructure version: 12.1.0.2
RDBMS version: 12.1.0.2
Node to be removed: 12cnode3

 

Here is a brief detail on the environment. The list of nodes captured using “olsnodes”.

 

[oracle@12cnode3 ~]$ olsnodes
12cnode1
12cnode2
12cnode3
[oracle@12cnode3 ~]$

 

Also this environment has Flex ASM enabled.

 

[oracle@12cnode3 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[oracle@12cnode3 ~]$
[oracle@12cnode3 ~]$ asmcmd showclusterstate
Normal
[oracle@12cnode3 ~]$

 

Below is the list of the resources managed by this cluster.

 

[oracle@12cnode3 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               OFFLINE OFFLINE      12cnode3                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               OFFLINE OFFLINE      12cnode3                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               OFFLINE OFFLINE      12cnode3                 STABLE
ora.net1.network
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
ora.ons
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
               ONLINE  ONLINE       12cnode3                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.12cnode1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.12cnode2.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.12cnode3.vip
      1        ONLINE  ONLINE       12cnode3                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       12cnode1                 169.254.3.189 192.16
                                                             8.1.107,STABLE
ora.asm
      1        ONLINE  ONLINE       12cnode1                 Started,STABLE
      3        ONLINE  ONLINE       12cnode2                 Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.srprim.db
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
      2        ONLINE  ONLINE       12cnode2                 Open,STABLE
      3        ONLINE  ONLINE       12cnode3                 Open,STABLE
ora.srprim.srprim_i03.svc
      1        ONLINE  ONLINE       12cnode1                 STABLE
--------------------------------------------------------------------------------

 
 

Since Flex ASM is enabled, I see that all my oracle database instances are running on all the nodes even though ASM is not currently running on node 12cnode3. But that should not be a show stopper for us.

 

[oracle@12cnode3 ~]$ srvctl status asm
ASM is running on 12cnode2,12cnode1

 

[oracle@12cnode3 ~]$ crsctl get node role config -all
Node '12cnode1' configured role is 'hub'
Node '12cnode2' configured role is 'hub'
Node '12cnode3' configured role is 'hub'

 

[oracle@12cnode3 ~]$  crsctl get node role status -all
Node '12cnode1' active role is 'hub'
Node '12cnode2' active role is 'hub'
Node '12cnode3' active role is 'hub'

 

Below is the detail of the votedisk and OCR files.

 

[oracle@12cnode3 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   eba1a2f1e10b4f60bfbe59bd50be0ae6 (/dev/DSK1) [DATA]
Located 1 voting disk(s).

 

[oracle@12cnode3 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1740
         Available space (kbytes) :     407828
         ID                       :  553047649
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@12cnode3 ~]$

 

Status of the database instances and the services currently running on the nodes.

 

[oracle@12cnode3 ~]$ srvctl status database -db srprim -v -f
Instance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.
Instance srprim2 is running on node 12cnode2. Instance status: Open.
Instance srprim3 is running on node 12cnode3. Instance status: Open.
[oracle@12cnode3 ~]$

 

Configuration of the database that is currently running on the cluster nodes.

 

[oracle@12cnode3 ~]$ srvctl config database -db srprim
Database unique name: srprim
Database name: srprim
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381
Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services: srprim_i03
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: srprim1,srprim2,srprim3
Configured nodes: 12cnode1,12cnode2,12cnode3
Database is administrator managed

 

Below is the configuration of the service configured for the “srprim” database.

 

[oracle@12cnode3 ~]$ srvctl config service -s srprim_i03 -db srprim
Service name: srprim_i03
Server pool:
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: srpdb1
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
GSM Flags: 0
Service is enabled
Preferred instances: srprim3
Available instances: srprim1,srprim2

 

If you notice above, the service srprim_i03 is configured to run preferably on srprim3 instance (node 12cnode3). Hence we need to modify the “preferred instances” for service srprim_i03 else we would face error with message stating that “Cannot delete selected instance since it belongs to following services ‘srprim_i03’ as only preferred instance. Modify the services and try again.”

 

So, let’s modify the preferrend instances list for “srprim_i03” service as “srprim1”, and “srprim2”.

 

[oracle@12cnode3 ~]$ srvctl modify service -service srprim_i03 -db srprim -n -i srprim1,srprim2
[oracle@12cnode3 ~]$ srvctl status service -s srprim_i03 -db srprim
Service srprim_i03 is running on instance(s) srprim1

 

Stop and start the service to get the new configuration in place.

 

[oracle@12cnode3 ~]$ srvctl stop service -s srprim_i03 -db srprim
[oracle@12cnode3 ~]$ srvctl start service -s srprim_i03 -db srprim
[oracle@12cnode3 ~]$ srvctl status service -s srprim_i03 -db srprim
Service srprim_i03 is running on instance(s) srprim1,srprim2

 

[oracle@12cnode1 ~]$ srvctl status database -db srprim -v -f
Instance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.
Instance srprim2 is running on node 12cnode2 with online services srprim_i03. Instance status: Open.
Instance srprim3 is running on node 12cnode3. Instance status: Open.

 

Now all looks good, so let’s move on with the node deletion. Removing node from a cluster involves 2 steps:

 

1. Removing any database instance residing on the node that needs to be deleted.
2. Removing the node from the cluster.

 

Now, first, let’s remove the database instance srprim3 residing on the node to be removed 12cnode3.

 

Run dbca from different node of the cluster and not on the node which you are trying to remove. I’m running “dbca” in silent mode here to delete the instance on the node which will be removed (srprim3).

 

[oracle@12cnode1 ~]$ dbca -silent -deleteInstance -gdbName srprim -instanceName srprim3 -nodeList 12cnode3 -sysDBAUserName sys -sysDBAPassword oracle
Deleting instance
1% complete
2% complete
6% complete
13% complete
20% complete
26% complete
33% complete
40% complete
46% complete
53% complete
60% complete
66% complete
Completing instance management.
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/srprim.log" for further details.
[oracle@12cnode1 ~]$

 

Let’s check the database and it’s instances’s status. Instance srprim3 has now been removed from node 12cnode3.

 

[oracle@12cnode1 ~]$ srvctl status database -db srprim -v -f
Instance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.
Instance srprim2 is running on node 12cnode2 with online services srprim_i03. Instance status: Open.
[oracle@12cnode1 ~]$

 

From the database level, it’s clear below that the thread information and the redo logs associated with srprim3 too have been removed.

 

[oracle@12cnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 3 19:19:40 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SYS@srprim1>select group#,thread#,bytes/1024/1024 from v$log;

    GROUP#    THREAD# BYTES/1024/1024
---------- ---------- ---------------
         1          1              50
         2          1              50
         3          2              50
         4          2              50

 

SYS@srprim1>select group#,member from v$logfile order by group#;

    GROUP# MEMBER
---------- ------------------------------------------------------------
         1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571
         1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571
         2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577
         2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575
         3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373
         3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375
         4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377
         4 +FRA/SRPRIM/ONLINELOG/group_4.260.904396379

8 rows selected.

 

SYS@srprim1>select name,value,inst_id from gv$parameter where name='undo_tablespace';

NAME                           VALUE                                                 INST_ID
------------------------------ -------------------------------------------------- ----------
undo_tablespace                UNDOTBS2                                                    2
undo_tablespace                UNDOTBS1                                                    1

 

Now, on the node to be removed, we need to update the inventory file. So, on 12cnode3, we update the inventory file with the database home listing only the node to be removed (12cnode3). This needs to be done only on the node to be removed and is done by running “oui” utility.

 

Current content of the inventory file on 12cnode3:

 

[oracle@12cnode3 ContentsXML]$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode3 ContentsXML]$

 

Updating the database home node list on 12cnode3. Please note that we need to specify the “-local” option to “oui” in order to update the inventory locally.

 

[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/oui/bin/
[oracle@12cnode3 bin]$
[oracle@12cnode3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={12cnode3}" -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10237 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[oracle@12cnode3 bin]$

 

Below is the latest inventory file on 12cnode3.

 

[oracle@12cnode3 ContentsXML]$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode3 ContentsXML]$

 

Once the inventory is updated with the database home nodelist being only the node to be removed on 12cnode3, let’s deinstall RDBMS home on 12cnode3.
Again please note that we need to specify the “-local” option to the “deinstall” utility to remove the RDBMS binary on 12cnode3.

 

[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/deinstall/

[oracle@12cnode3 deinstall]$ ./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DECONFIG TOOL START ############


######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/oracle/product/12.1.0.2/db_1
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: 12cnode3,12cnode2,12cnode1
Checking for sufficient temp space availability on node(s) : '12cnode3'

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2017-09-03_07-37-06-PM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2017-09-03_07-37-16-PM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured locally on this node for this Oracle home. Local configurations of the discovered databases will be removed [srprim3,srprim3]:
Database Check Configuration END
Oracle Configuration Manager check START
OCM check log file location : /u01/app/oraInventory/logs//ocm_check936.log
Oracle Configuration Manager check END

######################### DECONFIG CHECK OPERATION END #########################


####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: 12cnode3,12cnode2,12cnode1
The cluster node(s) on which the Oracle home deinstallation will be performed are:12cnode3
Oracle Home selected for deinstall is: /u01/app/oracle/product/12.1.0.2/db_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
The option -local will not modify any database configuration for this Oracle home.
Checking the config status for CCR
Oracle Home exists with CCR directory, but CCR is not configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2017-09-03_07-38-17-PM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2017-09-03_07-38-17-PM.log

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean936.log
Oracle Configuration Manager clean END

######################### DECONFIG CLEAN OPERATION END #########################


####################### DECONFIG CLEAN OPERATION SUMMARY #######################
Cleaning the config for CCR
As CCR is not configured, so skipping the cleaning of CCR configuration
CCR clean is finished
#######################################################################


############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2017-09-03_07-32-35PM/response/deinstall_2017-09-03_07-36-41-PM.rsp
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL TOOL START ############





####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to 12cnode3
Setting CLUSTER_NODES to 12cnode3
Setting CRS_HOME to false
Setting oracle.installer.invPtrLoc to /tmp/deinstall2017-09-03_07-32-35PM/oraInst.loc
Setting oracle.installer.local to true

## [END] Preparing for Deinstall ##

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/oracle/product/12.1.0.2/db_1' from the central inventory on the local node : Done

Delete directory '/u01/app/oracle/product/12.1.0.2/db_1' on the local node : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is in use by Oracle Home '/u01/app/12.1.0.2/grid'.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2017-09-03_07-32-35PM' on node '12cnode3'

## [END] Oracle install clean ##


######################### DEINSTALL CLEAN OPERATION END #########################


####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u01/app/oracle/product/12.1.0.2/db_1' from the central inventory on the local node.
Successfully deleted directory '/u01/app/oracle/product/12.1.0.2/db_1' on the local node.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL TOOL END #############

[oracle@12cnode3 deinstall]$

 
 

On the remaining nodes of the cluster, if we would check the inventory file, the database home would still list all the nodes. So, we need to update the inventory file on the remaining nodes with the node list under the Database home listing only the remaining nodes.

 

On 12cnode1, let me update the node list under the Database Home using OUI utility.

 

Current content of the inventory file on 12cnode1.

 

[oracle@12cnode1 ~]$ cd /u01/app/oraInventory/ContentsXML/
[oracle@12cnode1 ContentsXML]$ ls -lrt
total 12
-rw-rw----. 1 oracle oinstall 817 Mar 24  2016 inventory.xml
-rw-rw----. 1 oracle oinstall 292 Mar 24  2016 libs.xml
-rw-rw----. 1 oracle oinstall 329 Mar 24  2016 comps.xml
[oracle@12cnode1 ContentsXML]$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode1 ContentsXML]$

 

Update the Database Home nodelist using oui with the cluster_nodes option listing the remaining nodes of the cluster.

 

[oracle@12cnode1 ~]$ . oraenv
ORACLE_SID = [srprim1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@12cnode1 ~]$ cd $ORACLE_HOME/oui/bin
[oracle@12cnode1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={12cnode1,12cnode2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8125 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[oracle@12cnode1 bin]$

 

Once the inventory is updated, let’s verify the contents of the inventory file.

 

[oracle@12cnode1 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode1 bin]$

 

So, we have now successfully removed the database instance and the RDBMS home from 12cnode3. Let me now proceed with removing the node from the cluster.
Just briefing up, the “olsnodes” is listing out node 12cnode3. Further actions involve deconfiguring the cluster resources on 12cnode3 and then removing the GI home.

 

[oracle@12cnode1 ~]$ . oraenv
ORACLE_SID = [srprim1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ olsnodes -s
12cnode1        Active
12cnode2        Active
12cnode3        Active
[oracle@12cnode1 ~]$

 

[oracle@12cnode3 ~]$ olsnodes -s
12cnode1        Active
12cnode2        Active
12cnode3        Active
[oracle@12cnode3 ~]$

 

On 12cnode3, deconfigure the cluster resources by running “rootcrs.pl” script with “deconfig” option from the GI home.

 

[root@12cnode3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM3
The Oracle base has been set to /u01/app/oracle
[root@12cnode3 ~]#
[root@12cnode3 ~]# cd /u01/app/12.1.0.2/grid/crs/install
[root@12cnode3 install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node 12cnode1
VIP Name: 12cnode1-vip.mydomain
VIP IPv4 Address: 192.168.0.117
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node 12cnode2
VIP Name: 12cnode2-vip.mydomain
VIP IPv4 Address: 192.168.0.118
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node 12cnode3
VIP Name: 12cnode3-vip
VIP IPv4 Address: 192.168.0.121
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12cnode3'
CRS-2673: Attempting to stop 'ora.crsd' on '12cnode3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode3'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode3'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode3' has completed
CRS-2677: Stop of 'ora.crsd' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.storage' on '12cnode3'
CRS-2673: Attempting to stop 'ora.ctssd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.mdnsd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.gpnpd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12cnode3'
CRS-2677: Stop of 'ora.storage' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '12cnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.asm' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.ctssd' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.evmd' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode3'
CRS-2677: Stop of 'ora.mdnsd' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.gpnpd' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '12cnode3'
CRS-2677: Stop of 'ora.cssd' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.crf' on '12cnode3'
CRS-2677: Stop of 'ora.crf' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '12cnode3'
CRS-2677: Stop of 'ora.gipcd' on '12cnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12cnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/03 20:05:03 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2017/09/03 20:05:37 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2017/09/03 20:05:40 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

[root@12cnode3 install]#

 

Now, remove the node from the cluster using the “crsctl” utility.

 

[root@12cnode1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@12cnode1 ~]#
[root@12cnode1 ~]# crsctl delete node -n 12cnode3
CRS-4661: Node 12cnode3 successfully deleted.
[root@12cnode1 ~]#

 

The following steps involve removing the GI home which is similar to what we followed while removing the RDBMS home.
Firstly, update the inventory file on 12cnode3 with the information that this node is lonely and not part of the cluster. This is done by running the OUI utility from GI home.

 

Options passed to OUI are as below:

 

ORACLE_HOME: GI home
CLUSTER_NODES: 12cnode3 (The node to be removed)
CRS=TRUE: as we are dealing with GI home and not RDBMS home
local: which defines that the update needs to be done only on this node.

 

[oracle@12cnode3 ~]$ id
uid=501(oracle) gid=502(oinstall) groups=502(oinstall),501(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@12cnode3 ~]$
[oracle@12cnode3 ~]$ . oraenv
ORACLE_SID = [+ASM3] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@12cnode3 ~]$
[oracle@12cnode3 ~]$ cd $ORACLE_HOME/oui/bin
[oracle@12cnode3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={12cnode3}" CRS=TRUE -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10237 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

 

Verfiy the contents of the inventory file on 12cnode3.

 

[oracle@12cnode3 bin]$
[oracle@12cnode3 bin]$
[oracle@12cnode3 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode3"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode3 bin]$

 

Now, I can deinstall the GI home on 12cnode3 by running the “deinstall” script from the GI home.

 

[oracle@12cnode3 ~]$ echo $ORACLE_SID
+ASM3
[oracle@12cnode3 ~]$ cd /u01/app/12.1.0.2/grid/deinstall/
[oracle@12cnode3 deinstall]$ ./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2017-09-03_08-13-07PM/logs/

############ ORACLE DECONFIG TOOL START ############


######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/12.1.0.2/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: 12cnode3
Checking for sufficient temp space availability on node(s) : '12cnode3'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2017-09-03_08-13-07PM/logs//crsdc_2017-09-03_08-17-00PM.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/netdc_check2017-09-03_08-17-02-PM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/asmcadc_check2017-09-03_08-17-02-PM.log

Database Check Configuration START

Database de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/databasedc_check2017-09-03_08-17-02-PM.log

Database Check Configuration END

######################### DECONFIG CHECK OPERATION END #########################


####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: 12cnode3
The cluster node(s) on which the Oracle home deinstallation will be performed are:12cnode3
Oracle Home selected for deinstall is: /u01/app/12.1.0.2/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Option -local will not modify any ASM configuration.
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/databasedc_clean2017-09-03_08-17-55-PM.log
ASM de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/asmcadc_clean2017-09-03_08-17-55-PM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/netdc_clean2017-09-03_08-17-55-PM.log

Network Configuration clean config END


######################### DECONFIG CLEAN OPERATION END #########################


####################### DECONFIG CLEAN OPERATION SUMMARY #######################
Oracle Clusterware was already stopped and de-configured on node "12cnode3"
Oracle Clusterware is stopped and de-configured successfully.
#######################################################################


############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2017-09-03_08-13-07PM/response/deinstall_2017-09-03_08-16-54-PM.rsp
Location of logs /tmp/deinstall2017-09-03_08-13-07PM/logs/

############ ORACLE DEINSTALL TOOL START ############





####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to 12cnode3
Setting CLUSTER_NODES to 12cnode3
Setting CRS_HOME to true
Setting oracle.installer.invPtrLoc to /tmp/deinstall2017-09-03_08-13-07PM/oraInst.loc
Setting oracle.installer.local to true

## [END] Preparing for Deinstall ##

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/12.1.0.2/grid' from the central inventory on the local node : Done

Failed to delete the directory '/u01/app/12.1.0.2/grid/dc_ocm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/has'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/ohasd/12cnode3'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/ohasd'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/crs/12cnode3'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/crs'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/css/12cnode3'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/css'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/evm/12cnode3'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/evm'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/auth'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/wwg'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/diagnostics'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/sqlplus'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/demo'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/jdk'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/jlib'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ord'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/ologgerd/init/12cnode3.pid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/ologgerd/init/12cnode3'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ologgerd/init'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ologgerd'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/cdata'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_start.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclsvmon'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scrctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sysresv'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orald'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dg4pwdO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/patchgen'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_register.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraenv'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsecho'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsallociO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmsort'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gsd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmcmdcore'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adrci'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osysmond.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapbind'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjob'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmodify'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdtool'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rmanO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clscfg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrdump'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/bndlchk'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scriptagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appagent.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsdepxaO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssdrim'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjobO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appvipcfg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlplus'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrconfig.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crstmpl.scr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rman'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tkprofO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgid0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/skgxpinfo'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssvfupgd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbgeu_run_action.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrcheck.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcroute'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ghappctl.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netmgr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diskmon'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfod'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbshut'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkpatch'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcasst'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aqxmlctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/renamedg0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emcrsp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsdiag.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadpsp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluutil'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gnsd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dg4pwd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapki'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oprocd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmshow.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sysresv0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrcheck'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsctl.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrpatch'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/nidO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologgerd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_relocate'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sclsspawn'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmoddn'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbstart'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlldrO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdmonitor'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgwrap'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clscfg.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agapacheas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sbttestO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okadriverstate'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnsping'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsdriverstate'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologdbg.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tkprof'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/octssd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmwatch'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfssinglefsmount'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osdbagrp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgvip'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgidwrap'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnrctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okinit0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmshow'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnptool.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsid.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/nid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evt.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmlogger'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cursize'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkstore'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impdpO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_preapply'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_start'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ojvmtc'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/jssu'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclsomon'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_apply'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssvfupgd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgeut'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmklib.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsalloci'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapcompare'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gipcd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crswrapexece.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agsiebsrvras'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtomcatas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradnssd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oracle'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osysmond'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologdbg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdroot'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oranetmonitor.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/olsnodes'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdagent.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agsiebgtwyas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutls'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impdp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnodes'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mapsga'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrap'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmpost.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dsml2ldif'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tstshm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadjava'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnsping0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/amdu'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lbuilder'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_monitor'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oerr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsotsO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/uidrvciO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcldr0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emdwgrd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extprocO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oc4jctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dropjava'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmkbin.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ghappctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/symfind'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/maxmem'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsdb.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adapters'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ndfnceca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odnsd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gensyslib'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okdstry0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evminfo'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapwdO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmkbin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/fmputlhp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ohasd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cursizeO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/unzip'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extproc'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/schema'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genorasdksh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odnsd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kgmgrO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tstshmO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/usrvip'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sbttest'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genoccish'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oc4jctl.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpsappas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapki.bat'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_apply.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnpd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aggoldengateas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genclntsh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapwd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluvfy'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mapsga0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/onsctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkstore.bat'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmproxy'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oifcfg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapdelete'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsregistrymount'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldifmigrator'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnnfg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genclntst'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbua'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagsetup'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutlo.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_profile'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_getperm.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mdnsd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkpatchO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oerr.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ncomp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stat'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraagent.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/schemasync'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rawutl0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/coraenv'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expdpO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/srvctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oifcfg.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfod.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extusrupgrade'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbhome'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfsizeO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/uidrvci'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutlo'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_unregister.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okinit'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsdb'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapaddmt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmwatch.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orabase'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/linkshlib'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okdstry'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/owm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtctlO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crfsetenv'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_unregister'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpsbatchas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odig.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpspiaas'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/statusnc'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxchknlb'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expdp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/olsnodes.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orarootagent.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_getperm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orion'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsecho.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/qosctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsots'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjoboO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emdwgrd.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/xmlwf'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_relocate.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mdnsd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnpd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradaemonagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/umu'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gnsd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kgmgr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagcollection.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradism'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gennttab'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dumpsga'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadpspO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genagtsh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraxsl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/octssd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clssproxy.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/deploync'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appvipcfg.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcroute0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aqxmlctl.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_profile.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genezi'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rconfig'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dgmgrlO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/renamedg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrcO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclskd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmsort.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxegen'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odisrvreg'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdmonitor.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evminfo.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutls.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrconfig'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfed'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrc'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orionO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/eusm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ohasd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcsess'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdtool.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rhpctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_register'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlldr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rdtool'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/echodo'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsdepxa'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ojvmjava'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/vipca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/relink'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapsearch'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/exp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adrciO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/amduO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmcmd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmpost'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oidca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfs_client'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdload'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmlogger.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oracleO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genksms'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oranetmonitor'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluvfyrac.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfsize'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnodes.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcldr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnslsnr0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diskmon.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stop.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/plshprofO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrapO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emcrsp.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclskd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsreplcrs'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraping'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lmsgen'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netca_deinst.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gipcd.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oklist0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orajaxb'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scriptagent.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dumpsga0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oidprovtool'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_setperm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clssproxy'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgevtf'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/imp'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gennfgt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgmain'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsctl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oklist'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afddriverstate'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnslsnr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssdrim.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/maxmemO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lcsscan'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orarootagent'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsroot'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osdbagrp0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrdump.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfedO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradnssd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dgmgrl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjobo'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraxml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_initializer'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagcollection.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfodO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stat.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/zip'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_transport'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsload'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/platform_common'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapadd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/plshprof'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbv'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmklib'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnptool'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmodifymt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxinst'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okaload'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/fmputl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odig'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/srvconfig'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/skgxpinfoO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_setperm.bin'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/geneziO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rawutl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mgmtca'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbvO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stop'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okaroot'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdboot'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnrctl0'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/bin'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/javavm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/xdk'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/oracore'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ucp'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/ctss/init/12cnode3.pid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/ctss/init/12cnode3'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ctss/init'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ctss'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/osysmond/init/12cnode3.pid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/osysmond/init/12cnode3'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/osysmond/init'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/osysmond'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/install'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/assistants'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/profiles/peer'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/profiles'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/pa'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/root'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/prdr'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/peer'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/profiles/peer'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/profiles'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/pa'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/root'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/prdr'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/peer'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/tfa'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/racg'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/deinstall'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/cfgtoollogs'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/inventory'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ohasd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/sqlpatch'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/dbs'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/root.sh'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/OPatch'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/clone'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gipc'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/utl'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/admin/crf12cnode3.cfg'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run/crflogd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run/crfmond'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/admin/crf12cnode3.ora'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.l01'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.l02'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/localdump.hdr'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.log'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db/12cnode3'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crf'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/srvm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/cv'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/owm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/addnode'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/oc4j'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gnsd/init'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/gnsd'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/nls'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/scheduler'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/ldap'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/usm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/jdbc'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/eons'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmcred12.so'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/lib/stubs'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libba12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcorejava.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmddisk.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdlib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxn2.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/clntshcore.map'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntshcore.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfstoolsdriver.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/s0main.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.la'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcore12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnls12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboramysql12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcommon12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libuini12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpatchgensh12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsroot.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordimt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgx12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naeet.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nnfgt.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvm12.so0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libosbws12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcell12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfssinglefsmount.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.11.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagtsh.so.1.0'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippdcemerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgns12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_afdlib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskvol12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpksse4212.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdambr.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpd.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdroot.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/sscoreed.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsnls12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnoname12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdaudit.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcp12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcpaio12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplc12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsqlplus.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/librdjni12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_std.a.dbl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocijdbc12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnsgr12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntns12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcommand.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodm12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx212.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnetd12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libkubsagt12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpg.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsvml.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libacfs12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libwwg.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libonsx.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_base_non_fips.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgnsjni12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okaroot.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/activation.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmm.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcryptocme.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnz12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdlib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_asym.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libons.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippdcmerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipp_bz2.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpcompat.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxns.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libxml12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okalib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxp12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.12.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocrutl12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_accel_non_fips.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_base.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdtoolsdriver.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci.so.12.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmddiag.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnid.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcell12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/facility.lis'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libowm2.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdload.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcrf12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpsa12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libheteroxa12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdvol.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnzjs12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboraz.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfslib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnbeq12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_accel_fips.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2_sans_jaxp_services.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodm12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclnt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libeonsserver.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/oc4jctl_lib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdparser.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.10.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdamdu.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmperl12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcryptocme.sig'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2_jaxp_services.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdbase.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afddriverstate.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okadriverstate.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplp12_pic.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdanlz.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordim12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libirc.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libn12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntshcore.so.12.1'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgeneric12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsr12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nigcon.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okadriverstate.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsload.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapclnt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsregistrymount.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpls12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/sysliblist'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xschema.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okatoolsdriver.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libzx12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdroot.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libons.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_cee.a.dbl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntst12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nautab.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ldflagsO'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsqlplus.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nigtab.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippsemerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libncrypt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplp12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocr12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippsmerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvm12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liborion12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libzt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpr.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnus12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclntsh12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsreplcrs.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntmq12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcps12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libeons.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnz12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libdbcfg12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ntcontab.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagfw12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ldflags'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnro12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdexceptions.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libqsmashr.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_okalib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnque12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcxaguard.so.5'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcore.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdtmpl.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclib_jiio.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsregistrymount.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordim12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmhas12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdglobal.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libimf.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclient12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libztkg12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpls12_pic.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcrf_mdb12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/oc4jctl_common.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_cse.a.dbl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlmesg.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libxdb.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afddriverstate.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnhost12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_non_fips.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsroot.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okaload.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmql1.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapjclnt12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libhasgen12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naedhs.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsdriverstate.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipp_z.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapjclnt12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsql12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplc12_pic.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnl12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libserver12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libctxc12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osntabst.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/scorept.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/http_client.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcpemerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagent12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdattr.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclntsh12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnjni12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libctx12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpatchgensh12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdpasswd.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodmd12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboramysql12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdug.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnzst12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskjcx12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlcomp2.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okaroot.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libslax12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocrb12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdload.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libintlc.so.5'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsce12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okalib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdshare.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xsu12.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnfsodm12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libavstub12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcpmerged.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfslib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/clntsh.map'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfssinglefsmount.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liblxled.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx212.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboevm.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpksse4212.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnet12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsdriverstate.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagtsh.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipc1.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsload.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liborabz2.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordsdo12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naect.o'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so.1.5.2'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdsys.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_acfslib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmperl12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libafd12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmocr12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsra12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xml.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libunls12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/mail.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdxmlexceptions.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnldap12.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libusmcrs12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libopc12.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmql1.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmd_disk_header_format'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipc1.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okaload.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liblzopro.a'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/lib/s_oc4jctl_lib.pm'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/lib'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/plsql'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/acfs'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/diskmon'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/admin'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/log/12cnode3/alert12cnode3.log'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/ctssd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/client'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gipcd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crsd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/racg'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/ohasd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gpnpd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/evmd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crflogd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/mdnsd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/srvm'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gnsd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/cssd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/afd'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crfmond'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/xag'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/log'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/auth'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/demo'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/script'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/config'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/install'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/public'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/utl/12cnode3/crsconfig_fileperms'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/utl/12cnode3/crsconfig_dirs'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/utl/12cnode3'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/utl'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/trace'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/ohasd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/oka'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/init.ohasd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/12cnode3.pid'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/afd.sles'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/ohasd.sles'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/afd'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/12cnode3'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/init'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/lib'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/log'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/profile'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/mesg'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/diskmon.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/gpnp.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/cssd.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/asm.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/mdns.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/evm.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/drivers.acfs.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/daemon.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/application.tdf'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/ctss.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/storage.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/ohasdbase.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/generic.tdf'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/appvipx.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/TYPE_application.cap'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/crf.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/TYPE_generic.cap'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/driver.afd.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/oka.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/haip.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/gipc.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/registry.acfs.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/cssdmonitor.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/appvip.type'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/crs.type'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/template'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/sbs'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/crs'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/precomp'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/network'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/hs'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/dmu'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/opmn'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/instantclient'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/css'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/md'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/QOpatch'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/rest'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/evm'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/usableports.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/runstatus.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/.buildid'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jewt4.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/je-4.1.27.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/ojdbc6.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/share.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jdev-rt.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/Symlink.so'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jsch.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/RATFA.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/je-5.0.84.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/commons-io-2.2.jar'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/public.jks'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/getppid.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfasetup.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable/Wrap.pm'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable.pm'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/discover_ora_stack.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/profiling.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl_lib.pm'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/patchtfa.sh'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/collectfiles.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/metric_iorm.pl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.sunos'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.conf'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.aix'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.service'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.hpux'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/init.tfa.tmpl'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.linux'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa.jks'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa_directories.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/bin/tnt'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/bin'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/conf/tnt.prop.tmpl'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/conf'. The directory is not empty.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-cli-1.0.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-lang-2.1.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xmlparserv2.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xternal.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-logging-1.1.jar'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/tnt.jar'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/input'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa_setup.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa.md5'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/problemset.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/tasks.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/ignore_extensions.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/searchStrings.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/collect_all_directories.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/date_patterns.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns_internal.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/ignorefiles.txt'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/scanFileList.xml'. The file is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/mask_strings.xml'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/orachk'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools'. The directory is not empty.
Failed to delete the directory '/u01/app/12.1.0.2/grid/xag'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/perl'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/wlm'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/rootupgrade.sh'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/relnotes'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/mdns'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/rdbms'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/slax'. The directory is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid/oui'. The directory is in use.
Failed to delete the file '/u01/app/12.1.0.2/grid/oraInst.loc'. The file is in use.
Failed to delete the directory '/u01/app/12.1.0.2/grid'. The directory is not empty.
Delete directory '/u01/app/12.1.0.2/grid' on the local node : Failed <<<<

Delete directory '/u01/app/oraInventory' on the local node : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2017-09-03_08-13-07PM' on node '12cnode3'

## [END] Oracle install clean ##


######################### DEINSTALL CLEAN OPERATION END #########################


####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u01/app/12.1.0.2/grid' from the central inventory on the local node.
Failed to delete directory '/u01/app/12.1.0.2/grid' on the local node.
Successfully deleted directory '/u01/app/oraInventory' on the local node.
Oracle Universal Installer cleanup was successful.


Run 'rm -r /etc/oraInst.loc' as root on node(s) '12cnode3' at the end of the session.

Run 'rm -r /opt/ORCLfmap' as root on node(s) '12cnode3' at the end of the session.
Run 'rm -r /etc/oratab' as root on node(s) '12cnode3' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL TOOL END #############

[oracle@12cnode3 deinstall]$

 

The above error message can be ignored to remove the GI home.
Now, update the inventory file on the remaining nodes that the node list does not contain 12cnode3.

 

[oracle@12cnode1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ cd $ORACLE_HOME/oui/bin
[oracle@12cnode1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={12cnode1,12cnode2}" CRS=TRUE -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8126 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[oracle@12cnode1 bin]$

 

Just verifying the inventory file post update.

 

[oracle@12cnode1 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.2/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="12cnode1"/>
      <NODE NAME="12cnode2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@12cnode1 bin]$

 

“olsnodes” too now just shows 12cnode1 and 12cnode2 as the nodes of the cluster and not 12cnode3.

 

[oracle@12cnode1 bin]$ olsnodes -s
12cnode1        Active
12cnode2        Active
[oracle@12cnode1 bin]$

 

Verify the cluster resources status from any one of the existing nodes.

 

[oracle@12cnode1 bin]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
ora.net1.network
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
ora.ons
               ONLINE  ONLINE       12cnode1                 STABLE
               ONLINE  ONLINE       12cnode2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.12cnode1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.12cnode2.vip
      1        ONLINE  ONLINE       12cnode2                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       12cnode1                 169.254.3.189 192.16
                                                             8.1.107,STABLE
ora.asm
      1        ONLINE  ONLINE       12cnode1                 Started,STABLE
      3        ONLINE  ONLINE       12cnode2                 Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       12cnode1                 STABLE
ora.srprim.db
      1        ONLINE  ONLINE       12cnode1                 Open,STABLE
      2        ONLINE  ONLINE       12cnode2                 Open,STABLE
ora.srprim.srprim_i03.svc
      1        ONLINE  ONLINE       12cnode1                 STABLE
      2        ONLINE  ONLINE       12cnode2                 STABLE
--------------------------------------------------------------------------------
[oracle@12cnode1 bin]$

 

We can run the “cluvfy” utility to see if any messages are reported by post node deletion.

 

[oracle@12cnode1 ~]$ cluvfy stage -post nodedel -n 12cnode3 -verbose

Performing post-checks for node removal

Checking CRS integrity...
The Oracle Clusterware is healthy on node "12cnode1"

CRS integrity check passed

Clusterware version consistency passed.
Result:
Node removal check passed

Post-check for node removal was successful.
[oracle@12cnode1 ~]$

 

All looks good now and the node is removed from the cluster.

 

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

 

September 9, 2017 / Shivananda Rao P

Flex ASM configuration in Oracle 12c

This article speaks of how Flex ASM is configured and the advantage of having it. Flex ASM feature was introduced in 12c which allows the ASM instance to run on different node and not intended to run on each of the nodes in the cluster. This is configured based on the cardinality of the ASM. In versions prior to 12c, the ASM instance needed to be run on each of the nodes of the cluster. If the ASM instance was unable to start on any of the nodes, then the associated database instances too could not be started. But, from 12c, this is no more a concern.

 

Let’s see how Flex ASM can be configured and how it works.

 

Environment:

 

RAC nodes : 12cnode1, 12cnode2, 12cnode3
Cluster version: 12.1.0.2
Database Name: srprim
Instance Name: srprim1 on 12cnode1, srprim2 on 12cnode2, srprim3 on 12cnode3
OS Platform: OEL 6

 

Below shows the list of nodes within the cluster.

 

[oracle@12cnode1 ~]$ olsnodes
12cnode1
12cnode2
12cnode3
[oracle@12cnode1 ~]$

 

Check the Cluster mode through ASMCMD. It’s obvious that Flex mode would be disabled.

 

[oracle@12cnode1 ~]$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode disabled

 

Flex ASM requires a separate listener called ASMLISTENER to be configured on a different port number which is not being used by any other Listener. The other important thing is that FLEX ASM requires a separate network with which the ASM instances and it’s clients communicate. You can also make use of the private network ethernet (used for inter node communication) as the network for the ASM instances and it’s clients to communicate. In my case, I’m using the private network itself as ASM network too.

 

We now need to convert the Normal ASM to FlexASM. This can be done via ASMCA utility. Here I’m doing this using “ASMCA” in silent mode and not through GUI.

 

The options passed to ASMCA are:

 

-silent: to run in silent mode
-convertToFlexASM : to convert to FlexASM
-asmNetworks: ASM Network to be used in the form of "interface_name/Subnet"
-asmListenerPort: ASM Listener Port number to be used

 

Run this command on one node of the cluster.

 

[oracle@12cnode1 ~]$ asmca -silent -convertToFlexASM -asmNetworks eth1/192.168.1.0 -asmListenerPort 1526

To complete ASM conversion, run the following script as privileged user in local node.
/u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh

 

Once done, as directed we need to run the “converttoFlexASM.sh” script as ROOT user on the node where ASMCA was run.
This script configures ASM NET LISTENER on all the nodes of the cluster as well as stops and starts all cluster resources on all the nodes of the cluster sequentially.

 

[root@12cnode1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
CRS-2673: Attempting to stop 'ora.crsd' on '12cnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '12cnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '12cnode1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on '12cnode1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on '12cnode1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on '12cnode1'
CRS-2673: Attempting to stop 'ora.srprim.db' on '12cnode1'
CRS-2677: Stop of 'ora.FRA.dg' on '12cnode1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on '12cnode1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.12cnode1.vip' on '12cnode1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on '12cnode1'
CRS-2677: Stop of 'ora.12cnode1.vip' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.12cnode1.vip' on '12cnode3'
CRS-2677: Stop of 'ora.mgmtdb' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on '12cnode1'
CRS-2677: Stop of 'ora.scan1.vip' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on '12cnode3'
CRS-2677: Stop of 'ora.MGMTLSNR' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on '12cnode3'
CRS-2677: Stop of 'ora.srprim.db' on '12cnode1' succeeded
CRS-2676: Start of 'ora.12cnode1.vip' on '12cnode3' succeeded
CRS-2676: Start of 'ora.scan1.vip' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on '12cnode3'
CRS-2676: Start of 'ora.MGMTLSNR' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on '12cnode3'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on '12cnode3' succeeded
CRS-2676: Start of 'ora.mgmtdb' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '12cnode1'
CRS-2677: Stop of 'ora.ons' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '12cnode1'
CRS-2677: Stop of 'ora.net1.network' on '12cnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode1' has completed
CRS-2677: Stop of 'ora.crsd' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '12cnode1'
CRS-2673: Attempting to stop 'ora.evmd' on '12cnode1'
CRS-2673: Attempting to stop 'ora.storage' on '12cnode1'
CRS-2677: Stop of 'ora.storage' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '12cnode1'
CRS-2677: Stop of 'ora.ctssd' on '12cnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on '12cnode1' succeeded
CRS-2677: Stop of 'ora.asm' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '12cnode1'
CRS-2677: Stop of 'ora.cssd' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on '12cnode1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode1'
CRS-2676: Start of 'ora.evmd' on '12cnode1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '12cnode1'
CRS-2672: Attempting to start 'ora.diskmon' on '12cnode1'
CRS-2676: Start of 'ora.diskmon' on '12cnode1' succeeded
CRS-2676: Start of 'ora.cssd' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '12cnode1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode1'
CRS-2676: Start of 'ora.ctssd' on '12cnode1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on '12cnode1'
CRS-2676: Start of 'ora.asm' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12cnode1'
CRS-2676: Start of 'ora.storage' on '12cnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '12cnode1'
CRS-2676: Start of 'ora.crsd' on '12cnode1' succeeded
Oracle Grid Infrastructure restarted in node 12cnode1
PRCC-1014 : ASMNET1LSNR_ASM was already running
PRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already running
PRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnr
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode2'
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode1'
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode3'
ASM listener ASMNET1LSNR_ASM running already
CRS-2673: Attempting to stop 'ora.crsd' on '12cnode2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '12cnode2'
CRS-2673: Attempting to stop 'ora.cvu' on '12cnode2'
CRS-2673: Attempting to stop 'ora.oc4j' on '12cnode2'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode2'
CRS-2673: Attempting to stop 'ora.srprim.db' on '12cnode2'
CRS-2677: Stop of 'ora.cvu' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on '12cnode1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '12cnode2' succeeded
CRS-2677: Stop of 'ora.srprim.db' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.12cnode2.vip' on '12cnode2'
CRS-2677: Stop of 'ora.12cnode2.vip' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.12cnode2.vip' on '12cnode3'
CRS-2676: Start of 'ora.12cnode2.vip' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cvu' on '12cnode1' succeeded
CRS-2677: Stop of 'ora.oc4j' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on '12cnode1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on '12cnode2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on '12cnode2'
CRS-2677: Stop of 'ora.DATA.dg' on '12cnode2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on '12cnode2' succeeded
CRS-2676: Start of 'ora.oc4j' on '12cnode1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '12cnode2'
CRS-2677: Stop of 'ora.ons' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '12cnode2'
CRS-2677: Stop of 'ora.net1.network' on '12cnode2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode2' has completed
CRS-2677: Stop of 'ora.crsd' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '12cnode2'
CRS-2673: Attempting to stop 'ora.evmd' on '12cnode2'
CRS-2673: Attempting to stop 'ora.storage' on '12cnode2'
CRS-2677: Stop of 'ora.storage' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '12cnode2'
CRS-2677: Stop of 'ora.evmd' on '12cnode2' succeeded
CRS-2677: Stop of 'ora.ctssd' on '12cnode2' succeeded
CRS-2677: Stop of 'ora.asm' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '12cnode2'
CRS-2677: Stop of 'ora.cssd' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode2'
CRS-2672: Attempting to start 'ora.evmd' on '12cnode2'
CRS-2676: Start of 'ora.evmd' on '12cnode2' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '12cnode2'
CRS-2672: Attempting to start 'ora.diskmon' on '12cnode2'
CRS-2676: Start of 'ora.diskmon' on '12cnode2' succeeded
CRS-2676: Start of 'ora.cssd' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '12cnode2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode2'
CRS-2676: Start of 'ora.ctssd' on '12cnode2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.asm' on '12cnode2'
CRS-2676: Start of 'ora.asm' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12cnode2'
CRS-2676: Start of 'ora.storage' on '12cnode2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '12cnode2'
CRS-2676: Start of 'ora.crsd' on '12cnode2' succeeded
Oracle Grid Infrastructure restarted in node 12cnode2
CRS-2673: Attempting to stop 'ora.crsd' on '12cnode3'
CRS-2677: Stop of 'ora.crsd' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.evmd' on '12cnode3'
CRS-2673: Attempting to stop 'ora.storage' on '12cnode3'
CRS-2677: Stop of 'ora.storage' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '12cnode3'
CRS-2677: Stop of 'ora.evmd' on '12cnode3' succeeded
CRS-2677: Stop of 'ora.ctssd' on '12cnode3' succeeded
CRS-2675: Stop of 'ora.asm' on '12cnode3' failed
CRS-2679: Attempting to clean 'ora.asm' on '12cnode3'
CRS-2681: Clean of 'ora.asm' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '12cnode3'
CRS-2677: Stop of 'ora.cssd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.evmd' on '12cnode3'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode3'
CRS-2676: Start of 'ora.evmd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '12cnode3'
CRS-2672: Attempting to start 'ora.diskmon' on '12cnode3'
CRS-2676: Start of 'ora.diskmon' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cssd' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '12cnode3'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode3'
CRS-2676: Start of 'ora.ctssd' on '12cnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on '12cnode3'
CRS-2676: Start of 'ora.asm' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12cnode3'
CRS-2676: Start of 'ora.storage' on '12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '12cnode3'
CRS-2676: Start of 'ora.crsd' on '12cnode3' succeeded
Oracle Grid Infrastructure restarted in node 12cnode3
[root@12cnode1 ~]#

 

Now that the execution of the “converttoFlexASM.sh” script is completed, let’s check the cluster mode.
We should see as “Flex mode enabled”.

 

[oracle@12cnode1 ~]$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode enabled

 

Let’s check the configuration of the ASM listener. It’s clear below that the ASM cluster listener is configured on port 1526 and subnet that was specified while running ASMCA.

 

[oracle@12cnode1 ~]$ srvctl config listener -l ASMNET1LSNR_ASM
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 192.168.1.0
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
[oracle@12cnode1 ~]$

 

Let’s check the status of this listener. It needs to be running on all the nodes of the cluster.

 

[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl status listener -l ASMNET1LSNR_ASM
Listener ASMNET1LSNR_ASM is enabled
Listener ASMNET1LSNR_ASM is running on node(s): 12cnode3,12cnode2,12cnode1

 

Also, we shall check the configuration of the ASM to check the cardinality and see if the cluster ASM listener is being listed.

 

[oracle@12cnode1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
[oracle@12cnode1 ~]$

 

We see that ASM cardinality has been set to 3. Since this is a 3 node cluster and in order to check the functionality of FlexASM, I would change the cardinality of ASM to 2. By doing so, only 2 ASM instances will have to run.

 

[oracle@12cnode1 ~]$ srvctl modify asm -count 2
[oracle@12cnode1 ~]$
[oracle@12cnode1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM

 

We can see below that ASM is running only on 2 nodes (12cnode1 and 12cnode3) and no ASM instance is running on 12cnode2.

 

[oracle@12cnode1 ~]$ srvctl status asm
ASM is running on 12cnode3,12cnode1

 

Since ASM is not running on node 12cnode2, let’s check if the associated database instance is running on node 12cnode2 or not.

 

[oracle@12cnode3 ~]$ srvctl status database -db srprim -v -f
Instance srprim1 is running on node 12cnode1. Instance status: Open.
Instance srprim2 is running on node 12cnode2. Instance status: Open.
Instance srprim3 is running on node 12cnode3. Instance status: Open.

 

We can notice above that all the instances of database “srprim” are running on all the 3 nodes of the cluster which means that there is no impact seen on the associated database instance even if ASM is not running on a particular node.

 

So now, that ASM is not running on 12cnode2 but database instance “srprim2” is running, we need to figure out which ASM instance is used by database instance “srprim2”.

 

Let’s check this from ASM3 instance: (anyways I’m using “gv$” views-so can be run on any ASM instance that’s running)

 

SQL> select adg.name,adg.state,ac.instance_name,ac.db_name,adg.inst_id,ac.status,ac.cluster_name from gv$asm_diskgroup adg,gv$asm_client ac where adg.GROUP_NUMBER=ac.GROUP_NUMBER and adg.inst_id=ac.inst_id and ac.db_name='srprim' and adg.name='DATA' order by adg.inst_id;

NAME            STATE       INSTANCE_NAME        DB_NAME     INST_ID STATUS       CLUSTER_NAME
--------------- ----------- -------------------- -------- ---------- ------------ -------------------------------
DATA            MOUNTED     srprim1              srprim            1 CONNECTED    node12c-scan
DATA            MOUNTED     srprim3              srprim            3 CONNECTED    node12c-scan
DATA            MOUNTED     srprim2              srprim            3 CONNECTED    node12c-scan

 

It’s very clear from above that “srprim2” instance is being served by ASM instance 3 on node “12cnode3”. In other words, ASM instance 3 has 2 clients — 1. srprim2 from 12cnode2 and srprim3 from 12cnode3.

 

The same can be viewed from the alert log file of ASM3 instance:

 

NOTE: Flex client id 0x0 [srprim3:srprim:node12c-scan] attempting to connect
NOTE: registered owner id 0x10002 for srprim3:srprim:node12c-scan
NOTE: Flex client srprim3:srprim:node12c-scan registered, osid 23500, mbr 0x0, asmb 23305 (reg:4024785090)
NOTE: client srprim3:srprim:node12c-scan mounted group 1 (DATA)
NOTE: client srprim3:srprim:node12c-scan mounted group 2 (FRA)
Sat Feb 25 12:28:03 2017
NOTE: Flex client id 0x0 [srprim2:srprim:node12c-scan] attempting to connect
NOTE: registered owner id 0x10003 for srprim2:srprim:node12c-scan
NOTE: Flex client srprim2:srprim:node12c-scan registered, osid 23757, mbr 0x0, asmb 24702 (reg:1349866226)
NOTE: client srprim2:srprim:node12c-scan mounted group 1 (DATA)
NOTE: client srprim2:srprim:node12c-scan mounted group 2 (FRA)

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

 

September 6, 2017 / Shivananda Rao P

Manual Upgrade of RAC Database from 11.2.0.3 to 12.1.0.2

In the previous post, we have seen on upgrading the Grid Infrastructure from 11.2.0.3 to 12.1.0.2. In this, we shall see the steps involved in the database upgrade from 11.2.0.3 to 12.1.0.2

 
 

Environment:

 

RAC nodes: drnode1, drnode2
DB Name: srprim
DB Instances: srprim1, srprim2
Current DB version: 11.2.0.3.0
DB to be upgraded to version: 12.1.0.2.0
Cluster Storage used: ASM
Platform: OEL 6
Current DB HOME: /u01/app/oracle/product/11.2.0.3/db_1
New 12c DB HOME: /u01/app/oracle/product/12.1.0.2/db_1

 

Since this is an out-of-place upgrade, firstly install Oracle 12.1.0.2 database software. In this environment, the software here has been unzipped to location “/u03” and then the database software is installed using a response file in silent mode.

 

Here is the response file used to install the 12.1.0.2 database software.

 

[oracle@drnode1 ~]$ cd /u03/database
[oracle@drnode1 database]$ ./runInstaller -silent -responseFile /u02/db_install.rsp -ignoreSysPrereqs
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 32711 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 10229 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-02-21_07-55-11PM. Please wait ...[oracle@drnode1 database]$ You can find the log of this install session at:
 /u01/app/oraInventory/logs/installActions2017-02-21_07-55-11PM.log
The installation of Oracle Database 12c was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2017-02-21_07-55-11PM.log' for more details.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0.2/db_1/root.sh

Execute /u01/app/oracle/product/12.1.0.2/db_1/root.sh on the following nodes:
[drnode1, drnode2]


Successfully Setup Software.

 

Once the 12.1.0.2 database software has been installed, it’s time for us to upgrade our database.

 

Run the “preupgrd.sql” script available from the newly installed oracle 12c home on the 11.2 database. This script performs pre-requisite checks on the database to be upgraded and generates 2 scripts:

 

1. preupgrade_fixups.sql which needs to be run on the database to fix any issues reported while performing the pre-checks.
2. postupgrade_fixups.sql which needs to be run on the database once upgraded to 12.1.0.2 version.

 

Let’s run the “preupgrd.sql” script from the newly installed oracle 12.1.0.2 home (/u01/app/oracle/product/12.1.0.2/db_1) on the database.

 

[oracle@drnode1 ~]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql
-rw-r--r--. 1 oracle oinstall 14083 May 15  2014 /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql

 

[oracle@drnode1 ~]$
[oracle@drnode1 ~]$ sqlplus sys/oracle@srprim as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 21:22:35 2017

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @/u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql



Loading Pre-Upgrade Package...


***************************************************************************
Executing Pre-Upgrade Checks in SRPRIM...
***************************************************************************


      ************************************************************

                  ====>> ERRORS FOUND for SRPRIM <<====

 The following are *** ERROR LEVEL CONDITIONS *** that must be addressed
                    prior to attempting your upgrade.
            Failure to do so will result in a failed upgrade.

           You MUST resolve the above errors prior to upgrade

      ************************************************************

      ************************************************************

              ====>> PRE-UPGRADE RESULTS for SRPRIM <<====

ACTIONS REQUIRED:

1. Review results of the pre-upgrade checks:
 /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade.log

2. Execute in the SOURCE environment BEFORE upgrade:
 /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade_fixups.sql

3. Execute in the NEW environment AFTER upgrade:
 /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/postupgrade_fixups.sql

      ************************************************************

***************************************************************************
Pre-Upgrade Checks in SRPRIM Completed.
***************************************************************************

***************************************************************************
***************************************************************************

 

 

We see above that the preupgrd.sql script generated “preupgrade_fixups.sql” ( to fix any issues reported during pre-check phase) and “postupgrade_fixups.sql” (to be run post the upgrade) scripts that needs to be run on the database. Review the “preupgrade.log” and take necessary actions on the recommendations made.

 

================================================================================================

 

Preupgrade warning log:

[oracle@drnode1 ~]$ cat /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade.log
Oracle Database Pre-Upgrade Information Tool 02-21-2017 21:24:39
Script Version: 12.1.0.2.0 Build: 006
**********************************************************************
   Database Name:  SRPRIM
  Container Name:  Not Applicable in Pre-12.1 database
    Container ID:  Not Applicable in Pre-12.1 database
         Version:  11.2.0.3.0
      Compatible:  11.2.0.0.0
       Blocksize:  8192
        Platform:  Linux x86 64-bit
   Timezone file:  V14
**********************************************************************
                           [Update parameters]
         [Update Oracle Database 11.2.0.3.0 init.ora or spfile]

--> If Target Oracle is 32-bit, refer here for Update Parameters:
WARNING: --> "processes" needs to be increased to at least 300

--> If Target Oracle is 64-bit, refer here for Update Parameters:
WARNING: --> "processes" needs to be increased to at least 300
**********************************************************************
**********************************************************************
                          [Renamed Parameters]
                     [No Renamed Parameters in use]
**********************************************************************
**********************************************************************
                    [Obsolete/Deprecated Parameters]
             [No Obsolete or Desupported Parameters in use]
**********************************************************************
                            [Component List]
**********************************************************************
--> Oracle Catalog Views                   [upgrade]  VALID
--> Oracle Packages and Types              [upgrade]  VALID
--> JServer JAVA Virtual Machine           [upgrade]  VALID
--> Oracle XDK for Java                    [upgrade]  VALID
--> Real Application Clusters              [upgrade]  VALID
--> Oracle Workspace Manager               [upgrade]  VALID
--> OLAP Analytic Workspace                [upgrade]  VALID
--> Oracle Enterprise Manager Repository   [upgrade]  VALID
--> Oracle Text                            [upgrade]  VALID
--> Oracle XML Database                    [upgrade]  VALID
--> Oracle Java Packages                   [upgrade]  VALID
--> Oracle Multimedia                      [upgrade]  VALID
--> Oracle Spatial                         [upgrade]  VALID
--> Expression Filter                      [upgrade]  VALID
--> Rule Manager                           [upgrade]  VALID
--> Oracle Application Express             [upgrade]  VALID
--> Oracle OLAP API                        [upgrade]  VALID
**********************************************************************
                              [Tablespaces]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
     minimum required size: 1225 MB
--> SYSAUX tablespace is adequate for the upgrade.
     minimum required size: 1509 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
     minimum required size: 400 MB
--> TEMP tablespace is adequate for the upgrade.
     minimum required size: 60 MB
--> EXAMPLE tablespace is adequate for the upgrade.
     minimum required size: 310 MB

                      [No adjustments recommended]

**********************************************************************
**********************************************************************
                          [Pre-Upgrade Checks]
**********************************************************************
WARNING: --> Process Count may be too low

     Database has a maximum process count of 150 which is lower than the
     default value of 300 for this release.
     You should update your processes value prior to the upgrade
     to a value of at least 300.
     For example:
        ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE
     or update your init.ora file.

WARNING: --> Enterprise Manager Database Control repository found in the database

     In Oracle Database 12c, Database Control is removed during
     the upgrade. To save time during the Upgrade, this action
     can be done prior to upgrading using the following steps after
     copying rdbms/admin/emremove.sql from the new Oracle home
   - Stop EM Database Control:
    $> emctl stop dbconsole

   - Connect to the Database using the SYS account AS SYSDBA:

   SET ECHO ON;
   SET SERVEROUTPUT ON;
   @emremove.sql
     Without the set echo and serveroutput commands you will not
     be able to follow the progress of the script.

INFORMATION: --> OLAP Catalog(AMD) exists in database

     Starting with Oracle Database 12c, OLAP Catalog component is desupported.
     If you are not using the OLAP Catalog component and want
     to remove it, then execute the
     ORACLE_HOME/olap/admin/catnoamd.sql script before or
     after the upgrade.

INFORMATION: --> Older Timezone in use

     Database is using a time zone file older than version 18.
     After the upgrade, it is recommended that DBMS_DST package
     be used to upgrade the 11.2.0.3.0 database time zone version
     to the latest version which comes with the new release.
     Please refer to My Oracle Support note number 977512.1 for details.

INFORMATION: --> There are existing Oracle components that will NOT be
     upgraded by the database upgrade script.  Typically, such components
     have their own upgrade scripts, are deprecated, or obsolete.
     Those components are:  OLAP Catalog,OWB

INFORMATION: --> Oracle Application Express (APEX) can be
     manually upgraded prior to database upgrade

     APEX is currently at version 3.2.1.00.12 and will need to be
     upgraded to APEX version 4.2.5 in the new release.
     Note 1: To reduce database upgrade time, APEX can be manually
             upgraded outside of and prior to database upgrade.
     Note 2: See MOS Note 1088970.1 for information on APEX
             installation upgrades.


**********************************************************************
                      [Pre-Upgrade Recommendations]
**********************************************************************

                        *****************************************
                        ********* Dictionary Statistics *********
                        *****************************************

Please gather dictionary statistics 24 hours prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:
    EXECUTE dbms_stats.gather_dictionary_stats;

^^^ MANUAL ACTION SUGGESTED ^^^


**********************************************************************
                     [Post-Upgrade Recommendations]
**********************************************************************

                        *****************************************
                        ******** Fixed Object Statistics ********
                        *****************************************

Please create stats on fixed objects two weeks
after the upgrade using the command:
   EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

^^^ MANUAL ACTION SUGGESTED ^^^

**********************************************************************
                   ************  Summary  ************

 0 ERRORS exist in your database.
 2 WARNINGS that Oracle suggests are addressed to improve database performance.
 4 INFORMATIONAL messages that should be reviewed prior to your upgrade.

 After your database is upgraded and open in normal mode you must run
 rdbms/admin/catuppst.sql which executes several required tasks and completes
 the upgrade process.

 You should follow that with the execution of rdbms/admin/utlrp.sql, and a
 comparison of invalid objects before and after the upgrade using
 rdbms/admin/utluiobj.sql

 If needed you may want to upgrade your timezone data using the process
 described in My Oracle Support note 1509653.1
                   ***********************************

 

 

Oracle suggested a few recommendations as seen above that needs to be carried out before performing the upgrade. Let’s try running “fixup.sql” script that got generated previously.

 

Fixup sql:

 

[oracle@drnode1 ~]$ sqlplus sys/oracle@srprim as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 21:50:00 2017

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @/u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade_fixups.sql

 

If the “fixup.sql” script is unable to fix the recommendations made, then fix them manually.

 

Upgrade:

 

Now that the recommendations have been fixed, let’s move with the upgrade.

 

Create a pfile from the spfile on first instance of 11.2.0.3 database to a temporary location and comment out the following parameters.

 

1. instance_number
2. thread
3. 2nd instance's undo tablespace.
4. cluster_database

 


[oracle@drnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 22:10:00 2017

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>
SQL> create pfile='/u02/srprim.ora' from spfile;

File created.

 

[oracle@drnode1 u02]$ cat /u02/srprim.ora | grep ^#
#*.cluster_database=true
#srprim1.instance_number=1
#srprim2.instance_number=2
#srprim2.thread=2
#srprim1.thread=1
#srprim2.undo_tablespace='UNDOTBS2'

 

Stop the database on 11.2.0.3 home

 

[oracle@drnode1 u02]$ srvctl stop database -d srprim

 

Now on the first node, set the environment variables pointing to the newly oracle 12c Home

 

[oracle@drnode1 ~]$ export ORACLE_SID=srprim1
[oracle@drnode1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/db_1
[oracle@drnode1 ~]$ export PATH=$ORACLE_HOME/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin
[oracle@drnode1 ~]$ which sqlplus
/u01/app/oracle/product/12.1.0.2/db_1/bin/sqlplus

 

Start the first instance in upgrade mode from oracle 12c home using the pfile that was created previously in the temporary location.

 

[oracle@drnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Feb 21 22:18:19 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup upgrade pfile='/u02/srprim.ora';
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  2932632 bytes
Variable Size             729809000 bytes
Database Buffers          335544320 bytes
Redo Buffers                5455872 bytes
Database mounted.
Database opened.

 

Now it’s time to run the “catupgrd.sql” script available at 12c ORACLE HOME. Here, I’m running this using the parallel utility “catctl.pl” with 5 number of parallel processes.

 

[oracle@drnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/
[oracle@drnode1 admin]$
[oracle@drnode1 admin]$ nohup /u01/app/oracle/product/12.1.0.2/db_1/perl/bin/perl catctl.pl -n 5 catupgrd.sql > /u02/upgrade.log &

 

Here is the sample of the upgrade process log. Review the log that was spooled for the execution of “catupgrd.sql” and once the execution is completed successfully, copy the pfile and password file of the instances from 11.2.0.3 home to 12.1.0.2 oracle home.
Make sure that the pfile consists of the path to spfile and nothing else.

 

On first node:

 

[oracle@drnode1 admin]$ cat /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim1.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'          # line added by Agent

 

[oracle@drnode1 admin]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/orapwsrprim1 /u01/app/oracle/product/12.1.0.2/db_1/dbs/
[oracle@drnode1 admin]$
[oracle@drnode1 admin]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim1.ora /u01/app/oracle/product/12.1.0.2/db_1/dbs/
[oracle@drnode1 admin]$

 

On second node:

 

[oracle@drnode2 ~]$ cat /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim2.ora
SPFILE='+DATA/srprim/spfilesrprim.ora'          # line added by Agent
[oracle@drnode2 ~]$

 

[oracle@drnode2 ~]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/orapwsrprim2 /u01/app/oracle/product/12.1.0.2/db_1/dbs/
[oracle@drnode2 ~]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim2.ora /u01/app/oracle/product/12.1.0.2/db_1/dbs/
[oracle@drnode2 ~]$

 

Now, start the first instance from new 12c Home and review the parameters. Post that, run the postupgrade_fixups.sql script that was generated earlier.

 

[oracle@drnode1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Feb 22 01:43:55 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  2932632 bytes
Variable Size             775946344 bytes
Database Buffers          289406976 bytes
Redo Buffers                5455872 bytes
Database mounted.
Database opened.
SQL>
SQL> show parameter cluster

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_database                     boolean     TRUE
cluster_database_instances           integer     2
cluster_interconnects                string
SQL>

 

Execution of postupgrade_fixups.sql:

 

SQL> @/u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/postupgrade_fixups.sql


SQL> EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;


SQL> select count(*) from dba_objects where status = 'INVALID';

  COUNT(*)
----------
      6376

SQL> @?/rdbms/admin/utlrp.sql


SQL> select count(*) from dba_objects where status='INVALID';

  COUNT(*)
----------
         0


SQL> @?/rdbms/admin/catuppst.sql		 


SQL> @?/rdbms/admin/utlu121s.sql

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.



CATCTL REPORT = /u01/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/srprim/upgrade/upg_summary.log

PL/SQL procedure successfully completed.



Oracle Database 12.1 Post-Upgrade Status Tool           02-22-2017 02:17:23

Component                               Current         Version  Elapsed Time
Name                                    Status          Number   HH:MM:SS

Oracle Server                             VALID      12.1.0.2.0  00:45:29
JServer JAVA Virtual Machine              VALID      12.1.0.2.0  00:13:46
Oracle Real Application Clusters          VALID      12.1.0.2.0  00:00:06
Oracle Workspace Manager                  VALID      12.1.0.2.0  00:02:58
OLAP Analytic Workspace                   VALID      12.1.0.2.0  00:01:12
OLAP Catalog                         OPTION OFF      11.2.0.3.0  00:00:00
Oracle OLAP API                           VALID      12.1.0.2.0  00:02:09
Oracle XDK                                VALID      12.1.0.2.0  00:02:17
Oracle Text                               VALID      12.1.0.2.0  00:03:09
Oracle XML Database                       VALID      12.1.0.2.0  00:06:44
Oracle Database Java Packages             VALID      12.1.0.2.0  00:00:54
Oracle Multimedia                         VALID      12.1.0.2.0  00:07:37
Spatial                                   VALID      12.1.0.2.0  00:19:07
Oracle Application Express                VALID     4.2.5.00.08  01:02:58
Final Actions                                                    00:05:38
Post Upgrade                                                     00:00:11

Total Upgrade Time: 02:55:35

PL/SQL procedure successfully completed.

SQL>
SQL> --
SQL> -- Update Summary Table with con_name and endtime.
SQL> --
SQL> UPDATE sys.registry$upg_summary SET reportname = :ReportName,
  2                                  con_name = SYS_CONTEXT('USERENV','CON_NAME'),
  3                                  endtime  = SYSDATE
  4         WHERE con_id = -1;

1 row updated.

SQL> commit;

Commit complete.

SQL>

SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

 

From 11.2 home, crosscheck if the database is down and then remove its 11.2 configuration from the clusterware managing. (In this post, I’m removing the 11.2 database configuration from clusterware management and adding it back with 12c database configuration settings. You can avoid removing the old configuration and adding the new one by just running “srvctl upgrade database -db -oraclehome ” from the 12c ORACLE HOME.

 

[oracle@drnode1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/bin/
[oracle@drnode1 bin]$ ./srvctl status database -d srprim
Instance srprim1 is not running on node drnode1
Instance srprim2 is not running on node drnode2

 

[oracle@drnode1 bin]$ ./srvctl remove database -d srprim
Remove the database srprim? (y/[n]) y
[oracle@drnode1 bin]$

 

Now, add this database “srprim” with 12c home configuration to the clusterware managing.

 

[oracle@drnode1 bin]$ cd $ORACLE_HOME/bin
[oracle@drnode1 bin]$ pwd
/u01/app/oracle/product/12.1.0.2/db_1/bin
[oracle@drnode1 bin]$
[oracle@drnode1 bin]$ ./srvctl add database -d srprim -o /u01/app/oracle/product/12.1.0.2/db_1
[oracle@drnode1 bin]$
[oracle@drnode1 bin]$
[oracle@drnode1 bin]$ ./srvctl add instance -i srprim1 -d srprim -n drnode1
[oracle@drnode1 bin]$
[oracle@drnode1 bin]$
[oracle@drnode1 bin]$ ./srvctl add instance -i srprim2 -d srprim -n drnode2

 

[oracle@drnode1 bin]$
[oracle@drnode1 bin]$ ./srvctl status database -d srprim -v -f
Instance srprim1 is not running on node drnode1
Instance srprim2 is not running on node drnode2

 

If there were any services configured previously, then create them as such from 12c ORACLE HOME.
Start the database now using srvctl and also it’s services.

 

 

[oracle@drnode1 bin]$ ./srvctl add service -s srprim_any -d srprim -r srprim1,srprim2
[oracle@drnode1 bin]$ ./srvctl start database -d srprim
[oracle@drnode1 bin]$ ./srvctl start service -s srprim_any -d srprim
[oracle@drnode1 bin]$ ./srvctl status database -d srprim -v -f
Instance srprim1 is running on node drnode1 with online services srprim_any. Instance status: Open.
Instance srprim2 is running on node drnode2 with online services srprim_any. Instance status: Open.

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine

 

August 21, 2017 / Shivananda Rao P

Grid Infrastructure (GI) upgrade from 11.2.0.3 to 12.1.0.2 in silent mode on RAC

In this article, we shall see the steps involved in upgrading the Grid Infrastructure (CRS) from version 11.2.0.3 to 12.1.0.2 in silent mode.

 

Environment:

 

RAC nodes: drnode1, drnode2
Current GI (CRS) version: 11.2.0.3.0
GI to be upgraded to version: 12.1.0.2.0
Cluster Storage used: ASM
Platform: OEL 6
Current CRS HOME: /u01/app/11.2.0.3/grid
New 12c CRS HOME: /u01/app/12.1.0.2/grid

 

 

Below shows the current version of the CRS. The output is shown only for the first node.

 

[oracle@drnode1 ~]$ crsctl query crs softwareversion drnode1
Oracle Clusterware version on node [drnode1] is [11.2.0.3.0]

 

[oracle@drnode1 ~]$ crsctl query crs softwareversion drnode2
Oracle Clusterware version on node [drnode2] is [11.2.0.3.0]

 

[oracle@drnode1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]

 

 

Download oracle 12.1.0.2 Grid Infrastructure and unzip. Here, the software is unzipped under “/u03”.
Let’s run the “cluvfy” utility to perform pre-requisite checks before upgrading. This is achieved by running the “runcluvfy.sh” script from the unzipped 12c Grid Infrastructure software location.

 

The parameters passed to this script are:

 

-pre crsinst: To perform pre-checks before the CRS installation
-upgrade: To perform upgrade pre-checks
-rolling: To perform rolling upgrade
-src_crshome: Location of the Source GI home
-dest_crshome: Location of the Destination GI Home
-dest_version: The version to which GI will be upgraded

 

[oracle@drnode1 ~]$ cd /u03/grid
[oracle@drnode1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/11.2.0.3/grid -dest_crshome /u01/app/12.1.0.2/grid -dest_version 12.1.0.2.0

Performing pre-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "drnode1"


Checking user equivalence...
User equivalence check passed for user "oracle"
Package existence check passed for "cvuqdisk"

Check: Grid Infrastructure home writeability of path /u01/app/12.1.0.2/grid
PRVG-11932 : Path "/u01/app/12.1.0.2/grid" cannot be created on node "drnode1".
PRVG-11932 : Path "/u01/app/12.1.0.2/grid" cannot be created on node "drnode2".
Grid Infrastructure home check failed

Checking CRS user consistency
CRS user consistency check successful
Checking network configuration consistency.
Check for network configuration consistency passed.
Checking ASM disk size consistency
All ASM disks are correctly sized
Checking if default discovery string is being used by ASM
ASM discovery string "/dev/DSK*" is not the default discovery string
Checking if ASM parameter file is in use by an ASM instance on the local node
ASM instance is using parameter file "+DATA/drnode-scan/asmparameterfile/registry.253.936445691" on node "drnode1" on which upgrade is requested.

Checking OLR integrity...
Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed

WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) drnode1,drnode2
TCP connectivity check passed for subnet "192.168.1.0"


Check: Node connectivity using interfaces on subnet "192.168.0.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) drnode1,drnode2
TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Task ASM Integrity check started...


Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all specified nodes
Disk Group Check passed. At least one Disk Group configured

Task ASM Integrity check passed...

Checking OCR integrity...
Disks "+DATA" are managed by ASM.

OCR integrity check passed

Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check failed
Check failed on nodes:
        drnode2,drnode1
Available memory check passed
Swap space check passed
Free disk space check passed for "drnode2:/usr,drnode2:/var,drnode2:/etc,drnode2:/sbin"
Free disk space check passed for "drnode1:/usr,drnode1:/var,drnode1:/etc,drnode1:/sbin"
Free disk space check passed for "drnode2:/u01/app/11.2.0.3/grid"
Free disk space check passed for "drnode1:/u01/app/11.2.0.3/grid"
Free disk space check passed for "drnode2:/tmp"
Free disk space check passed for "drnode1:/tmp"
Check for multiple users with UID value 501 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
There are no oracle patches required for home "/u01/app/11.2.0.3/grid".
There are no oracle patches required for home "/u01/app/11.2.0.3/grid".
Source home "/u01/app/11.2.0.3/grid" is suitable for upgrading to version "12.1.0.2.0".
System architecture check passed
Kernel version check failed
Check failed on nodes:
        drnode2,drnode1
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
drnode2,drnode1
Clock synchronization check using Network Time Protocol(NTP) failed

Core file name pattern consistency check passed.

User "oracle" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: drnode1,drnode2

Check for integrity of file "/etc/resolv.conf" failed


UDev attributes check for OCR locations started...
UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
UDev attributes check passed for Voting Disk locations

Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed.

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"

Starting check for Reverse path filter setting ...

Check for Reverse path filter setting passed

Starting check for Network interface bonding status of private interconnect network interfaces ...

Check for Network interface bonding status of private interconnect network interfaces passed

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
[oracle@drnode1 grid]$

 

Some of the checks failed such as “Clock synchronization check using Network Time Protocol(NTP)” where NTP daemon or service was not running, “DNS response check failed”. I safely ignore these checks as both NTP and DNS are not configured on these nodes.

 

Now, update the “grid_install.rsp” file available under the unzipped location of the 12c GI software. The response file needs to be updated appropriately based on the environment. Here is the updated grid_install.rsp file used for this upgrade.

 

Before proceeding with the upgrade, unset the CRS HOME if set any on the environment and run the “runInstaller” from the unzipped location of the 12c GI software. Also, pass the parameters “-silent” to run the upgrade in SILENT mode and “-responsefile” specifying the location of the above updated response file.

 

[oracle@drnode1 u02]$ cd /u03/grid
[oracle@drnode1 grid]$
[oracle@drnode1 grid]$ unset ORA_CRS_HOME
[oracle@drnode1 grid]$ echo $ORA_CRS_HOME

[oracle@drnode1 grid]$ nohup ./runInstaller -silent -ignorePrereq -responsefile /u02/grid_install.rsp &
[1] 16082
[oracle@drnode1 grid]$ nohup: ignoring input and appending output to 'nohup.out'

 

Let’s view the output file to see what’s happening.

 

[oracle@drnode1 grid]$ cat nohup.out
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 32905 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 10237 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-02-21_10-10-33AM. Please wait ...[WARNING] [INS-41813] OSDBA for ASM, OSOPER for ASM, and OSASM are the same OS group.
   CAUSE: The group you selected for granting the OSDBA for ASM group for database access, and the OSOPER for ASM group for startup and shutdown of Oracle ASM, is the same group as the OSASM group, whose members have SYSASM privileges on Oracle ASM.
   ACTION: Choose different groups as the OSASM, OSDBA for ASM, and OSOPER for ASM groups.
[WARNING] [INS-41874] Oracle ASM Administrator (OSASM) Group specified is same as the inventory group.
   CAUSE: Operating system group oinstall specified for OSASM Group is same as the inventory group.
   ACTION: It is not recommended to have OSASM group same as inventory group. Select any of the group other than the inventory group to avoid incorrect configuration.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/installActions2017-02-21_10-10-33AM.log
The installation of Oracle Grid Infrastructure 12c was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2017-02-21_10-10-33AM.log' for more details.

As a root user, execute the following script(s):
        1. /u01/app/12.1.0.2/grid/rootupgrade.sh

Execute /u01/app/12.1.0.2/grid/rootupgrade.sh on the following nodes:
[drnode1, drnode2]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.

Successfully Setup Software.
As install user, execute the following script to complete the configuration.
        1. /u01/app/12.1.0.2/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=<response_file>

        Note:
        1. This script must be run on the same host from where installer was run.
        2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation).

 

We now need to run the “rootupgrade.sh” script from the new 12c GI home as ROOT user on all the nodes. Post that, the runInstaller demands to run “configToolAllCommands” on the first node drnode1 where runInstaller was run.

 

Below is the output of the execution of the rootupgrade.sh script on first node drnode1.

 


[root@drnode1]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Check /u01/app/12.1.0.2/grid/install/root_drnode1.mydomain_2017-02-21_11-08-09.log for the output of root script
[root@drnode1]#

=====================================================================================================================
[oracle@drnode1 grid]$ cat /u01/app/12.1.0.2/grid/install/root_drnode1.mydomain_2017-02-21_11-08-09.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/02/21 11:08:21 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 11:09:12 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 11:09:19 CLSRSC-464: Starting retrieval of the cluster configuration data

2017/02/21 11:09:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2017/02/21 11:09:38 CLSRSC-363: User ignored prerequisites during installation

2017/02/21 11:09:58 CLSRSC-515: Starting OCR manual backup.

2017/02/21 11:10:03 CLSRSC-516: OCR manual backup successful.

2017/02/21 11:10:10 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2017/02/21 11:10:10 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.3/grid -oldCRSVersion 11.2.0.3.0 -nodeNumber 1 -firstNode true -startRolling true'


ASM configuration upgraded in local node successfully.

2017/02/21 11:10:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2017/02/21 11:10:27 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2017/02/21 11:12:46 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2017/02/21 11:20:36 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/02/21 11:27:18 CLSRSC-472: Attempting to export the OCR

2017/02/21 11:27:18 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'

2017/02/21 11:27:32 CLSRSC-473: Successfully exported the OCR

2017/02/21 11:27:39 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2017/02/21 11:27:39 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.

2017/02/21 11:27:39 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2017/02/21 11:27:39 CLSRSC-543:
 3. The downgrade command must be run on the node drnode1 with the '-lastnode' option to restore global configuration data.

2017/02/21 11:28:05 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/02/21 11:28:37 CLSRSC-474: Initiating upgrade of resource types

2017/02/21 11:29:48 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.3.0 -d 12.1.0.2.0 -p first'

2017/02/21 11:29:48 CLSRSC-475: Upgrade of resource types successfully initiated.

2017/02/21 11:30:23 CLSRSC-325:	Configure Oracle Grid Infrastructure for a Cluster ... succeeded

=====================================================================================================================

 

Below is the output of the execution of the rootupgrade.sh script on the second (last) node drnode2.

 

[root@drnode2 u01]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Check /u01/app/12.1.0.2/grid/install/root_drnode2.mydomain_2017-02-21_12-22-07.log for the output of root script
[root@drnode2 u01]#

==============================================================================================================

[root@drnode2 u01]# cat /u01/app/12.1.0.2/grid/install/root_drnode2.mydomain_2017-02-21_12-22-07.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/02/21 12:22:13 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 12:22:13 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 12:22:22 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 12:22:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2017/02/21 12:22:38 CLSRSC-464: Starting retrieval of the cluster configuration data

2017/02/21 12:22:47 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2017/02/21 12:22:47 CLSRSC-363: User ignored prerequisites during installation


ASM configuration upgraded in local node successfully.

2017/02/21 12:23:11 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2017/02/21 12:25:21 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2017/02/21 12:26:47 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/02/21 12:34:02 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
2017/02/21 12:34:31 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded

2017/02/21 12:34:31 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl set crs activeversion'

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
Started to upgrade the CSS.
The CSS was successfully upgraded.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 12.1.0.2.0
2017/02/21 12:37:03 CLSRSC-479: Successfully set Oracle Clusterware active version

2017/02/21 12:37:14 CLSRSC-476: Finishing upgrade of resource types

2017/02/21 12:37:27 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.3.0 -d 12.1.0.2.0 -p last'

2017/02/21 12:37:27 CLSRSC-477: Successfully completed upgrade of resource types

2017/02/21 12:38:31 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

=======================================================================================================

 

 

Now let’s run the “configToolAllCommands” on drnode1. This script perform PostInstallation tasks such as creation of the “Grid Infrastructure Management Repository Database” with the Container Database Name as “MGMTDB” and it’s Pluggabe Database Name as “”.

 

[oracle@drnode1 bin]$ /u01/app/12.1.0.2/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/u02/grid_install.rsp
Setting the invPtrLoc to /u01/app/12.1.0.2/grid/oraInst.loc

perform - mode is starting for action: configure

Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: UpdateNodelist data:
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.oui_loc:/u01/app/12.1.0.2/grid/oui
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.jre_loc:/u01/app/12.1.0.2/grid/jdk/jre
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.doNotUpdateNodeList:true
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.rootOwnedHome:true
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: OracleHomeToUpdate:/u01/app/11.2.0.3/grid;isCRS:false;isCFS:false;isLocal:false
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: From map: Hosts:[drnode1, drnode2] => Nodelist:[drnode1, drnode2]
Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: Before calling api: Hosts:[drnode1, drnode2] => Nodelist:[drnode1, drnode2], update localnode? true
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: UpdateNodelist data:
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.oui_loc:/u01/app/12.1.0.2/grid/oui
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.jre_loc:/u01/app/12.1.0.2/grid/jdk/jre
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.doNotUpdateNodeList:true
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: oracle.installer.rootOwnedHome:
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: OracleHomeToUpdate:/u01/app/12.1.0.2/grid;isCRS:true;isCFS:false;isLocal:false
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: From map: Hosts:[drnode1, drnode2] => Nodelist:[drnode1, drnode2]
Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob call
INFO: Before calling api: Hosts:[drnode1, drnode2] => Nodelist:[drnode1, drnode2], update localnode? true
Feb 21, 2017 1:44:03 PM oracle.install.config.crs.MgmtDBCDBInternalPlugIn invoke
INFO: MgmtDBCDBInternalPlugin: ... adding </oui_internal>
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
INFO: Executing MGMTDBCDB
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
INFO: Command /u01/app/12.1.0.2/grid/bin/dbca  -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName DATA -datafileJarLocation /u01/app/12.1.0.2/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck   -oui_internal
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn.handleProcess() entered.
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: checking secretArguments.
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: No arguments to pass to stdin
Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: starting read loop.
Feb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Registering database with Oracle Grid Infrastructure
Feb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Registering database with Oracle Grid Infrastructure
Feb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: End of argument passing to stdin
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 5% complete
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 5% complete
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Copying database files
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Copying database files
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 7% complete
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 7% complete
Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 9% complete
Feb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 9% complete
Feb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 16% complete
Feb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 16% complete
Feb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 23% complete
Feb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 23% complete
Feb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 30% complete
Feb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 30% complete
Feb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 37% complete
Feb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 37% complete
Feb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 41% complete
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 41% complete
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Creating and starting Oracle instance
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Creating and starting Oracle instance
Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 43% complete
Feb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 43% complete
Feb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 48% complete
Feb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 48% complete
Feb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 49% complete
Feb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 49% complete
Feb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 50% complete
Feb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 50% complete
Feb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 55% complete
Feb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 55% complete
Feb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 60% complete
Feb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 60% complete
Feb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 61% complete
Feb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 61% complete
Feb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 64% complete
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 64% complete
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Completing Database Creation
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Completing Database Creation
Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 68% complete
Feb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 68% complete
Feb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 79% complete
Feb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 79% complete
Feb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 89% complete
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 89% complete
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 100% complete
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 100% complete
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb.log" for further details.
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb.log" for further details.
Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:00:18 PM oracle.install.config.crs.MgmtDBPDBInternalPlugIn invoke
INFO: MgmtDBPDBInternalPlugin: ... adding </oui_internal>
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
INFO: Executing MGMTDBPDB
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
<strong>INFO: Command /u01/app/12.1.0.2/grid/bin/dbca  -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName drnode_scan -createPDBFrom RMANBACKUP -PDBBackUpfile /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck -oui_internal</strong>
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn.handleProcess() entered.
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: checking secretArguments.
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: No arguments to pass to stdin
Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: starting read loop.
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Creating Pluggable Database
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Creating Pluggable Database
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: End of argument passing to stdin
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 4% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 4% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 12% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 12% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 21% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 21% complete
Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 38% complete
Feb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 38% complete
Feb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 55% complete
Feb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 55% complete
Feb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 85% complete
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 85% complete
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Completing Pluggable Database Creation
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Completing Pluggable Database Creation
Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 100% complete
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: 100% complete
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/drnode_scan/_mgmtdb.log" for further details.
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
WARNING: Skipping line: Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/drnode_scan/_mgmtdb.log" for further details.
Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0

perform - mode finished for action: configure

You can see the log file: /u01/app/12.1.0.2/grid/cfgtoollogs/oui/configActions2017-02-21_01-39-03-PM.log

 

 

Post execution of all the demanded scripts, let’s check the latest CRS version on both the nodes.

 

On drnode1:

 

[oracle@drnode1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
[oracle@drnode1 ~]$

 

[oracle@drnode1 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oracle@drnode1 ~]$

 

[oracle@drnode1 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs softwareversion
Oracle Clusterware version on node [drnode1] is [12.1.0.2.0]

 

On drnode2:

 

[oracle@drnode2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
[oracle@drnode2 ~]$

 

[oracle@drnode2 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs softwareversion
Oracle Clusterware version on node [drnode2] is [12.1.0.2.0]

 

[oracle@drnode2 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oracle@drnode2 ~]$

 

Verify that all ASM instances and all the diskgroups are MOUNTED on both the nodes.

 

Finally, you can detach the Old GI home (11.2.0.3) from the ORAINVENTORY file.

 

[oracle@drnode1 ~]$ cd /u01/app/11.2.0.3/grid/oui/bin/
[oracle@drnode1 bin]$ ./runInstaller -detachHome -silent ORACLE_HOME=/u01/app/11.2.0.3/grid
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 10236 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'DetachHome' was successful.

 

 

COPYRIGHT

© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.

 

 

DISCLAIMER

The views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine