Archive for September 2010

opatch auto problem with Grid Infrastructure PSUs

With Grid Infrastructure, Oracle customers are encouraged to seperate privileges between Grid Infrastructure and RDBMS owner. Therefore we are using grid11 as GI owner and rdbms11 as RDBMS owner. However, due to a bug in opatch (which is currently being worked on by Oracle) “opatch auto” functionality does not work as documented in the PSU readme. Workaround is to apply the patch manually. The procedure is similar to the CRS bundles we know from previous releases.

In the following, I have documented the procedure to apply the PSU patch manually.

mkdir GI_PSU
cd GI_PSU
unzip –d . ../GI_PSU_112012_p9655006_112010_Linux-x86-64.zip
 
[grid11@node1 GI_PSU]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /opt/oracle/grid11/11.2.0.1/grid -phBaseDir ./9655006
...
Prereq "checkConflictAgainstOHWithDetail" passed.
 
as root:
 
[root@node1 grid]# cd /stage/GI_PSU
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch  version
Invoking OPatch 11.2.0.1.3
 
OPatch Version: 11.2.0.1.3
 
OPatch succeeded.
 
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:27:24: Parsing the host name
2010-09-22 10:27:24: Checking for super user privileges
2010-09-22 10:27:24: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.node1.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2676: Start of 'ora.node1.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2673: Attempting to stop 'ora.eons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2677: Stop of 'ora.eons' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'node1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.diskmon' on 'node1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-28-11AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-32-47AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
[root@node2 ~]# cd /stage/GI_PSU
[root@node2 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:39:36: Parsing the host name
2010-09-22 10:39:36: Checking for super user privileges
2010-09-22 10:39:36: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.crsd' on 'node2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.node2.vip' on 'node1'
CRS-2677: Stop of 'ora.registry.acfs' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node2'
CRS-2673: Attempting to stop 'ora.eons' on 'node2'
CRS-2677: Stop of 'ora.ons' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node2'
CRS-2677: Stop of 'ora.net1.network' on 'node2' succeeded
CRS-2677: Stop of 'ora.eons' on 'node2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completed
CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node2'
CRS-2673: Attempting to stop 'ora.evmd' on 'node2'
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node2'
CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'
CRS-2673: Attempting to stop 'ora.diskmon' on 'node2'
CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node2'
CRS-2677: Stop of 'ora.gipcd' on 'node2' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'node2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-40-25AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-42-48AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
RDBMS Part:
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-28AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkapplicable"
 
Prereq "checkApplicable" for patch 9655006 passed.
 
OPatch succeeded.
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-45AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkconflictagainstoh"
 
Prereq "checkConflictAgainstOH" passed.
 
OPatch succeeded.
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9654983 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
./9655006/custom/server/9655006/custom/scripts/prepatch.sh completed successfully.
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch napply ./9655006/custom/server/9655006 -local -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ cd 9654983/
[rdbms11@node1 9654983]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch apply -local -oh $ORACLE_HOME
 
cd ..
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgmain
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgeut
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/diskmon.bin
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/lsnodes
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/osdbagrp
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/rawutl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/ractrans
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/getcrshome
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/gnsd
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/crsdiag.pl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libhasgen11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libclsra11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libdbcfg11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocr11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrb11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrutl11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libuini11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/librdjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgns11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgnsjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libagfw11.so
 
The same steps have to be repeated for Node2.
 
[grid11@node1 ~]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-20-41AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-20-41AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:35:11 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:32:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[grid11@node2 9655006]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-21-17AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-17AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:43:57 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:42:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node1 GI_PSU]$  /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-36AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-36AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:15:52 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:14:30 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node2 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-57AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-57AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:19:38 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:19:03 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.


Upgrade/Install of Grid Infrastructure to 11.2.0.2 is failing

I have come across a quite difficult issue when trying to install/upgrade to Grid Infrastructure 11.2.0.2 on Linux. We have performed the following trials:

  • Installation of 11.1.0.6 to have Cluster Files (OCR/Votedisk) outside of ASM, then upgrade to 11.2.0.1. After that, we installed all the latest patches (PSU 11.2.0.1.2). Firstly we considered running RDBMS 11.1.0.7 in a 11.2 Grid Infrastructure cluster. However, there are quite a few issues when running this mixed setup. For most of them, patches or workarounds do exist. (see Note 948456.1) However, during failover testing, we realized that the remote database instance does not survive the failure of the public lan of the local node. This is fixed in 11.2.0.2 which motivated us to upgrade.
  • We tried to upgrade our GI 11.2.0.1.2 installation to 11.2.0.2. Everything went fine until it failed when running rootupgrade.sh on the last (second) node. Here, the second node could not join the cluster. Important to know is that 11.2.0.2 brings the new feature HAIP (High available IP), which allows to have multiple private interconnect interfaces (each with a different private interconnect network ip) to be used for failover/load balancing by Grid Infrastructure.  For this feature, Oracle uses Multicast on network “230.0.1.0”. This is stated in the updated version of the README for Oracle Database 11.2: http://download.oracle.com/docs/cd/E11882_01/readmes.112/e17129/toc.htm#CHDIEHCH
ACFS-9309: ADVM/ACFS installation correctness verified.
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at
/opt/oracle/grid11/11.2.0.2/grid/crs/install/crsconfig_lib.pm line 1016.
/opt/oracle/grid11/11.2.0.2/grid/perl/bin/perl -I/opt/oracle/grid11/11.2.0.2/grid/perl/lib -
I/opt/oracle/grid11/11.2.0.2/grid/crs/install
/opt/oracle/grid11/11.2.0.2/grid/crs/install/rootcrs.pl execution failed
  • The next try was to remove everything and directly install 11.2.0.1 with cluster files stored inside of ASM. This worked fine and we then proceeded in patching with latest PSU 11.2.0.1.2. Then, we gave it another try to upgrade to 11.2.0.2. The same problem occurred again at rootupgrade.sh on second node.
  • Then we decided to remove everything again and try a direct installation of 11.2.0.2 without any upgrade. This also failed at root.sh on last node.

With the hints of several colleagues from OTN forum (http://forums.oracle.com/forums/messageview.jspa?messageID=5393319&stqc=true) we realized that it might be related to multicast setup. We verified that multicast works with the script from MetaLink Note 413783.1. However this script does not take into account that we want multicasting via private interconnect interface instead of public LAN.

I realized that it might be necessary to add a network route to our system to direct multicast to the private interconnect network:

e.g.: /sbin/route add -net 224.0.0.0 netmask 240.0.0.0 dev bond1

After doing that, the multicast test script from MOS succeeded.

starting receiver:

[grid11@Node2 ]$ java MultiCastTestReceive

starting sender:

[grid11@Node1 ]$ java MultiCastTestSend
Sent 10 bytes to /230.0.1.0

message received by receiver:

Received data from: /10.128.128.1:13139 with length: 10

Please note that in this case 10.128.128.1 is private interconnect ip. Although it is currently just a strong suspicion, I am quite positive that this was the problem and we will consider retrying the upgrade if time allows and will report the results.

Unfortunately, Oracle support was of no help at all with this problem.