ora-solutions.net - Martin Decker » Bugs http://www.ora-solutions.net/web Indepented Oracle consultant Wed, 25 Jan 2012 14:04:18 +0000 en hourly 1 http://wordpress.org/?v= Bug: ORA-4031 for subheap “KTC latch subh” in 11.2.0.3 http://www.ora-solutions.net/web/2012/01/25/bug-ora-4031-for-subheap-ktc-latch-subh-in-11-2-0-3/ http://www.ora-solutions.net/web/2012/01/25/bug-ora-4031-for-subheap-ktc-latch-subh-in-11-2-0-3/#comments Wed, 25 Jan 2012 14:04:18 +0000 Martin http://www.ora-solutions.net/web/?p=1135 I just saw MOS Note

ORA-4031 After Upgrade From 11.2.0.1 To 11.2.0.3 With Leak in ‘KTC latch subh’ Subheaps [ID 1398984.1]

giving very little information about this bug. At the time of this writing, there is no patch available but I expect it to be in a few days. I have checked some 11.2.0.3 instances and found only one with > 60 MB for this subheap.

]]> http://www.ora-solutions.net/web/2012/01/25/bug-ora-4031-for-subheap-ktc-latch-subh-in-11-2-0-3/feed/ 0 Oracle SCN Problem http://www.ora-solutions.net/web/2012/01/19/oracle-scn-problem/ http://www.ora-solutions.net/web/2012/01/19/oracle-scn-problem/#comments Thu, 19 Jan 2012 21:56:52 +0000 Martin http://www.ora-solutions.net/web/?p=1125 In case you have not heard all the buzz about the Oracle SCN flaw, which was published by Infoworld after the release of January CPU, I summarize the essentials.

  • There is a risk that the SCN reaches it´s maximum value and this could lead to an outage of the database.
  • There is a bug in “ALTER DATABASE BEGIN BACKUP”, which increases the SCN dramatically. (Bug 12371955 – fixed in 11.2.0.2.4 and others)
  • When you query a remote database via database link from a database with elevated SCN, the remote SCN gets increased to the higher value as well. This has the dramatic effect, that a database will infect the other database.
  • Imagine an unpatched environment with user-managed online backups (e.g. for SAN split mirror technology) and dozens of interconnected databases with database links.
  • Now imagine an evil employee with this little innocent database account with only “create session” privilege  … you get the picture.
  • The January 2012 CPU/PSU contains a patch that should restrict the ways to increase the SCN. However, I am not sure that the issue with db links is solved. I could bump up the SCN to 12562779343042 by a remote select even after patching with CPU January 2012.

Additionally, there are already quite a few articles showing how a potential attacker could use this flaw, e.g. this one.

References:

  • Information on the System Change Number (SCN) and how it is used in the Oracle Database [ID 1376995.1]
  • Installing, Executing and Interpreting output from the “scnhealthcheck.sql” script [ID 1393363.1]
  • Bug 12371955 – Backup task can cause increased SCN growth rate leading to ORA-600 [2252] errors [ID 12371955.8]
]]> http://www.ora-solutions.net/web/2012/01/19/oracle-scn-problem/feed/ 1 Is 11.2.0.2 ready for production? Judge yourself … http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/ http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/#comments Mon, 18 Apr 2011 09:34:12 +0000 Martin http://www.ora-solutions.net/web/?p=1036 Murphy said: if problems arise, they all come at the same time. After several weeks of stability, this week brought hell of a lot of Oracle problems with 11.2.0.2 and RAC. To list the worst:

  • Downtime because of ASM process limit reached. Normally, the formula to calculate depends on the number of concurrent datafile extensions. In our case 100 ASM processes were not enough. Increased to 200 and found  Note 1287496.1 which describes the issue. Merge Patch for bug is available for 11.2.0.2 GI Bundle 1 but not yet for GI Bundle 2. Ups.
  • CPU Starvation because of Adaptive Cursor Sharing: One server process was taking up 100% of cpu time for the last several hours. It turned out to be an OEM monitoring query having several thousands of child cursors.
    PID 	USER 	PR NI VIRT 	RES SHR S %CPU %MEM TIME+ COMMAND
8667 	ora11 25 0 	8423m 33m 28m R 99.7 	0.1 358:27.78 ora_pz98_MDDB

    I then checked different databases and found that almost all of our 11.2.0.2 databases have several hundreds or even thousands of child cursors.

    SELECT sql_id, is_shareable, min(child_number), max(child_number), count(*)
FROM gv$sql
GROUP BY sql_id, is_shareable HAVING count(*) >100 ORDER BY 6 DESC
 
SQL_ID        IS_SHAREABLE MIN(CHILD_NUMBER) MAX(CHILD_NUMBER)  COUNT(*)
------------- ------------ ----------------- ------------------ ---------
c7kt3njhnmtkm Y            0                 5097               1397
c7kt3njhnmtkm N            3                 3544               1836
1vnhgmpc17vv0 Y            0                 3022               2697
1vnhgmpc17vv0 N            6                 2185               444
93qh89pxuxzuw Y            0                 1949               1522
93qh89pxuxzuw N            2                 1625               428
5fk0v8km2f811 Y            0                 1281               1763
4f3ufvfcgfqsg Y            0                 792                794
cjbwk0ww7j5rv Y            0                 627                1251
dyqdzgxcv4d3t Y            0                 626                1252
5fk0v8km2f811 N            3                 543                260
f0jxh8d6b5af2 Y            0                 494                564
f0jxh8d6b5af2 N            0                 290                130
dbvkky621gqtr Y            0                 266                267
32rqmpqpfv0gm Y            0                 255                257
g9uwxs7pr8tjm Y            0                 254                257
40k6jjt90n4fa Y            3                 199                129

    I suspect this to be Bug 10182051 Extended cursor sharing generates many shareable child cursors and there is a workaround:

    alter system set “_optimizer_extended_cursor_sharing_rel”=none;

  • Limited database availability because of failing queries on gv$ tables:
    SQL> select count(*) from gv$session
2 ;
select count(*) from  gv$session
*
ERROR at line 1:
ORA-12850: Could not allocate slaves on  all specified instances: 2 needed, 1
allocated
ORA-12801: error signaled  in parallel query server P001, instance 3599

    Currently no other known workaround than bouncing all the RAC instances.

  • DataGuard ASYNC Redo Transport not reliable: We have a RAC primary / single instance physical standby setup and use async redo transport. During times of heavy ETL on the primary, the standby databases stops at recovery of one archivelog with “(in transit)”. Primary is showing this error in alert log: 
    ARC7: Creating remote archive destination LOG_ARCHIVE_DEST_2: 'HAMDB' (thread 2 sequence 4044) (MUCDB2)
ARC7: Archive log rejected (thread 2 sequence 4044) at host 'HAMDB'
FAL[server, ARC7]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance MUCDB2 - Archival Error. Archiver continuing.

    On standby side, it says:
    Media Recovery Waiting for thread 2 sequence 4044 (in transit)

    The standby database never recovers from this problem, except when standby database is bounced. The problem appears with and without broker configuration. Currently there is no known workaround.

Maybe some of this issues will be addressed in upcoming PSU April, which will be released this week.

]]> http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/feed/ 2 Recyclebin Bug – ORA-600 [ktcdso-1] on Oracle 11.2.0.2.1 http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/ http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/#comments Fri, 04 Mar 2011 16:08:29 +0000 Martin http://www.ora-solutions.net/web/?p=1031 Just a short note:

If you are using 11.2.0.2 and 11.2.0.2.1, chances are high that database stability is endangered because of massive ORA-600 [ktcdso-1], unless you have recyclebin deactived with init.ora recyclebin=off.

Oracle is currently working on this issue with Bug 10427260: ORA-00600 [KTCDSO-1], [], [], [] WHEN PURGE RECYCLE BIN.

Unfortunately, the instance has to be bounced in order to deactivate recyclebin.

]]> http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/feed/ 2 suboptimal prorated density calculation with DESC index – cardinality feedback feature http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/ http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/#comments Mon, 25 Oct 2010 19:53:38 +0000 Martin http://www.ora-solutions.net/web/?p=899 At a customer site, I came across a problem of the Oracle cost based optimizer with selectivity calculation with a composite index with one column sorted descending. The following testcase describes the issue. Currently Oracle development is working on the issue. During testing on 11.2.0.1, I realized that the new cardinality feedback issue takes care of the problem and subsequent executions are using the correct plan.

-- drop table if exists
DROP TABLE ADMDECKER.DUMMY purge;
 
-- create table
CREATE TABLE ADMDECKER.DUMMY (
      a number NOT NULL,
      B number NOT NULL,
      C number NOT NULL,
      D number NOT NULL,
      E number ) NOLOGGING;
 
-- fill table
INSERT /*+ APPEND */  INTO ADMDECKER.DUMMY
SELECT  rownum AS a,
        MOD(rownum,100) AS B,
        MOD(rownum,10000) AS C,
        1 AS D,
        234 AS E
        FROM (
               SELECT rownum r
               FROM
                       (SELECT rownum r FROM dual connect BY rownum <= 100) a,
                       (SELECT rownum r FROM dual connect BY rownum <= 100) b,
                       (SELECT rownum r FROM dual connect BY rownum <= 100) c
               WHERE rownum <= 1000000
       );
 
commit;
 
-- create indexes
CREATE INDEX ADMDECKER.DUMMY_IDX1 ON ADMDECKER.DUMMY(C,B) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX2 ON ADMDECKER.DUMMY(B DESC,a) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX3 ON ADMDECKER.DUMMY(C,B,D) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX4 ON ADMDECKER.DUMMY(B,a) nologging;
 
-- gather statistics (including tab stats for virtual hidden column for "B desc" column.
begin DBMS_STATS.GATHER_TABLE_STATS(
      OWNNAME=>'ADMDECKER',
      TABNAME=>'DUMMY',
      METHOD_OPT=>'FOR ALL COLUMNS SIZE 1',
      cascade=>true,
      ESTIMATE_PERCENT=>100);
END;
/
 
SELECT * FROM DBA_TAB_STATISTICS WHERE TABLE_NAME = 'DUMMY';
SELECT * FROM DBA_TAB_COL_STATISTICS WHERE TABLE_NAME = 'DUMMY';
 
-- note min/max/density values for system generated virtual hidden column
SELECT * FROM DBA_TAB_COLS WHERE TABLE_NAME = 'DUMMY';
SELECT * FROM dba_ind_statistics  WHERE TABLE_NAME = 'DUMMY';

I enable 10053 tracing and execute the query. Afterwards I display the execution plan.

SQL&gt; ALTER SESSION SET EVENTS='10053 trace name context forever, level 1';
 
Session altered.
 
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1;
 
  COUNT(E)
----------
         0
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
---------------------
SQL_ID  f5mkaprk7fzt8, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1
 
Plan hash value: 1903509519
 
------------------------------------------------------------------------------------------------
| Id |Operation                    |Name      |E-Rows| Cost (%CPU)|A-Rows|   A-Time   | Buffers |
------------------------------------------------------------------------------------------------
|   0|SELECT STATEMENT             |          |      |     4 (100)|     1|00:00:00.07 |   23437 |
|   1| SORT AGGREGATE              |          |     1|            |     1|00:00:00.07 |   23437 |
|*  2|  TABLE ACCESS BY INDEX ROWID|DUMMY     |     7|     4   (0)|     0|00:00:00.07 |   23437 |
|*  3|   INDEX RANGE SCAN          |DUMMY_IDX2|     1|     3   (0)| 70000|00:00:00.03 |     197 |
------------------------------------------------------------------------------------------------
 
Predicate Information (IDENTIFIED BY operation id):
---------------------------------------------------
 
   2 - filter(("C"=9592 AND "D"=1))
   3 - access("DUMMY"."SYS_NC00006$"<=HEXTORAW('3EA1FF') )
       filter(SYS_OP_UNDESCEND("DUMMY"."SYS_NC00006$")>=93)

The complete 10053 can be viewed here: 10053 Trace of bad plan.

It can be seen that the execution plan contains the following statement and uses a very, very low density.

  Using prorated density: 0.000002 of col #6 as selectvity of out-of-range/non-existent value pred

So, instead of using Index DUMMY_IDX3, which contains all three predicate columns, especially more selective equality predicate “C = 9592″, it uses quite unselective index DUMMY_IDX2, only because density is calculated wrongly.

If the statement is executed a second time, the right plan is used. Cardinality feedback feature kicks in here. Please note the higher costs and higher cardinality for index range scan at row source operation id 3.

connect ADmdecker
Enter password:
Connected.
ALTER SESSION SET EVENTS='10053 trace name context forever, level 1';
 
Session altered.
 
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1;
 
  COUNT(E)
----------
         0
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
----------------------
SQL_ID  f5mkaprk7fzt8, child number 1
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM MDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1
 
Plan hash value: 1171978535
 
-------------------------------------------------------------------------------------------------
| Id |Operation                    |Name       |E-Rows| Cost (%CPU)|A-Rows |   A-Time   |Buffers |
-------------------------------------------------------------------------------------------------
|   0|SELECT STATEMENT             |           |      |    11 (100)|     1 |00:00:00.01 |      3 |
|   1| SORT AGGREGATE              |           |    1 |            |     1 |00:00:00.01 |      3 |
|   2|  TABLE ACCESS BY INDEX ROWID|DUMMY      |    7 |    11   (0)|     0 |00:00:00.01 |      3 |
|*  3|   INDEX RANGE SCAN          |DUMMY_IDX3 |    7 |     3   (0)|     0 |00:00:00.01 |      3 |
-------------------------------------------------------------------------------------------------
 
Predicate Information (IDENTIFIED BY operation id):
---------------------------------------------------
 
   3 - access("C"=9592 AND "B">=93 AND "D"=1)
       filter("D"=1)
 
Note
-----
 - cardinality feedback used FOR this statement

The complete 10053 of the new plan (using cardinality feedback feature) can be viewed here: 10053 Trace of good plan using cardinality feedback. More information about this little documented new feature can be found at Dion Cho´s blog and from this Oracle paper.

Update: 06.10.2011

Oracle Support has provided Patch 11072246, which fixes a bug in DBMS_STATS on how min/max values are set for hidden desc columns. After installing the patch, the bug is gone and the execution plan is optimal.

]]> http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/feed/ 0 opatch auto problem with Grid Infrastructure PSUs http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/ http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/#comments Wed, 29 Sep 2010 11:33:06 +0000 Martin http://www.ora-solutions.net/web/?p=878 With Grid Infrastructure, Oracle customers are encouraged to seperate privileges between Grid Infrastructure and RDBMS owner. Therefore we are using grid11 as GI owner and rdbms11 as RDBMS owner. However, due to a bug in opatch (which is currently being worked on by Oracle) “opatch auto” functionality does not work as documented in the PSU readme. Workaround is to apply the patch manually. The procedure is similar to the CRS bundles we know from previous releases.

In the following, I have documented the procedure to apply the PSU patch manually.

mkdir GI_PSU
cd GI_PSU
unzip –d . ../GI_PSU_112012_p9655006_112010_Linux-x86-64.zip
 
[grid11@node1 GI_PSU]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /opt/oracle/grid11/11.2.0.1/grid -phBaseDir ./9655006
...
Prereq "checkConflictAgainstOHWithDetail" passed.
 
as root:
 
[root@node1 grid]# cd /stage/GI_PSU
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch  version
Invoking OPatch 11.2.0.1.3
 
OPatch Version: 11.2.0.1.3
 
OPatch succeeded.
 
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:27:24: Parsing the host name
2010-09-22 10:27:24: Checking for super user privileges
2010-09-22 10:27:24: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.node1.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2676: Start of 'ora.node1.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2673: Attempting to stop 'ora.eons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2677: Stop of 'ora.eons' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'node1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.diskmon' on 'node1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-28-11AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-32-47AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
[root@node2 ~]# cd /stage/GI_PSU
[root@node2 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:39:36: Parsing the host name
2010-09-22 10:39:36: Checking for super user privileges
2010-09-22 10:39:36: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.crsd' on 'node2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.node2.vip' on 'node1'
CRS-2677: Stop of 'ora.registry.acfs' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node2'
CRS-2673: Attempting to stop 'ora.eons' on 'node2'
CRS-2677: Stop of 'ora.ons' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node2'
CRS-2677: Stop of 'ora.net1.network' on 'node2' succeeded
CRS-2677: Stop of 'ora.eons' on 'node2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completed
CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node2'
CRS-2673: Attempting to stop 'ora.evmd' on 'node2'
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node2'
CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'
CRS-2673: Attempting to stop 'ora.diskmon' on 'node2'
CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node2'
CRS-2677: Stop of 'ora.gipcd' on 'node2' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'node2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-40-25AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-42-48AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
RDBMS Part:
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-28AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkapplicable"
 
Prereq "checkApplicable" for patch 9655006 passed.
 
OPatch succeeded.
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-45AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkconflictagainstoh"
 
Prereq "checkConflictAgainstOH" passed.
 
OPatch succeeded.
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9654983 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
./9655006/custom/server/9655006/custom/scripts/prepatch.sh completed successfully.
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch napply ./9655006/custom/server/9655006 -local -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ cd 9654983/
[rdbms11@node1 9654983]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch apply -local -oh $ORACLE_HOME
 
cd ..
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgmain
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgeut
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/diskmon.bin
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/lsnodes
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/osdbagrp
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/rawutl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/ractrans
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/getcrshome
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/gnsd
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/crsdiag.pl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libhasgen11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libclsra11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libdbcfg11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocr11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrb11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrutl11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libuini11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/librdjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgns11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgnsjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libagfw11.so
 
The same steps have to be repeated for Node2.
 
[grid11@node1 ~]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-20-41AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-20-41AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:35:11 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:32:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[grid11@node2 9655006]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-21-17AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-17AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:43:57 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:42:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node1 GI_PSU]$  /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-36AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-36AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:15:52 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:14:30 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node2 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-57AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-57AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:19:38 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:19:03 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
]]> http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/feed/ 1 Highly Dangerous Oracle Database Security Vulnerability http://www.ora-solutions.net/web/2010/02/26/highly-dangerous-oracle-database-security-vulnerability/ http://www.ora-solutions.net/web/2010/02/26/highly-dangerous-oracle-database-security-vulnerability/#comments Fri, 26 Feb 2010 20:39:25 +0000 Martin http://www.ora-solutions.net/web/?p=802 I would like to draw your attention to a particularly dangerous security vulnerability, which was recently published by David Litchfield.

How dangerous is the vulnerability?

Any database user, who has “create session” privilege, which means, who can log into the database, can use the security hole to execute any OS command in the ownership of the oracle database owner. This means, that both denial of service as well as access to all data is exposed.

Which versions are affected?

Affected are database versions 10.2.0.4 (incl. 10.2.0.4.3 containing latest security patches as of January 2010) as well as 11g (incl. 11.2.0.1).

What can I do to close this security vulernability?

You can revoke privileges from PUBLIC:

revoke execute on dbms_java from PUBLIC;
revoke execute on dbms_java_test from PUBLIC;
revoke execute on “oracle/aurora/util/Wrapper” from PUBLIC;
grant execute on sys.dbms_jvm_exp_perms to IMP_FULL_DATABASE;
grant execute on sys.dbms_jvm_exp_perms to EXP_FULL_DATABASE;
revoke execute on sys.dbms_jvm_exp_perms from PUBLIC;

If you are using a third party vendor application, you should contact your vendor to check compatibility with revoked privileges or test before implementing in production.

]]> http://www.ora-solutions.net/web/2010/02/26/highly-dangerous-oracle-database-security-vulnerability/feed/ 0 Oracle Clusterware / ASM 11.1.0.7: ASM Instance crash http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/ http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/#comments Wed, 28 Oct 2009 15:27:00 +0000 Martin http://www.ora-solutions.net/web/?p=779 During a 11gR1 Clusterware installation for a Single Instance Failover Cluster at a customer site, I have experienced an interesting behaviour, which was caused by an Oracle Bug.

The environment was:

  • 2 Node Oracle Clusterware 11.1.0.7 Cluster on Linux x86_64 using latest Recommended Patches as of October 19th. (pre PSU 11.1.0.7.1)
  • Clusterware installed as unix user crs
  • ASM installed as unix user oracle

The ASM instances could be started with SQL*Plus without any problems, but if the ASM instances were started by means of clusterware using srvctl (either from root, crs or oracle) the  ASM instances would crash at diskgroup mount with:

ORA-07445: exception encountered: core dump  [sskgds_find_rtn_hdr()+1171]
[SIGBUS] [ADDR:0x2AACD701342C] [PC:0x25799DF]
[Non-existent physical address] []

Oracle Support identified this behaviour as Bug 6952915, for which there are patches for Linux x86, x86_64 and Solaris Sparc64.

]]> http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/feed/ 0 High CPU Utilization – waits on cursor: pin S http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/ http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/#comments Tue, 11 Aug 2009 05:42:14 +0000 Martin http://www.ora-solutions.net/web/?p=736 I have recently encountered a problem at a customer site, where the database instances resource utilization was so high that the application did not work anymore. Version was 10.2.0.4 on Linux with Oracle Recommended Generic Patches installed.

Unfortunately, the customer decided to bounce the instance, so that there is no possibility for intensive diagnosis. However, ASH report shows that a dozen sessions either waited on Wait event “cursor: pin S” or were active (ON CPU) without any SQL_ID. An AWR Report showed:

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
cursor: pin S                   140,036,615      24,833      0 ######      Other
CPU time                                             19          72.9
log file sync                         8,767           8      1   30.2     Commit
log file parallel write               9,039           8      1   29.9 System I/O
control file parallel write           1,269           5      4   20.2 System I/O

Oracle Support confirmed that this is Bug 6904068 High CPU usage when there are “cursor: pin S” waits. We have filed a backport request for it, as there is no patch for our platform available yet.

]]> http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/feed/ 0 Default 10gR2 RAC TNS Configuration can cause TCP Timeouts for application http://www.ora-solutions.net/web/2009/05/12/default-10gr2-rac-tns-configuration-can-cause-tcp-timeouts-for-application/ http://www.ora-solutions.net/web/2009/05/12/default-10gr2-rac-tns-configuration-can-cause-tcp-timeouts-for-application/#comments Tue, 12 May 2009 14:57:06 +0000 Martin http://www.ora-solutions.net/web/?p=625 The default RAC installation does normally not set “local_listener” init.ora parameter. If the listener is running on port 1521, then the database does not need the parameter in order to find and register with the local TNS listener process. However, if you have *not* set local_listener, it means that the database registers at the listener with the physical IP address instead of the virtual (vip) address.

You can determine if this happens when you take a look at “lsnrctl serv” output from your rac nodes:

Service "S_APP" has 1 instance(s).
  Instance "MDDB1", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=ora-vm1.intra)(PORT=1521))

Instead of ora-vm1.intra, this should be ora-vm1-vip.intra.

Why should I care?

If you use the default configuration, then you are using the parameter “REMOTE_LISTENER” and therefore Server Side Connect Time Load Balancing. This means, that the listeners of all nodes receive load information from all instances of all nodes and they can redirect connections to the least loaded instance, even if the instance is on another node. But the connect string they then send back to the client contains the physical IP address instead of the virtual.

In case of node crashes or kernel panics, etc. the client has to wait for the TCP timeout until this is detected.

Solution

tnsnames.ora:

LISTENER_MDDB1 =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ora-vm1-vip.intra)(PORT = 1521))
  )

LISTENER_MDDB2 =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ora-vm2-vip.intra)(PORT = 1521))
  )

init.ora:
alter system set local_listener = 'LISTENER_MDDB1' sid='MDDB1';
alter system set local_listener = 'LISTENER_MDDB2' sid='MDDB2';
]]> http://www.ora-solutions.net/web/2009/05/12/default-10gr2-rac-tns-configuration-can-cause-tcp-timeouts-for-application/feed/ 0