ora-solutions.net - Martin Decker » MetaLink http://www.ora-solutions.net/web Indepented Oracle consultant Wed, 25 Jan 2012 14:04:18 +0000 en hourly 1 http://wordpress.org/?v= Performance Degradation for Query on DBA_SEGMENTS.BYTES in 11gR2 http://www.ora-solutions.net/web/2011/07/18/performance-degradation-for-query-on-dba_segments-bytes-in-11gr2/ http://www.ora-solutions.net/web/2011/07/18/performance-degradation-for-query-on-dba_segments-bytes-in-11gr2/#comments Mon, 18 Jul 2011 14:21:47 +0000 admin http://www.ora-solutions.net/web/?p=1085 I have been troubleshooting a performance issue in a DWH environment, which is quite interesting. It was a query on DBA_SEGMENTS in 11gR2 with lots of partitions and it was taking almost 10 minutes versus only a few seconds in 10gR2. The problem could be stripped downt to this SQL query:

select bytes from dba_segments

The dba_segment view can be seen in DBA_VIEWS:

SELECT ...
       decode(bitand(segment_flags, 131072), 131072, blocks,
           (decode(bitand(segment_flags,1),1,
            <strong>dbms_space_admin.segment_number_blocks</strong>(tablespace_id, relative_fno,
            header_block, segment_type_id, buffer_pool_id, segment_flags,
            segment_objd, blocks), blocks)))*blocksize,
      ...
FROM sys_dba_segs

The response time was dominated by Waits for “db file sequential reads” taking almost all of the response time.

11gR2:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch    12563     23.17     546.74     157447     580414          1      188421
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    12565     23.17     546.74     157447     580414          1      188421

10gR2:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.06       0.06          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch     5104      1.77       2.54          0      96688          0       76542
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total     5106      1.83       2.61          0      96688          0       76542

The 10046 Trace File contains more data regarding those waits:

WAIT #4: nam='db file sequential read' ela= 134 file#=21 block#=2313482 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 120 file#=15 block#=2128019 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 128 file#=21 block#=2313490 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 372 file#=21 block#=2313498 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 108 file#=21 block#=2313506 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 130 file#=21 block#=2313514 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 132 file#=21 block#=2313522 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 121 file#=21 block#=2313530 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 158 file#=15 block#=2128003 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 146 file#=21 block#=2313538 blocks=1 obj#=-1 
WAIT #4: nam='db file sequential read' ela= 116 file#=21 block#=2313546 blocks=1 obj#=-1 
(Output modified for formatting)

Strange thing that obj# is -1. I made some block dumps of those blocks and found out that all those blocks (file#/block#) have something in common: They were all segment header blocks. Another strange thing was that when the query was executed a second time, all those I/O requests were performed again – so no caching.

At that point, I opened a Support service request. After a couple of weeks, Support suggested running this for each tablespace of the user segments:

exec  dbms_space_admin.TABLESPACE_FIX_SEGMENT_EXTBLKS('<tablespace_name>');

Finally this fixed the problem. Now, we are trying to find out why this was necessary for a database, that was freshly created with dbca 11.2.0.2 from the seeded template and filled with data pump import.

]]> http://www.ora-solutions.net/web/2011/07/18/performance-degradation-for-query-on-dba_segments-bytes-in-11gr2/feed/ 4 Is 11.2.0.2 ready for production? Judge yourself … http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/ http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/#comments Mon, 18 Apr 2011 09:34:12 +0000 Martin http://www.ora-solutions.net/web/?p=1036 Murphy said: if problems arise, they all come at the same time. After several weeks of stability, this week brought hell of a lot of Oracle problems with 11.2.0.2 and RAC. To list the worst:

  • Downtime because of ASM process limit reached. Normally, the formula to calculate depends on the number of concurrent datafile extensions. In our case 100 ASM processes were not enough. Increased to 200 and found  Note 1287496.1 which describes the issue. Merge Patch for bug is available for 11.2.0.2 GI Bundle 1 but not yet for GI Bundle 2. Ups.
  • CPU Starvation because of Adaptive Cursor Sharing: One server process was taking up 100% of cpu time for the last several hours. It turned out to be an OEM monitoring query having several thousands of child cursors.
    PID 	USER 	PR NI VIRT 	RES SHR S %CPU %MEM TIME+ COMMAND
8667 	ora11 25 0 	8423m 33m 28m R 99.7 	0.1 358:27.78 ora_pz98_MDDB

    I then checked different databases and found that almost all of our 11.2.0.2 databases have several hundreds or even thousands of child cursors.

    SELECT sql_id, is_shareable, min(child_number), max(child_number), count(*)
FROM gv$sql
GROUP BY sql_id, is_shareable HAVING count(*) &gt;100 ORDER BY 6 DESC
 
SQL_ID        IS_SHAREABLE MIN(CHILD_NUMBER) MAX(CHILD_NUMBER)  COUNT(*)
------------- ------------ ----------------- ------------------ ---------
c7kt3njhnmtkm Y            0                 5097               1397
c7kt3njhnmtkm N            3                 3544               1836
1vnhgmpc17vv0 Y            0                 3022               2697
1vnhgmpc17vv0 N            6                 2185               444
93qh89pxuxzuw Y            0                 1949               1522
93qh89pxuxzuw N            2                 1625               428
5fk0v8km2f811 Y            0                 1281               1763
4f3ufvfcgfqsg Y            0                 792                794
cjbwk0ww7j5rv Y            0                 627                1251
dyqdzgxcv4d3t Y            0                 626                1252
5fk0v8km2f811 N            3                 543                260
f0jxh8d6b5af2 Y            0                 494                564
f0jxh8d6b5af2 N            0                 290                130
dbvkky621gqtr Y            0                 266                267
32rqmpqpfv0gm Y            0                 255                257
g9uwxs7pr8tjm Y            0                 254                257
40k6jjt90n4fa Y            3                 199                129

    I suspect this to be Bug 10182051 Extended cursor sharing generates many shareable child cursors and there is a workaround:

    alter system set “_optimizer_extended_cursor_sharing_rel”=none;

  • Limited database availability because of failing queries on gv$ tables:
    SQL> select count(*) from gv$session
2 ;
select count(*) from  gv$session
*
ERROR at line 1:
ORA-12850: Could not allocate slaves on  all specified instances: 2 needed, 1
allocated
ORA-12801: error signaled  in parallel query server P001, instance 3599

    Currently no other known workaround than bouncing all the RAC instances.

  • DataGuard ASYNC Redo Transport not reliable: We have a RAC primary / single instance physical standby setup and use async redo transport. During times of heavy ETL on the primary, the standby databases stops at recovery of one archivelog with “(in transit)”. Primary is showing this error in alert log: 
    ARC7: Creating remote archive destination LOG_ARCHIVE_DEST_2: 'HAMDB' (thread 2 sequence 4044) (MUCDB2)
ARC7: Archive log rejected (thread 2 sequence 4044) at host 'HAMDB'
FAL[server, ARC7]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance MUCDB2 - Archival Error. Archiver continuing.

    On standby side, it says:
    Media Recovery Waiting for thread 2 sequence 4044 (in transit)

    The standby database never recovers from this problem, except when standby database is bounced. The problem appears with and without broker configuration. Currently there is no known workaround.

Maybe some of this issues will be addressed in upcoming PSU April, which will be released this week.

]]> http://www.ora-solutions.net/web/2011/04/18/is-11-2-0-2-ready-for-production-judge-yourself/feed/ 2 Recyclebin Bug – ORA-600 [ktcdso-1] on Oracle 11.2.0.2.1 http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/ http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/#comments Fri, 04 Mar 2011 16:08:29 +0000 Martin http://www.ora-solutions.net/web/?p=1031 Just a short note:

If you are using 11.2.0.2 and 11.2.0.2.1, chances are high that database stability is endangered because of massive ORA-600 [ktcdso-1], unless you have recyclebin deactived with init.ora recyclebin=off.

Oracle is currently working on this issue with Bug 10427260: ORA-00600 [KTCDSO-1], [], [], [] WHEN PURGE RECYCLE BIN.

Unfortunately, the instance has to be bounced in order to deactivate recyclebin.

]]> http://www.ora-solutions.net/web/2011/03/04/recyclebin-bug-ora-600-ktcdso-1-on-oracle-11-2-0-2-1/feed/ 2 ADR packages and Oracle Support http://www.ora-solutions.net/web/2010/11/12/adr-packages-and-oracle-support/ http://www.ora-solutions.net/web/2010/11/12/adr-packages-and-oracle-support/#comments Fri, 12 Nov 2010 15:36:54 +0000 Martin http://www.ora-solutions.net/web/?p=975 Today I decided to take advantage of adrci and incident package generation for upload to Oracle support. Guess what the first request from Oracle support was:

Please upload the text version of the ASM alert.log.

.

Very funny. :-(

]]> http://www.ora-solutions.net/web/2010/11/12/adr-packages-and-oracle-support/feed/ 0 suboptimal prorated density calculation with DESC index – cardinality feedback feature http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/ http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/#comments Mon, 25 Oct 2010 19:53:38 +0000 Martin http://www.ora-solutions.net/web/?p=899 At a customer site, I came across a problem of the Oracle cost based optimizer with selectivity calculation with a composite index with one column sorted descending. The following testcase describes the issue. Currently Oracle development is working on the issue. During testing on 11.2.0.1, I realized that the new cardinality feedback issue takes care of the problem and subsequent executions are using the correct plan.

-- drop table if exists
DROP TABLE ADMDECKER.DUMMY purge;
 
-- create table
CREATE TABLE ADMDECKER.DUMMY (
      a number NOT NULL,
      B number NOT NULL,
      C number NOT NULL,
      D number NOT NULL,
      E number ) NOLOGGING;
 
-- fill table
INSERT /*+ APPEND */  INTO ADMDECKER.DUMMY
SELECT  rownum AS a,
        MOD(rownum,100) AS B,
        MOD(rownum,10000) AS C,
        1 AS D,
        234 AS E
        FROM (
               SELECT rownum r
               FROM
                       (SELECT rownum r FROM dual connect BY rownum <= 100) a,
                       (SELECT rownum r FROM dual connect BY rownum <= 100) b,
                       (SELECT rownum r FROM dual connect BY rownum <= 100) c
               WHERE rownum <= 1000000
       );
 
commit;
 
-- create indexes
CREATE INDEX ADMDECKER.DUMMY_IDX1 ON ADMDECKER.DUMMY(C,B) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX2 ON ADMDECKER.DUMMY(B DESC,a) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX3 ON ADMDECKER.DUMMY(C,B,D) nologging;
CREATE INDEX ADMDECKER.DUMMY_IDX4 ON ADMDECKER.DUMMY(B,a) nologging;
 
-- gather statistics (including tab stats for virtual hidden column for "B desc" column.
begin DBMS_STATS.GATHER_TABLE_STATS(
      OWNNAME=>'ADMDECKER',
      TABNAME=>'DUMMY',
      METHOD_OPT=>'FOR ALL COLUMNS SIZE 1',
      cascade=>true,
      ESTIMATE_PERCENT=>100);
END;
/
 
SELECT * FROM DBA_TAB_STATISTICS WHERE TABLE_NAME = 'DUMMY';
SELECT * FROM DBA_TAB_COL_STATISTICS WHERE TABLE_NAME = 'DUMMY';
 
-- note min/max/density values for system generated virtual hidden column
SELECT * FROM DBA_TAB_COLS WHERE TABLE_NAME = 'DUMMY';
SELECT * FROM dba_ind_statistics  WHERE TABLE_NAME = 'DUMMY';

I enable 10053 tracing and execute the query. Afterwards I display the execution plan.

SQL&gt; ALTER SESSION SET EVENTS='10053 trace name context forever, level 1';
 
Session altered.
 
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1;
 
  COUNT(E)
----------
         0
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
---------------------
SQL_ID  f5mkaprk7fzt8, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1
 
Plan hash value: 1903509519
 
------------------------------------------------------------------------------------------------
| Id |Operation                    |Name      |E-Rows| Cost (%CPU)|A-Rows|   A-Time   | Buffers |
------------------------------------------------------------------------------------------------
|   0|SELECT STATEMENT             |          |      |     4 (100)|     1|00:00:00.07 |   23437 |
|   1| SORT AGGREGATE              |          |     1|            |     1|00:00:00.07 |   23437 |
|*  2|  TABLE ACCESS BY INDEX ROWID|DUMMY     |     7|     4   (0)|     0|00:00:00.07 |   23437 |
|*  3|   INDEX RANGE SCAN          |DUMMY_IDX2|     1|     3   (0)| 70000|00:00:00.03 |     197 |
------------------------------------------------------------------------------------------------
 
Predicate Information (IDENTIFIED BY operation id):
---------------------------------------------------
 
   2 - filter(("C"=9592 AND "D"=1))
   3 - access("DUMMY"."SYS_NC00006$"<=HEXTORAW('3EA1FF') )
       filter(SYS_OP_UNDESCEND("DUMMY"."SYS_NC00006$")>=93)

The complete 10053 can be viewed here: 10053 Trace of bad plan.

It can be seen that the execution plan contains the following statement and uses a very, very low density.

  Using prorated density: 0.000002 of col #6 as selectvity of out-of-range/non-existent value pred

So, instead of using Index DUMMY_IDX3, which contains all three predicate columns, especially more selective equality predicate “C = 9592″, it uses quite unselective index DUMMY_IDX2, only because density is calculated wrongly.

If the statement is executed a second time, the right plan is used. Cardinality feedback feature kicks in here. Please note the higher costs and higher cardinality for index range scan at row source operation id 3.

connect ADmdecker
Enter password:
Connected.
ALTER SESSION SET EVENTS='10053 trace name context forever, level 1';
 
Session altered.
 
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM ADMDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1;
 
  COUNT(E)
----------
         0
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
----------------------
SQL_ID  f5mkaprk7fzt8, child number 1
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS RUN2 */ COUNT(E) FROM MDECKER.DUMMY
WHERE C = 9592 AND B >= 93 AND D=1
 
Plan hash value: 1171978535
 
-------------------------------------------------------------------------------------------------
| Id |Operation                    |Name       |E-Rows| Cost (%CPU)|A-Rows |   A-Time   |Buffers |
-------------------------------------------------------------------------------------------------
|   0|SELECT STATEMENT             |           |      |    11 (100)|     1 |00:00:00.01 |      3 |
|   1| SORT AGGREGATE              |           |    1 |            |     1 |00:00:00.01 |      3 |
|   2|  TABLE ACCESS BY INDEX ROWID|DUMMY      |    7 |    11   (0)|     0 |00:00:00.01 |      3 |
|*  3|   INDEX RANGE SCAN          |DUMMY_IDX3 |    7 |     3   (0)|     0 |00:00:00.01 |      3 |
-------------------------------------------------------------------------------------------------
 
Predicate Information (IDENTIFIED BY operation id):
---------------------------------------------------
 
   3 - access("C"=9592 AND "B">=93 AND "D"=1)
       filter("D"=1)
 
Note
-----
 - cardinality feedback used FOR this statement

The complete 10053 of the new plan (using cardinality feedback feature) can be viewed here: 10053 Trace of good plan using cardinality feedback. More information about this little documented new feature can be found at Dion Cho´s blog and from this Oracle paper.

Update: 06.10.2011

Oracle Support has provided Patch 11072246, which fixes a bug in DBMS_STATS on how min/max values are set for hidden desc columns. After installing the patch, the bug is gone and the execution plan is optimal.

]]> http://www.ora-solutions.net/web/2010/10/25/suboptimal-prorated-density-calculation-with-desc-index-cardinality-feedback-feature/feed/ 0 opatch auto problem with Grid Infrastructure PSUs http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/ http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/#comments Wed, 29 Sep 2010 11:33:06 +0000 Martin http://www.ora-solutions.net/web/?p=878 With Grid Infrastructure, Oracle customers are encouraged to seperate privileges between Grid Infrastructure and RDBMS owner. Therefore we are using grid11 as GI owner and rdbms11 as RDBMS owner. However, due to a bug in opatch (which is currently being worked on by Oracle) “opatch auto” functionality does not work as documented in the PSU readme. Workaround is to apply the patch manually. The procedure is similar to the CRS bundles we know from previous releases.

In the following, I have documented the procedure to apply the PSU patch manually.

mkdir GI_PSU
cd GI_PSU
unzip –d . ../GI_PSU_112012_p9655006_112010_Linux-x86-64.zip
 
[grid11@node1 GI_PSU]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /opt/oracle/grid11/11.2.0.1/grid -phBaseDir ./9655006
...
Prereq "checkConflictAgainstOHWithDetail" passed.
 
as root:
 
[root@node1 grid]# cd /stage/GI_PSU
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch  version
Invoking OPatch 11.2.0.1.3
 
OPatch Version: 11.2.0.1.3
 
OPatch succeeded.
 
[root@node1 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:27:24: Parsing the host name
2010-09-22 10:27:24: Checking for super user privileges
2010-09-22 10:27:24: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.node1.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2676: Start of 'ora.node1.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2673: Attempting to stop 'ora.eons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2677: Stop of 'ora.eons' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'node1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.diskmon' on 'node1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-28-11AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-32-47AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
[root@node2 ~]# cd /stage/GI_PSU
[root@node2 GI_PSU]# /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch auto . -oh /opt/oracle/grid11/11.2.0.1/grid
Executing /usr/bin/perl /opt/oracle/grid11/11.2.0.1/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /opt/oracle/grid11/11.2.0.1/grid -paramfile /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
2010-09-22 10:39:36: Parsing the host name
2010-09-22 10:39:36: Checking for super user privileges
2010-09-22 10:39:36: User has super user privileges
Using configuration parameter file: /opt/oracle/grid11/11.2.0.1/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.crsd' on 'node2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'node2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.node2.vip' on 'node1'
CRS-2677: Stop of 'ora.registry.acfs' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node2'
CRS-2673: Attempting to stop 'ora.eons' on 'node2'
CRS-2677: Stop of 'ora.ons' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node2'
CRS-2677: Stop of 'ora.net1.network' on 'node2' succeeded
CRS-2677: Stop of 'ora.eons' on 'node2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completed
CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node2'
CRS-2673: Attempting to stop 'ora.evmd' on 'node2'
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node2'
CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'
CRS-2673: Attempting to stop 'ora.diskmon' on 'node2'
CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node2'
CRS-2677: Stop of 'ora.gipcd' on 'node2' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'node2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /opt/oracle/grid11/11.2.0.1/grid
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-40-25AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/pre -apply 9654983 ':
 
Return Code = 0
 
Applying patch 9654983...
 
ApplySession applying interim patch '9654983' to OH '/opt/oracle/grid11/11.2.0.1/grid'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ] , [ oracle.precomp.common, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9654983' for rollback. This might take a while...
 
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/opt/oracle/grid11/11.2.0.1/grid/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
...
ApplySession adding interim patch '9654983' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9654983 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9654983 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target itnslsnr
Execution of 'sh /stage/GI_PSU/9654983/custom/scripts/post -apply 9654983 ':
 
Return Code = 0
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
UTIL session
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_10-42-48AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9655006
 
Do you want to proceed? [y|n]
y
User Responded with: Y
 
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
 
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
 
You selected -local option, hence OPatch will patch the local system only.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/grid11/11.2.0.1/grid')
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
 
Applying patch 9655006...
 
ApplySession applying interim patch '9655006' to OH '/opt/oracle/grid11/11.2.0.1/grid'
Backing up files affected by the patch '9655006' for rollback. This might take a while...
 
Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/opt/oracle/grid11/11.2.0.1/grid/bin/appagent.bin"
...
ApplySession adding interim patch '9655006' to inventory
 
Verifying the update...
Inventory check OK: Patch ID 9655006 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9655006 are present in Oracle Home.
Running make for target install_srvm
Running make for target install
 
The local system has been patched and can be restarted.
 
UtilSession: N-Apply done.
 
OPatch succeeded.
CRS-4123: Oracle High Availability Services has been started.
 
RDBMS Part:
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-28AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkapplicable"
 
Prereq "checkApplicable" for patch 9655006 passed.
 
OPatch succeeded.
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9655006/custom/server/9655006 -oh $ORACLE_HOME
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-10-45AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Invoking prereq "checkconflictagainstoh"
 
Prereq "checkConflictAgainstOH" passed.
 
OPatch succeeded.
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckComponents -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckApplicable -ph 9654983 -oh $ORACLE_HOME
 
/u01/app/oracle/product/11.2.0.1/db/OPatch/opatch prereq CheckConflictAgainstOH -ph 9654983 -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
./9655006/custom/server/9655006/custom/scripts/prepatch.sh completed successfully.
 
[rdbms11@node1 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch napply ./9655006/custom/server/9655006 -local -oh $ORACLE_HOME
 
[rdbms11@node1 GI_PSU]$ cd 9654983/
[rdbms11@node1 9654983]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch apply -local -oh $ORACLE_HOME
 
cd ..
[rdbms11@node1 GI_PSU]$ ./9655006/custom/server/9655006/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0.1/db
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Reading /u01/app/oracle/product/11.2.0.1/db/install/params.ora..
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Parsing file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Verifying file /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgwrap
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvctl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/srvconfig
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgmain
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/racgeut
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/diskmon.bin
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/lsnodes
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/osdbagrp
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/rawutl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/ractrans
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/srvm/admin/getcrshome
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/gnsd
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/bin/crsdiag.pl
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libhasgen11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libclsra11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libdbcfg11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocr11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrb11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libocrutl11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libuini11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/librdjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgns11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libgnsjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0.1/db/lib/libagfw11.so
 
The same steps have to be repeated for Node2.
 
[grid11@node1 ~]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-20-41AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-20-41AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:35:11 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:32:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[grid11@node2 9655006]$ /opt/oracle/grid11/11.2.0.1/grid/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /opt/oracle/grid11/11.2.0.1/grid
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /opt/oracle/grid11/11.2.0.1/grid/oui
Log file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch2010-09-22_11-21-17AM.log
 
Patch history file: /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /opt/oracle/grid11/11.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-17AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9655006      : applied on Wed Sep 22 10:43:57 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722
 
Patch  9654983      : applied on Wed Sep 22 10:42:30 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node1 GI_PSU]$  /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-36AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-36AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:15:52 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:14:30 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node1
  Remote node = node2
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
 
[rdbms11@node2 GI_PSU]$ /u01/app/oracle/product/11.2.0.1/db/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
 
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u01/app/oracle/product/11.2.0.1/db
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/db/oui
Log file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch2010-09-22_11-21-57AM.log
 
Patch history file: /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.1/db/cfgtoollogs/opatch/lsinv/lsinventory2010-09-22_11-21-57AM.txt
 
--------------------------------------------------------------------------------
Installed Top-level Products (1):
 
Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.
 
Interim patches (2) :
 
Patch  9654983      : applied on Wed Sep 22 11:19:38 CEST 2010
Unique Patch ID:  12651761
   Created on 18 Jun 2010, 00:16:02 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736
 
Patch  9655006      : applied on Wed Sep 22 11:19:03 CEST 2010
Unique Patch ID:  12651761
   Created on 6 Jul 2010, 12:00:17 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748
 
Rac system comprising of multiple nodes
  Local node = node2
  Remote node = node1
 
--------------------------------------------------------------------------------
 
OPatch succeeded.
]]> http://www.ora-solutions.net/web/2010/09/29/opatch-auto-problem-with-grid-infrastructure-psus/feed/ 1 Oracle Clusterware / ASM 11.1.0.7: ASM Instance crash http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/ http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/#comments Wed, 28 Oct 2009 15:27:00 +0000 Martin http://www.ora-solutions.net/web/?p=779 During a 11gR1 Clusterware installation for a Single Instance Failover Cluster at a customer site, I have experienced an interesting behaviour, which was caused by an Oracle Bug.

The environment was:

  • 2 Node Oracle Clusterware 11.1.0.7 Cluster on Linux x86_64 using latest Recommended Patches as of October 19th. (pre PSU 11.1.0.7.1)
  • Clusterware installed as unix user crs
  • ASM installed as unix user oracle

The ASM instances could be started with SQL*Plus without any problems, but if the ASM instances were started by means of clusterware using srvctl (either from root, crs or oracle) the  ASM instances would crash at diskgroup mount with:

ORA-07445: exception encountered: core dump  [sskgds_find_rtn_hdr()+1171]
[SIGBUS] [ADDR:0x2AACD701342C] [PC:0x25799DF]
[Non-existent physical address] []

Oracle Support identified this behaviour as Bug 6952915, for which there are patches for Linux x86, x86_64 and Solaris Sparc64.

]]> http://www.ora-solutions.net/web/2009/10/28/oracle-clusterware-asm-11-1-0-7-asm-instance-crash/feed/ 0 Patch Set Update (PSU) October 2009 released http://www.ora-solutions.net/web/2009/10/21/patch-set-update-psu-october-2009-released/ http://www.ora-solutions.net/web/2009/10/21/patch-set-update-psu-october-2009-released/#comments Wed, 21 Oct 2009 17:15:20 +0000 Martin http://www.ora-solutions.net/web/?p=773 Oracle has released the October 2009 Patch Set Update (PSU) which contains several interesting news:

  • first PSU, which is available for Grid Control 10.2.0.5
  • seperate PSU Patches for Clusterware (CRS)
  • Patch Set Updates are now released for ALL non-Windows platform whereas previous PSUs were released for special platforms (e.g. Linux Itanium) on request only.

More info can be found in Metalink Note 854428.1.

]]> http://www.ora-solutions.net/web/2009/10/21/patch-set-update-psu-october-2009-released/feed/ 0 High CPU Utilization – waits on cursor: pin S http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/ http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/#comments Tue, 11 Aug 2009 05:42:14 +0000 Martin http://www.ora-solutions.net/web/?p=736 I have recently encountered a problem at a customer site, where the database instances resource utilization was so high that the application did not work anymore. Version was 10.2.0.4 on Linux with Oracle Recommended Generic Patches installed.

Unfortunately, the customer decided to bounce the instance, so that there is no possibility for intensive diagnosis. However, ASH report shows that a dozen sessions either waited on Wait event “cursor: pin S” or were active (ON CPU) without any SQL_ID. An AWR Report showed:

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
cursor: pin S                   140,036,615      24,833      0 ######      Other
CPU time                                             19          72.9
log file sync                         8,767           8      1   30.2     Commit
log file parallel write               9,039           8      1   29.9 System I/O
control file parallel write           1,269           5      4   20.2 System I/O

Oracle Support confirmed that this is Bug 6904068 High CPU usage when there are “cursor: pin S” waits. We have filed a backport request for it, as there is no patch for our platform available yet.

]]> http://www.ora-solutions.net/web/2009/08/11/high-cpu-utilization-waits-on-cursor-pin-s/feed/ 0 Oracle introduces Patch Set Updates (PSU) for 10.2.0.4 Database http://www.ora-solutions.net/web/2009/07/16/oracle-introduces-patch-set-updates-psu-for-10204-database/ http://www.ora-solutions.net/web/2009/07/16/oracle-introduces-patch-set-updates-psu-for-10204-database/#comments Thu, 16 Jul 2009 12:58:39 +0000 Martin http://www.ora-solutions.net/web/?p=713 On July 14th Oracle announced on MetaLink the release of a new patching strategy for the Oracle Database.

The new Patch Set Updates (PSU) will contain cumulative patches, which contain recommended bugfixes. They will be provided on a the same quarterly basis as the Critical Patch Update (CPU), therefore release months will be January, April, July and October. The Patch Set Update will be described in the release version. E.g. 10.2.0.4.1 will be the first Patch Set Update (PSU), 10.2.0.4.2 the second PSU etc.

As described in the release information, the 10.2.0.4.1 PSU Patch (Patch 8576156) contains all the recommended patch bundles up to July 2009 (Generic, CRS, RAC, Services, DataGuard) as well as the Critical Patch Update July 2009. Moreover 5 additional critical bugfixes are included. OPatch version 10.2.0.4.7 is required for installation of PSU 10.2.0.4.1 and the PSU is rolling installable on RAC environments without downtime.

Later PSU patches can be installed on either the base release or on top of any previous PSU. For example, PSU 10.2.0.4.3 can be installed on top of Base 10.2.0.4.0, PSU 10.2.0.4.1, PSU 10.2.4.0.2.

As already mentioned, the customer has the option to install security patches only by installing the quarterly Critical Patch Update or to install security plus non-security bugfixes by installint the Patch Set Update (PSU). As the PSU 10.2.0.4.1 already contains Critical Patch Update July 2009, the documentation states that future security patches are recommended to be installed not by CPU Patches but through PSU Patches.

Further information can be found in these MetaLink Notes:

854428.1 – Intro to Patch Set Updates (PSU)
850471.1 – Oracle Announces First Patch Set Update For Oracle Database Release 10.2
8576156.8 – Bug 8576156 10.2.0.4.1 Patch Set Update (PSU)
854473.1 – Known Issues with this Patch Set Update 10.2.0.4.1

]]> http://www.ora-solutions.net/web/2009/07/16/oracle-introduces-patch-set-updates-psu-for-10204-database/feed/ 2