ora-solutions.net - Martin Decker » Real Application Clusters

Speech at DOAG Conference 2009 – RAC PreProduction Testing

Martin — Thu, 19 Nov 2009 22:06:48 +0000

I just came back from the DOAG Conference 2009, the german Oracle user group conference in Nürnberg where I had a speech about RAC PreProduction Testing. I have uploaded the slides and the paper to the papers section. At this time, the presentation is available in german only.

Multipathing Configuration issue waiting to happen

Martin — Wed, 05 Aug 2009 20:52:49 +0000

Quite some time ago, I came across a quite hard to find issue during a consulting engagement, which i find worth mentioning. A 2 node RAC cluster running on RHEL4 x86-64 was relocated to a different data center. Apart from making sure, that the switch ports and Fibre Channel Ports are available on the new location, there is not much to worry about.

After the relocation, on one node the multipathing configuration, implemented with dev-mapper-multipath would not work. The command “multipath -ll” would just not return any output. After more than an hour, we pinned the issue down to the error message:

# multipath -v 3
#
# all paths in cache :
#
…
…
…
path sdh not found in pathvec

When checking what device sdh was, we realized that this was a KVM device, plugged in by the sysadmins.

May 27 14:46:26 host1 kernel: Attached scsi removable disk sdh at scsi10, channel 0, id 0, lun 0
May 27 14:46:26 host1 kernel: Type: Direct-Access ANSI SCSI revision: 02
May 27 14:46:26 host1 kernel: Vendor: KVM Model: vmDisk Rev: 0.01
May 27 14:46:26 host1 kernel: scsi10 : SCSI emulation for USB Mass Storage devices
May 27 14:46:26 host1 kernel: sr1: scsi3-mmc drive: 0x/0x caddy
May 27 14:46:26 host1 kernel: Type: CD-ROM ANSI SCSI revision: 02
May 27 14:46:26 host1 kernel: Vendor: KVM Model: vmDisk-CD Rev: 0.01

BTW: What is a KVM device?
Wikipedia states: A KVM switch (with KVM being an abbreviation for Keyboard, Video or Visual Display Unit, Mouse) is a hardware device that allows a user to control multiple computers from a single keyboard, video monitor and mouse.

We then added the device sdh to the multipath blacklist section in /etc/multipath.conf, and the problem was solved:

devnode_blacklist {
devnode “^sdh$”
}

Default 10gR2 RAC TNS Configuration can cause TCP Timeouts for application

Martin — Tue, 12 May 2009 14:57:06 +0000

The default RAC installation does normally not set “local_listener” init.ora parameter. If the listener is running on port 1521, then the database does not need the parameter in order to find and register with the local TNS listener process. However, if you have *not* set local_listener, it means that the database registers at the listener with the physical IP address instead of the virtual (vip) address.

You can determine if this happens when you take a look at “lsnrctl serv” output from your rac nodes:

Service "S_APP" has 1 instance(s).
  Instance "MDDB1", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=ora-vm1.intra)(PORT=1521))

Instead of ora-vm1.intra, this should be ora-vm1-vip.intra.

Why should I care?

If you use the default configuration, then you are using the parameter “REMOTE_LISTENER” and therefore Server Side Connect Time Load Balancing. This means, that the listeners of all nodes receive load information from all instances of all nodes and they can redirect connections to the least loaded instance, even if the instance is on another node. But the connect string they then send back to the client contains the physical IP address instead of the virtual.

In case of node crashes or kernel panics, etc. the client has to wait for the TCP timeout until this is detected.

Solution

tnsnames.ora:

LISTENER_MDDB1 =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ora-vm1-vip.intra)(PORT = 1521))
  )

LISTENER_MDDB2 =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ora-vm2-vip.intra)(PORT = 1521))
  )

init.ora:
alter system set local_listener = 'LISTENER_MDDB1' sid='MDDB1';
alter system set local_listener = 'LISTENER_MDDB2' sid='MDDB2';

Your experience with RAC Dynamic Remastering (DRM) in 10gR2?

Martin — Tue, 12 May 2009 14:37:48 +0000

One of my customers is having severe RAC performance issues, which appeared a dozen times so far. Each time, the performance impact lasted around 10 minutes and caused basically a hang of the application. ASH investigation revealed that the time frame of performance issues exactly matches a DRM operation of the biggest segment of the database. During the problematic time period, there are 20-50 instead of 5-10 active sessions and they are mostly waiting for gc related events: “gc buffer busy”,”gc cr block busy”, “gc cr block 2-way”, “gc current block 2-way”, “gc current request”, “gc current grant busy”, etc.

In addition, there is one single session which has wait event “kjbdrmcvtq lmon drm quiesce: ping completion” (on instance 1) and 1-3 sessions with wait event “gc remaster“. (on instance 2) The cur_obj# of the session waiting on “gc remaster” is pointing to the segment being remastered.

Does anybody have any experience with DRM problems with 10.2.0.4 on Linux Itanium?

I know that it is possible to deactive DRM, but usually it should be beneficial to have it enabled. I could not find any reports of performance impact during DRM operation on metalink. Support is involved but clueless so far.

Regards,
Martin

http://forums.oracle.com/forums/message.jspa?messageID=3447436#3447436

Autostart Oracle Services at Clusterware startup

Martin — Thu, 19 Mar 2009 09:54:53 +0000

I recently recommended the utilization of Oracle Services to a new client. I knew that you have to start a service manually if you restart your cluster. I told them that I will investigate on how to set the services to autostart. I knew that there is an AUTO_START property of the clusterware service resource set to “restore”, which means that at instance crash, Clusterware will restore the services to the status before crash. However, on manual shutdown/startup, the services will not be autostarted.

[root@ora-vm1 tmp]# crs_stat -p ora.MDDB.S_TEST.MDDB1.srv
NAME=ora.MDDB.S_TEST.MDDB1.srv
TYPE=application
AUTO_START=restore
…

I did some research and learned that the easiest way to configure autostart is with user callouts. A quick web search referred me to the OTN site “Real Application Clusters Sample Scripts” which contains a FAN callout script to autostart services.

The steps to install the scripts are:

Copy them to $ORA_CRS_HOME/racg/usrco on both nodes
Set permissions to 710 and ownership to oracle:oinstall or whatever your installation is using.
Modify perl Scripts to contain your ORA_CRS_HOME, ORACLE_HOME. Moreover, if you have used short hostnames for your CRS installation, you have to change “/bin/hostname” to “/bin/hostname -s”.

I just realized that Dan Norris also blogged about this topic.

Installing LTOM for RAC Hanganalyze

Martin — Wed, 17 Dec 2008 12:40:35 +0000

I have recently written a short installation instruction for a customer to use LTOM for RAC hanganalyze.

Download von MetaLink Note 352363.1:
https://metalink2.oracle.com/cgi-bin/cr/getfile.cgi?p_attid=352363.1:ltom

unzip as User Oracle in directory with sufficient space. subdir ltom will be created

tar xfv ltom411.tar

Install statspack

Installation: db account ltom/ltom will be created

Java Binary in path
which java

if not, place java in path in ~/.bash_profile:
als User oracle: add $ORACLE_HOME/jre/1.4.2/bin to PATH Variable

e.g.:
export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/jre/1.4.2/bin

Put TOM_HOME in .~/.bash_profile:
export TOM_HOME=/home/oracle/ltom/tom_base/install/../tom

Sourcen of TOM_HOME

cd /home/oracle/ltom/tom_base/install
./autoinstall.sh

Sample Output:
[oracle@ora-vm1 install]$ ./autoinstall.sh
TOM_HOME=/home/oracle/ltom/tom_base/install/../tom
rm: cannot remove `autoinstall.log’: No such file or directory

Preparing to install LTOM v4.0.0…

Checking your Oracle Environment…

ORACLE_HOME=/u01/app/oracle/product/10.2.0
ORACLE_SID=ORA102
TNS_ADMIN=
TOM_HOME=/home/oracle/ltom/tom_base/install/../tom

Is this the correct Oracle environment you wish to use for installing LTOM?
Hit Return to Accept Default:
============================> \c

Continuing the install with this environment

Checking for Java…

Java found.

Checking for top utility…
Found top utility.

Checking for vmstat utility…
Found vmstat utility.

Need to run some sql now to create TOM user
it Return to allow setup to do this for you, enter other to do this manually:
============================> \c

Tom user created successfully.
Warning…Please change tom password immediately!

Grant user Tom successful.
Create package dbmssupp successful.
Grant execute on dbmssupp successful.

Checking for statspack…

Cannot verify statspack is installed on your system.
LTOM requires statspack to be installed to work properly.

Enter your email id to get notification of hangs via email…
Hit Return to ignore hang notification via email:
============================> \c

Configuring tom_deploy.properties file…

TOM/LTOM successfully installed.

Please review any messages you received above.

REMEMBER: You must have TOM_HOME defined and java in your PATH
before you can run TOM/LTOM.

Configuration

Change ltom/tom_base/tom/init/hangDetect.properties

from:
#RAC_AWARE=TRUE
HANGANALYZE=A
SYSTEMSTATE=266
DELAY=30
HANGANALYZE=A
SYSTEMSTATE=266

to:
RAW_AWARE=TRUE
HANGANALYZE=A
SYSTEMSTATE=266
DELAY=30
HANGANALYZE=A
SYSTEMSTATE=266

Change tom db passwort:

SQL> alter user tom identified by secret;

Add credentials to config file:

vi tom_deploy.properties

DB_USERID=tom
:q!

./startltom.sh -p
Enter same password as in sqlplus. Password will be saved encrypted in tom_deploy.properties

edit tom_deploy.properties to decide which of the 3 modules should be running during background operations: hangdetect, profiler, session recorder

Start/Stop Scripts

vi start_ltom.sh
#!/bin/bash
. ~/.bash_profile
nohup $TOM_HOME/startltom.sh -s &

vi stop_ltom.sh
#!/bin/bash
touch $TOM_HOME/tmp/hang.stop
touch $TOM_HOME/tmp/profile.stop
touch $TOM_HOME/tmp/session.stop

chmod u+x start_ltom.sh
chmod u+x stop_ltom.sh

If you want to run ltom during a specific timeframe, use cron (as user oracle)

crontab -e
45 01 * * * /home/oracle/ltom/start_ltom.sh >/dev/null 2>/dev/null 00 05 * * * /home/oracle/ltom/start_ltom.sh >/dev/null 2>/dev/null

Troublshooting Trace Files of DIAG Background Process

Martin — Wed, 10 Dec 2008 16:25:18 +0000

In trace files of the background process DIAG, I saw an interesting section that I would like to share.

*** 2008-12-09 03:46:33.280 Dump requested by process [orapid=6] REQUEST:custom dump [2] with parameters [6][27][4][2] Dumping process info of pid[27.31384] requested by pid[6.31160] ORA-00494: Message 494 not found; No message file for product=RDBMS, facility=ORA; arguments: [ [CF]] [ (more than 900 seconds)] [inst 2, osid 31384] ------------------------------------------------------------------------------- ENQUEUE [CF] HELD FOR TOO LONG enqueue holder: 'inst 2, osid 31384' Process 'inst 2, osid 31384' is holding an enqueue for maximum allowed time. The process will be terminated. Oracle Support Services triaging information: to find the root-cause, look at the call stack of process 'inst 2, osid 31384' located below. Ask the developer that owns the first NON-service layer in the stack to investigate. Common service layers are enqueues (ksq), latches (ksl), library cache pins and locks (kgl), and row cache locks (kqr).

It is very nice of diag to tell me what to do with the output. Especially:

Ask the developer that owns the first NON-service layer in the stack to investigate.

System State Dump Evaluation with ass.awk

Martin — Wed, 10 Dec 2008 16:21:31 +0000

I have found out that there is an Oracle Support utility called ass.awk, which can be used to analyze and format a System State dump. It is contained in LTOM, which can be downloaded from MetaLink.

The AWK script is called ass109.awk and can be used in this manner:

[oracle@ora-vm1 ~]$ awk -f ass109.awk mddb1_diag_12345.trc
System State ASS: Starting Systemstate 1 .............................................................................. ..... Ass.Awk Version 1.0.9 - Processing mddb1_diag_12345.trc System State 1 ~~~~~~~~~~~~~~~~ 1: 2: waiting for 'pmon timer' wait 3: last wait for 'ksdxexeotherwait' 4: waiting for 'rdbms ipc message' wait 5: waiting for 'rdbms ipc message' wait 6: waiting for 'ges remote message' wait 7: waiting for 'gcs log flush sync' wait 8: waiting for 'gcs log flush sync' wait 9: waiting for 'rdbms ipc message' wait 10: waiting for 'rdbms ipc message' wait 11: waiting for 'enq: CF - contention' [Enqueue CF-00000000-00000000] wait 12: waiting for 'rdbms ipc message' wait 13: waiting for 'buffer busy waits' (2,a9,3b) wait 14: waiting for 'rdbms ipc message' wait 15: waiting for 'rdbms ipc message' wait 16: waiting for 'buffer busy waits' (2,a9,3b) wait 17: waiting for 'rdbms ipc message' wait 18: 19: 20: waiting for 'rdbms ipc message' wait 21: waiting for 'log file sync' wait 22: waiting for 'ASM background timer' wait 23: waiting for 'rdbms ipc message' wait 24: waiting for 'log file sync' wait 25: waiting for 'class slave wait' wait 26: waiting for 'SQL*Net message from client' wait 27: waiting for 'SQL*Net message from client' wait 28: waiting for 'enq: CF - contention' [Enqueue CF-00000000-00000000] wait 29: waiting for 'enq: CF - contention' [Enqueue CF-00000000-00000000] wait 30: waiting for 'Streams AQ: qmn coordinator idle wait' wait 31: waiting for 'SQL*Net message from client' wait 32: waiting for 'SQL*Net message from client' wait Cmd: Select 33: waiting for 'log file sync' wait 34: waiting for 'Streams AQ: qmn slave idle wait' wait 35: for 'Streams AQ: waiting for time management or cleanup tasks' wait 36: waiting for 'log file sync' wait 37: waiting for 'SQL*Net message from client' wait Cmd: Select 38: waiting for 'SQL*Net message from client' wait 39: waiting for 'SQL*Net message from client' wait 40: waiting for 'PX Deq: Execution Msg' wait 41: waiting for 'log file sync' wait 42: waiting for 'log file sync' wait 43: waiting for 'enq: CF - contention' [Enqueue CF-00000000-00000000] wait Cmd: Select 44: waiting for 'SQL*Net message from client' wait 45: waiting for 'SQL*Net message from client' wait 46: waiting for 'SQL*Net message from client' wait 47: waiting for 'SQL*Net message from client' wait 48: waiting for 'SQL*Net message from client' wait 49: waiting for 'log file sync' wait 50: waiting for 'SQL*Net message from client' wait 51: waiting for 'SQL*Net message from client' wait Cmd: Select 52: waiting for 'row cache lock' [Rcache object=0xa5414098,] wait Cmd: Select 53: waiting for 'SQL*Net message from client' wait 54: waiting for 'SQL*Net message from client' wait 55: for 'Streams AQ: waiting for messages in the queue' wait 56: waiting for 'SQL*Net message from client' wait 57: waiting for 'SQL*Net message from client' wait 58: waiting for 'log file sync' wait Cmd: Insert 59: waiting for 'jobq slave wait' wait 60: waiting for 'jobq slave wait' wait 61: waiting for 'SQL*Net message from client' wait 62: waiting for 'SQL*Net message from client' wait 63: waiting for 'SQL*Net message from client' wait 65: waiting for 'SQL*Net message from client' wait Cmd: Select 66: waiting for 'SQL*Net message from client' wait 67: last wait for 'SQL*Net message from client' 68: waiting for 'SQL*Net message from client' wait 69: waiting for 'SQL*Net message from client' wait 70: waiting for 'SQL*Net message from client' wait 71: waiting for 'SQL*Net message from client' wait 72: waiting for 'SQL*Net message from client' wait 73: waiting for 'SQL*Net message from client' wait 74: waiting for 'SQL*Net message from client' wait Cmd: Select 75: waiting for 'SQL*Net message from client' wait Cmd: Select 76: waiting for 'SQL*Net message from client' wait 77: waiting for 'log file sync' wait 78: waiting for 'SQL*Net message from client' wait 80: waiting for 'SQL*Net message from client' wait Cmd: Select 82: waiting for 'SQL*Net message from client' wait 84: waiting for 'log file sync' wait 86: waiting for 'SQL*Net message from client' wait 93: waiting for 'SQL*Net message from client' wait 94: waiting for 'SQL*Net message from client' wait Blockers ~~~~~~~~ Above is a list of all the processes. If they are waiting for a resource then it will be given in square brackets. Below is a summary of the waited upon resources, together with the holder of that resource. Notes: ~~~~~ o A process id of '???' implies that the holder was not found in the systemstate. Resource Holder State Enqueue CF-00000000-00000000 12: waiting for 'rdbms ipc message' Rcache object=0xa5414098, ??? Blocker Object Names ~~~~~~~~~~~~ Enqueue CF-00000000-00000000 Rcache object=0xa5414098,

With this information, you can investigate further into the System state dump by looking at rcache object 0xa5414098. The “State Objects” are a hierarchical structure. Every state object has an adress “SO: 0xa6a5955c” and an owner, which is the parent entry. In this case, you can construct a hierarchical tree of state objects and find dependencies.

SO: 0xa6a5995c, type: 2, owner: (nil), flag: INIT/-/-/0x00 (process) Oracle pid=52, calls cur/top: 0x9d62eb1c/0x9d5ccfcc, flag: (0) - int error: 0, call error: 0, sess error: 0, txn error 0 (post info) last post received: 0 0 163 last post received-location: kqrbtm last process to post me: a6a4e1dc 1 6 last post sent: 0 0 24 last post sent-location: ksasnd last process posted by me: a6a4e1dc 1 6 (latch info) wait_event=0 bits=0 Process Group: DEFAULT, pseudo proc: 0xa6baf79c O/S info: user: oracle, term: UNKNOWN, ospid: 29032 OSD pid info: Unix process pid: 29032, image: oracle@ora-vm1 Short stack dump: ksdxfstk()+19<-ksdxcb()+1321<-sspuser()+102<-__kernel_vsyscall()+16
SO: 0xa4123d20, type: 4, owner: 0xa6a5995c, flag: INIT/-/-/0x00 (session) sid: 955 trans: (nil), creator: 0xa6a5995c, flag: (41) USR/- BSY/-/-/-/-/- DID: 0001-0034-00002227, short-term DID: 0001-0034-00002228 txn branch: (nil) oct: 3, prv: 0, sql: 0x8fee9d44, psql: 0x8fee9f00, user: 23/DBSNMP service name: SYS$USERS O/S info: user: oracle, term: , ospid: 28849, machine: ora-vm1 program: emagent@ora-vm1(TNS V1-V3) application name: emagent@ora-vm1(TNS V1-V3), hash value=2420928497 waiting for 'row cache lock' blocking sess=0x(nil) seq=127 wait_time=0 seconds since wait started=0 cache id=d, mode=0, request=5 Dumping Session Wait History for 'row cache lock' count=1 wait_time=1337230 cache id=d, mode=0, request=5 for 'row cache lock' count=1 wait_time=2930561 cache id=d, mode=0, request=5 for 'row cache lock' count=1 wait_time=2929106 cache id=d, mode=0, request=5 for 'row cache lock' count=1 wait_time=2933326
SO: 0x9d5ccfcc, type: 3, owner: 0xa6a5995c, flag: INIT/-/-/0x00 (call) sess: cur a4123d20, rec a4123d20, usr a4123d20; depth: 0 ---------------------------------------- SO: 0x9d51b1a0, type: 3, owner: 0x9d5ccfcc, flag: INIT/-/-/0x00 (call) sess: cur a4123d20, rec a417f990, usr a4123d20; depth: 1 ---------------------------------------- SO: 0x9d69d7f0, type: 3, owner: 0x9d51b1a0, flag: INIT/-/-/0x00 (call) sess: cur a417f990, rec a417f990, usr a4123d20; depth: 2 ---------------------------------------- SO: 0x9d5e547c, type: 3, owner: 0x9d69d7f0, flag: INIT/-/-/0x00 (call) sess: cur a417f990, rec a417f990, usr a4123d20; depth: 3 ---------------------------------------- SO: 0x9d62eb1c, type: 3, owner: 0x9d5e547c, flag: INIT/-/-/0x00 (call) sess: cur a417f990, rec a417f990, usr a4123d20; depth: 4 ---------------------------------------- SO: 0x9fedf748, type: 40, owner: 0x9d62eb1c, flag: INIT/-/-/0x00 (trans) flg = 0x21, flg2 = 0x80000, prx = 0x0, ros = 2147483647 bsn = 0x395 bndsn = 0x395 spn = 0x395 efd = 34 parent xid: 0x0000.000.00000000 env: (scn: 0x0000.00000000 xid: 0x0000.000.00000000 uba: 0x00000000.0000.00 statement num=0 parent xid: xid: 0x0000.000.00000000 scn: 0x0000.00000000 0sch: scn: 0x0000.00000000) cev: (spc = 0 arsp = 0 ubk tsn: 0 rdba: 0x00000000 useg tsn: 0 rdba: 0x00000000 hwm uba: 0x00000000.0000.00 col uba: 0x00000000.0000.00 num bl: 0 bk list: 0x0) cr opc: 0x0 spc: 0 uba: 0x00000000.0000.00 (enqueue) TX-00010011-00D3B4CF DID: 0001-0051-00001323 lv: 7c ac ad 4d 03 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x45 res: 0xa447bb2c, lock_flag: 0x0 own: 0xa4157e90, sess: 0xa4157e90, prv: 0x9fedf774 slk: (nil) xga: 0x0, heap: UGA Trans IMU st: 0 Pool index 65535, Redo pool 0x9fedfb74, Undo pool 0x9fedfbf0 Redo pool range [0x40ed57e8 0x40ed57e8 0x40ed7fe8] Undo pool range [0x40ed2fe8 0x40ed2fe8 0x40ed57e8] ---------------------------------------- SO: 0x9d42237c, type: 50, owner: 0x9fedf748, flag: INIT/-/-/0x00 row cache enqueue: count=1 session=0xa4123d20 object=0xa5414098, request=X savepoint=0x395 row cache parent object: address=0xa5414098 cid=13(dc_sequences) hash=9649e7e typ=9 transaction=(nil) flags=00000000 own=0xa5414104[0xa5414104,0xa5414104] wat=0xa541410c[0x9d422398,0x9d422398] mode=N status=-/-/-/-/-/-/-/-/- request=X release=FALSE flags=2 instance lock id=QN 010cdbb3 b3b24847

Tanel Poder showed in his seminar to use "grep -A 3 SO:" to find all lines containing the State Object Adresses and the following 3 lines to get the tree.

Add Dependency of Database Instance on ASM to OCR

Martin — Tue, 02 Dec 2008 22:30:03 +0000

Today at a presentation of Grégory Guillou and Alex Gorbachev from The Pythian Group at UKOUG 2008, I learned that the dependency of RAC database instances on ASM is not set up when adding the database instances to OCR via srvctl add instance.

I was wondering in the past, why the database instances would sometimes not start up correctly when booting and fail at loading the spfile from ASM.

So, if you did not use DBCA to create and register your RAC database with OCR but srvctl, as shown here:



 $ srvctl add database -d BOSTON –o /opt/oracle/product/10g_db_rac

 $ srvctl add instance -d BOSTON -i BOSTON1 -n boston_host1

 $ srvctl add instance -d BOSTON -i BOSTON2 -n boston_host2

then you have to create this dependency manually in order to determine the appropriate startup order:


$ srvctl modify instance –d BOSTON –i BOSTON1 –s +ASM1

$ srvctl modify instance –d BOSTON –i BOSTON2 –s +ASM2

Excellent Presentations on

Martin — Sun, 16 Nov 2008 18:27:08 +0000

The database specialist Riyaj Shamsudeen from The Pythian Group has published some excellent presentations on his blog. Don´t miss it!