If you are managing volatile systems, you might have encountered Archiver-Stuck in the past.

1
2

0RA-00257:archiver error, connect internal only until freed
ORA-16014:log 2 sequence# 1789 not archived, no available destinations

If this is the case, you might be interested in a rarely used feature called “ALTERNATE Archive Locations”. This is a second independent archive log destination, that is only used in case the primary destination is failing (full). In that case, the dest_2 is automatically enabled, preventing stuck database. As soon as the archivelog backup has cleaned up the full DEST_1 destination, there is an AUTOMATIC FALLBACK to the DEST_1 destination.

1
2
3
4

ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=/disk1 MANDATORY MAX_FAILURE=1 ALTERNATE=LOG_ARCHIVE_DEST_2';
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_1=ENABLE;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ALTERNATE;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='LOCATION=/disk2 MANDATORY ALTERNATE=LOG_ARCHIVE_DEST_1';

Testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Thread 1 advanced to log sequence 174 (LGWR switch)
  Current log# 3 seq# 174 mem# 0: /u01/app/oracle/oradata/CDB122/redo03.log
2018-04-27T17:18:47.685736+02:00
ARC3: Encountered disk I/O error 19502
2018-04-27T17:18:47.685800+02:00
ARC3: Closing local archive destination LOG_ARCHIVE_DEST_1 '/disk1/1_173_971869447.dbf' (error 19502) (CDB122)
2018-04-27T17:18:47.687074+02:00
Errors in file /u01/app/oracle/diag/rdbms/cdb122/CDB122/trace/CDB122_arc3_31260.trc:
ORA-27072: File I/O error
Additional information: 4
Additional information: 65536
Additional information: 315392
ORA-19502: write error on file "/disk1/1_173_971869447.dbf", block number 65536 (block size=512)
2018-04-27T17:18:47.816938+02:00
Errors in file /u01/app/oracle/diag/rdbms/cdb122/CDB122/trace/CDB122_arc3_31260.trc:
ORA-19502: write error on file "/disk1/1_173_971869447.dbf", block number 65536 (block size=512)
ORA-27072: File I/O error
Additional information: 4
Additional information: 65536
Additional information: 315392
ORA-19502: write error on file "/disk1/1_173_971869447.dbf", block number 65536 (block size=512)
ARC3: I/O error 19502 archiving log 2 to '/disk1/1_173_971869447.dbf'

After the error, archivelogs are placed in /dir2 directory and as soon as space is available again, archive destination is switched back to /dir1 automatically.

Archiver Stuck never again, right?

Gone are the day when cleartext passwords had to be stored in scripts for Oracle database access. The solution to this requirement is “Oracle Secure External Password Store (SEPS)”. This article will give a short introduction and a practical example of the solution.

Key facts:

Installation

• Oracle Client: Unix-Account, who wishes to connect to the database without providing a password needs to have an Oracle Client installed. Actually it can be an Instant Client, however the Instant Client does not have the Oracle Wallet Manager (owm).
• TNS_ADMIN: If the installed Oracle-Client is also used by other unix-accounts, you have to set Environment-Variable TNS_ADMIN to point to a dedicated destination for sqlnet.ora

echo "export TNS_ADMIN=/home/techuser/oracle/network/admin" >> ~/.bash_profile

• sqlnet.ora in $TNS_ADMIN: In this file, the path to your personal wallet is defined:

WALLET_LOCATION =
(
   SOURCE =
      (METHOD = FILE)
      (METHOD_DATA =
         (DIRECTORY = /home/techuser/oracle/network/wallet)
      )
)
SQLNET.WALLET_OVERRIDE = TRUE
SSL_CLIENT_AUTHENTICATION = FALSE

Be aware that sqlnet.ora has very sensitive formatting! Blank at the beginning of the line means that previous line is continued. If the first character of a line is not a blank, then it is supposed to be new directive. Therefore note the identation of wallet_location parameter.

• tnsnames.ora in $TNS_ADMIN: This file contains TNS Aliases, which are used to connect to database. If the unix-account needs to  connect to one database but with 2 different database accounts, then it needs to use 2 distinct tns aliases!
• Creation of Wallet: When the wallet is created, a wallet password is requested. This password is needed when viewing, adding, modifying and deleting wallet entries.

mkstore –wrl /home/techuser/oracle/network/wallet –create

Management of Credentials in Wallet:

mkstore -wrl /home/techuser/oracle/network/wallet \
-createCredential

mkstore -wrl /home/techuser/oracle/network/wallet \
-deleteCredential TNSAlias

mkstore -wrl /home/techuser/oracle/network/wallet \
-modifyCredential N102 scott newpassword
Enter password:
Modify credential Modify 1
 
mkstore -wrl /home/techuser/oracle/network/wallet \
-modifyEntry oracle.security.client.password1 newpass
Enter password:

mkstore -wrl /home/techuser/oracle/network/wallet -list
 
Oracle Secret Store entries:
oracle.security.client.connect_string1
oracle.security.client.password1
oracle.security.client.username
 
mkstore -wrl /home/techuser/oracle/network/wallet \
-viewEntry oracle.security.client.connect_string1
Enter password: 
 
oracle.security.client.connect_string1 = N102
 
mkstore -wrl /home/techuser/oracle/network/wallet -viewEntry oracle.security.client.username1
Enter password:
oracle.security.client.username1 = scott
 
mkstore -wrl /home/techuser/oracle/network/wallet -viewEntry oracle.security.client.password1
Enter password:
oracle.security.client.password1 = tiger

• Changing Wallet Password

orapki wallet change_pwd -wallet /home/techuser/oracle/network/wallet \
-oldpwd welcome1 -newpwd welcome2

Example:

DBs:
 
TESTDB1 (11.2.0.3)
TESTDB2 (11.2.0.3)
 
Techuser:
techuser1 - uses RDBMS Server Home als Client
techuser2 - uses Oracle Instant Client
 
Requirements:
techuser1 should be able TO CONNECT TO ro_user@TESTDB2 AND rw_user@TESTDB2
techuser2 should be able TO CONNECT TO ro_user@TESTDB2 AND rw_user@TESTDB1 
 
Preparations:
 
AS oracle:
. TESTDB2.env
 
sqlplus "/as sysdba"
 
SQL> GRANT CREATE SESSION TO ro_user IDENTIFIED BY ro_password;
 
GRANT succeeded.
 
SQL> GRANT CREATE SESSION TO rw_user IDENTIFIED BY rw_password;
 
GRANT succeeded.
 
. TESTDB1.env
 
sqlplus "/as sysdba"
 
SQL> GRANT CREATE SESSION TO rw_user IDENTIFIED BY rw_password;
 
GRANT succeeded.
 
AS root:
useradd techuser1
useradd techuser2
 
su - techuser1
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH
export TNS_ADMIN=/home/techuser1/oracle/network/admin
mkdir -p oracle/network/admin oracle/network/wallet
 
vi $TNS_ADMIN/sqlnet.ora:
WALLET_LOCATION =
   (SOURCE =
      (METHOD = FILE)
         (METHOD_DATA=
           (DIRECTORY=/home/techuser1/oracle/network/wallet)
         )
    )
 
SQLNET.WALLET_OVERRIDE=TRUE
SSL_CLIENT_AUTHENTICATION=FALSE
 
vi $TNS_ADMIN/tnsnames.ora:
 
TESTDB2_ro =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TESTDB2_SITE1)
    )
  )
 
TESTDB2_rw =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TESTDB2_SITE1)
    )
  )
 
[techuser1@oravm1 admin]$ tnsping TESTDB2_ro
 
TNS Ping Utility FOR Linux: Version 11.2.0.3.0 - Production ON 01-MAR-2012 11:13:34
 
Copyright (c) 1997, 2011, Oracle.  ALL rights reserved.
 
Used parameter files:
/home/techuser1/oracle/network/admin/sqlnet.ora
 
Used TNSNAMES adapter TO resolve the alias
Attempting TO contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)
(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TESTDB2_SITE1)))
OK (0 msec)
[techuser1@oravm1 admin]$ tnsping TESTDB2_rw
 
TNS Ping Utility FOR Linux: Version 11.2.0.3.0 - Production ON 01-MAR-2012 11:13:36
 
Copyright (c) 1997, 2011, Oracle.  ALL rights reserved.
 
Used parameter files:
/home/techuser1/oracle/network/admin/sqlnet.ora
 
Used TNSNAMES adapter TO resolve the alias
Attempting TO contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)
(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TESTDB2_SITE1)))
OK (10 msec)
 
[techuser1@oravm1 admin]$ mkstore -wrl /home/techuser1/oracle/network/wallet -CREATE
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter password:
Enter password again:
 
mkstore -wrl /home/techuser1/oracle/network/wallet \
 -createCredential TESTDB2_rw rw_user rw_password
 
mkstore -wrl /home/techuser1/oracle/network/wallet \
-createCredential TESTDB2_ro ro_user ro_password
 
[techuser1@oravm1 admin]$ mkstore -wrl /home/techuser1/oracle/network/wallet \
-createCredential TESTDB2_rw rw_user rw_password
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter wallet password:
CREATE credential oracle.security.client.connect_string1
 
[techuser1@oravm1 admin]$ mkstore -wrl /home/techuser1/oracle/network/wallet \
 -createCredential TESTDB2_ro ro_user ro_password
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter wallet password:
CREATE credential oracle.security.client.connect_string2
 
[techuser1@oravm1 admin]$ sqlplus /@TESTDB2_rw
SQL*Plus: Release 11.2.0.3.0 Production ON Thu Mar 1 11:19:40 2012
 
Copyright (c) 1982, 2011, Oracle.  ALL rights reserved.
 
Connected TO:
Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP, DATA Mining AND REAL Application Testing options
 
SQL> SHOW USER
USER IS "RW_USER"
SQL> quit
Disconnected FROM Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP, DATA Mining AND REAL Application Testing options
[techuser1@oravm1 admin]$ sqlplus /@TESTDB2_ro
 
SQL*Plus: Release 11.2.0.3.0 Production ON Thu Mar 1 11:19:45 2012
 
Copyright (c) 1982, 2011, Oracle.  ALL rights reserved.
 
Connected TO:
Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP, DATA Mining AND REAL Application Testing options
 
SQL> SHOW USER
USER IS "RO_USER"
 
[techuser1@oravm1 wallet]$ ls -la /home/techuser1/oracle/network/wallet/
total 24
drwxrwxr-x 2 techuser1 techuser1 4096 Mar  1 11:15 .
drwxrwxr-x 4 techuser1 techuser1 4096 Mar  1 11:01 ..
-rw------- 1 techuser1 techuser1 4341 Mar  1 11:17 cwallet.sso
-rw------- 1 techuser1 techuser1 4264 Mar  1 11:17 ewallet.p12
[techuser1@oravm1 wallet]$
 
techuser2 WITH instant client:
 
unzip instantclient-basic-linux.x64-11.2.0.3.0.zip
unzip instantclient-sqlplus-linux.x64-11.2.0.3.0.zip
 
export ORACLE_HOME=/home/techuser2/instantclient_11_2
export PATH=$ORACLE_HOME:$PATH
export TNS_ADMIN=/home/techuser2/oracle/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME:$LD_LIBRARY_PATH
 
[techuser2@oravm1 ~]$ mkdir -p /home/techuser2/oracle/network/admin
[techuser2@oravm1 ~]$ mkdir -p /home/techuser2/oracle/network/wallet
 
vi $TNS_ADMIN/sqlnet.ora
 
WALLET_LOCATION =
   (SOURCE =
      (METHOD = FILE)
         (METHOD_DATA=
           (DIRECTORY=/home/techuser2/oracle/network/wallet)
         )
    )
 
SQLNET.WALLET_OVERRIDE=TRUE
SSL_CLIENT_AUTHENTICATION=FALSE
 
vi $TNS_ADMIN/tnsnames.ora
 
TESTDB2_ro =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TESTDB2_SITE1)
    )
  )
 
TESTDB1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oravm1.intra)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TESTDB1)
    )
  )
 
Test interactive Password:
 
[techuser2@oravm1 instantclient_11_2]$ sqlplus ro_user@TESTDB2_ro
 
SQL*Plus: Release 11.2.0.3.0 Production ON Thu Mar 1 11:28:53 2012
 
Copyright (c) 1982, 2011, Oracle.  ALL rights reserved.
 
Enter password:
 
Connected TO:
Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WITH the Partitioning, OLAP, DATA Mining AND REAL Application Testing options
 
SQL> quit
 
Problem: no mkstore Tool IN Instantclient
Workaround: CREATE wallet WITH oracle  binaries owner AND copy afterwards
 
[oracle@oravm1 ~]$ mkdir /tmp/wallet
[oracle@oravm1 ~]$  /u01/app/oracle/product/11.2.0/dbhome_1/bin/mkstore \
-wrl /tmp/wallet -CREATE
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter password:
Enter password again:
[oracle@oravm1 ~]$
[oracle@oravm1 ~]$
[oracle@oravm1 ~]$
[oracle@oravm1 ~]$
[oracle@oravm1 ~]$ /u01/app/oracle/product/11.2.0/dbhome_1/bin/mkstore \
-wrl /tmp/wallet -createCredential TESTDB2_ro ro_user ro_password
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter wallet password:
 
CREATE credential oracle.security.client.connect_string1
[oracle@oravm1 ~]$ /u01/app/oracle/product/11.2.0/dbhome_1/bin/mkstore \
-wrl /tmp/wallet -createCredential TESTDB1 rw_user rw_password
Oracle Secret Store Tool : Version 11.2.0.3.0 - Production
Copyright (c) 2004, 2011, Oracle AND/OR its affiliates. ALL rights reserved.
 
Enter wallet password:
CREATE credential oracle.security.client.connect_string2
 
[techuser2@oravm1 wallet]$ ls -al
total 28
drwxr-xr-x  2 oracle oinstall 4096 Mar  1 11:32 .
drwxrwxrwt 31 root   root     4096 Mar  1 11:32 ..
-rw-------  1 oracle oinstall 4333 Mar  1 11:33 cwallet.sso
-rw-------  1 oracle oinstall 4256 Mar  1 11:33 ewallet.p12
 
[root@oravm1 techuser2]# chown -R techuser2:techuser2 /tmp/wallet
 
[root@oravm1 techuser2]# ls -la /tmp/wallet/
total 28
drwxr-xr-x  2 techuser2 techuser2 4096 Mar  1 11:32 .
drwxrwxrwt 31 root      root      4096 Mar  1 11:35 ..
-rw-------  1 techuser2 techuser2 4333 Mar  1 11:33 cwallet.sso
-rw-------  1 techuser2 techuser2 4256 Mar  1 11:33 ewallet.p12
 
[techuser2@oravm1 wallet]$ pwd
/home/techuser2/oracle/network/wallet
[techuser2@oravm1 wallet]$ mv /tmp/wallet/* .
[techuser2@oravm1 wallet]$ ls -la
total 24
drwxrwxr-x 2 techuser2 techuser2 4096 Mar  1 11:36 .
drwxrwxr-x 4 techuser2 techuser2 4096 Mar  1 11:25 ..
-rw------- 1 techuser2 techuser2 4333 Mar  1 11:33 cwallet.sso
-rw------- 1 techuser2 techuser2 4256 Mar  1 11:33 ewallet.p12
 
[techuser2@oravm1 wallet]$ sqlplus /@TESTDB2_ro
 
SQL*Plus: Release 11.2.0.3.0 Production on Thu Mar 1 11:36:58 2012
 
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
 
SQL> show user
USER is "RO_USER"
SQL> quit
 
[techuser2@oravm1 wallet]$ sqlplus /@TESTDB1
 
SQL*Plus: Release 11.2.0.3.0 Production on Thu Mar 1 11:38:06 2012
 
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
 
SQL> show user
USER is "RW_USER"

Referenz:

In case you have not heard all the buzz about the Oracle SCN flaw, which was published by Infoworld after the release of January CPU, I summarize the essentials.

• There is a risk that the SCN reaches it´s maximum value and this could lead to an outage of the database.
• There is a bug in “ALTER DATABASE BEGIN BACKUP”, which increases the SCN dramatically. (Bug 12371955 – fixed in 11.2.0.2.4 and others)
• When you query a remote database via database link from a database with elevated SCN, the remote SCN gets increased to the higher value as well. This has the dramatic effect, that a database will infect the other database.
• Imagine an unpatched environment with user-managed online backups (e.g. for SAN split mirror technology) and dozens of interconnected databases with database links.
• Now imagine an evil employee with this little innocent database account with only “create session” privilege  … you get the picture.
• The January 2012 CPU/PSU contains a patch that should restrict the ways to increase the SCN. However, I am not sure that the issue with db links is solved. I could bump up the SCN to 12562779343042 by a remote select even after patching with CPU January 2012.

Additionally, there are already quite a few articles showing how a potential attacker could use this flaw, e.g. this one.

References:

• Information on the System Change Number (SCN) and how it is used in the Oracle Database [ID 1376995.1]
• Installing, Executing and Interpreting output from the “scnhealthcheck.sql” script [ID 1393363.1]
• Bug 12371955 – Backup task can cause increased SCN growth rate leading to ORA-600 [2252] errors [ID 12371955.8]
  

I have uploaded the presentation material from my DOAG Conference presentation on “Minimal Downtime Oracle 11g Upgrade”. The material contains a paper, presentation and an online demo which shows how Logical Standby SQL Apply can be used for minimal downtime 10g to 11g Upgrade. Additionally, for convenience, i provide all 3 components in a ZIP package.

DOAG Conference 2010: Minimal Downtime Oracle 11g Upgrades

• Minimal Downtime Oracle 11g Upgrades - Paper (german)
• Minimal Downtime Oracle 11g Upgrades - Presentation (german)
• Minimal Downtime Oracle 11g Upgrades – Online Demo: Upgrade via SQL Apply
• Minimal Downtime Oracle 11g Upgrades - (Paper, Presentation, Demo)

I would like to draw your attention to a particularly dangerous security vulnerability, which was recently published by David Litchfield.

How dangerous is the vulnerability?

Any database user, who has “create session” privilege, which means, who can log into the database, can use the security hole to execute any OS command in the ownership of the oracle database owner. This means, that both denial of service as well as access to all data is exposed.

Which versions are affected?

Affected are database versions 10.2.0.4 (incl. 10.2.0.4.3 containing latest security patches as of January 2010) as well as 11g (incl. 11.2.0.1).

What can I do to close this security vulernability?

You can revoke privileges from PUBLIC:

revoke execute on dbms_java from PUBLIC;
revoke execute on dbms_java_test from PUBLIC;
revoke execute on “oracle/aurora/util/Wrapper” from PUBLIC;
grant execute on sys.dbms_jvm_exp_perms to IMP_FULL_DATABASE;
grant execute on sys.dbms_jvm_exp_perms to EXP_FULL_DATABASE;
revoke execute on sys.dbms_jvm_exp_perms from PUBLIC;

If you are using a third party vendor application, you should contact your vendor to check compatibility with revoked privileges or test before implementing in production.

I just came back from the DOAG Conference 2009, the german Oracle user group conference in Nürnberg where I had a speech about RAC PreProduction Testing. I have uploaded the slides and the paper to the papers section. At this time, the presentation is available in german only.

Oracle has released the October 2009 Patch Set Update (PSU) which contains several interesting news:

• first PSU, which is available for Grid Control 10.2.0.5
• seperate PSU Patches for Clusterware (CRS)
• Patch Set Updates are now released for ALL non-Windows platform whereas previous PSUs were released for special platforms (e.g. Linux Itanium) on request only.

More info can be found in Metalink Note 854428.1.

It is not very known that you can run into serious problems if you run Linux x86-32bit with a big amount of RAM installed, if using RHEL below 5. The official name for the issue is called “Low Memory Starvation”. The best solution is to use x86-64bit to be able to address the whole amount of RAM efficiently.

However, if that is not feasible, then make sure that you at least run the hugemem kernel if you use RHEL < 5. In RHEL5-32bit, the hugemem kernel is default. A quick demonstration about what can happen if you don´t use hugemem kernel is shown here: We realized that RMAN backup is taking more than 24 hours. Querying v$session, we find out that one session is in ACTION "STARTED", whereas the other sessions are FINISHED.

SQL> select program, module,action 
      from v$session 
      where username = 'SYS' and program like 'rman%'
/      

PROGRAM                    MODULE                       ACTION             
-------------------------- ---------------------------  -------------------
rman@ora-vm1 (TNS V1-V3)    backup full datafile        0000078 FINISHED129
rman@ora-vm1 (TNS V1-V3)    backup full datafile        0000272 STARTED16  
rman@ora-vm1 (TNS V1-V3)    backup full datafile        0000084 FINISHED129
rman@ora-vm1 (TNS V1-V3)    rman@ora-vm1 (TNS V1-V3)                       
rman@ora-vm1 (TNS V1-V3)    rman@ora-vm1 (TNS V1-V3)    0000004 FINISHED131
rman@ora-vm1 (TNS V1-V3)    backup full datafile        0000092 FINISHED129

Then we check the SID,serial# from v$session of this session and also query the UNIX PID from v$process.spid

SQL> select sid,serial# from v$session where event like 'RMAN%';

       SID    SERIAL#
---------- ----------
      4343       5837

We activate SQL Tracing for this session to determine its activity:

SQL> select spid from v$process where addr = 
   (select paddr from v$session where sid = 4343);

SPID
------------
1681

SQL> begin dbms_monitor.session_trace_enable(4343,5837,true,true);
  2  end;
  3  /

However, no trace file gets created. Then we start tracing system calls with strace:

ora-vm1:# strace -fp 1681
attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted

“Not permitted”? – Although I am connected as root?

ps -ef|grep 1681
oracle    1681 1582  0 Aug24 ?        00:00:09 [oracle] <defunct>

The linux command “ps” reports the server process as “defunct”.

ora-vm1:/usr/oracle/admin/labo/udump$ ps -ef|grep 1582
oracle   1582 21578  0 Aug24 ?        00:00:02 rman oracle/product/10.2.0/bin/rman nocatalog
oracle   21663 1582  0 Aug24 ?        00:00:01 oraclelabo (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   21665 1582  0 Aug24 ?        00:00:03 oraclelabo (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   1681 1582   0 Aug24 ?        00:00:09 [oracle] <defunct>
oracle   21691 1582  0 Aug24 ?        00:01:36 oraclelabo (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   21695 1582  0 Aug24 ?        00:01:41 oraclelabo (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   21793 1582  0 Aug24 ?        00:01:30 oraclelabo (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

Next, I checked logfile /var/log/messages.1 and realized that the kernel out-of-memory killer (OOM) killed this PID because of low memory starvation.

/var/log/messages.1:
Aug  24 22:32:44 ora-vm1 kernel: Out of Memory: Killed process 1681 (oracle).

I have recently encountered a problem at a customer site, where the database instances resource utilization was so high that the application did not work anymore. Version was 10.2.0.4 on Linux with Oracle Recommended Generic Patches installed.

Unfortunately, the customer decided to bounce the instance, so that there is no possibility for intensive diagnosis. However, ASH report shows that a dozen sessions either waited on Wait event “cursor: pin S” or were active (ON CPU) without any SQL_ID. An AWR Report showed:

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
cursor: pin S                   140,036,615      24,833      0 ######      Other
CPU time                                             19          72.9
log file sync                         8,767           8      1   30.2     Commit
log file parallel write               9,039           8      1   29.9 System I/O
control file parallel write           1,269           5      4   20.2 System I/O

Oracle Support confirmed that this is Bug 6904068 High CPU usage when there are “cursor: pin S” waits. We have filed a backport request for it, as there is no patch for our platform available yet.

I have come across a short post on Pete Finnigan´s Oracle Security Weblog, who informed about the release of new Metasploit modules usable for penetration testing of Oracle databases.

What is Metasploit?

Metasploit is a framework, which enables automatic utilization of all kinds of exploits to test security of a system. Among others, there is an Oracle module.

To get some idea about what is possible, watch this: Attacking Oracle with the Metasploit Framework Shmoocon Firetalk Demo Video. In a very impressive 5 minute video, the presenter demonstrates how to use Oracle Listener version identification, SID brute force, well known username/password combinations (e.g. scott/tiger), gets access to scott, privilege escalates to dba, plants a java class to exec os commands, etc… You get the idea….

This will be something to watch out for, because it will enable script-kiddies to attack badly secured databases connected to the internet, or well trained rogue internal employees to attack databases, which do not have critical patch updates for well known security vulnerabilities installed.

A reuters report about this new release can you find here.

Update 2009-08-13: The metasploit developer has uploaded new demo videos of how to hack an oracle database with metasploit.