{"id":958,"date":"2010-11-11T10:03:11","date_gmt":"2010-11-11T08:03:11","guid":{"rendered":"http:\/\/www.ora-solutions.net\/web\/?p=958"},"modified":"2010-11-26T16:58:11","modified_gmt":"2010-11-26T14:58:11","slug":"grid-control-11g-diagnosing-oms-high-cpu-utilization","status":"publish","type":"post","link":"https:\/\/www.ora-solutions.net\/web\/2010\/11\/11\/grid-control-11g-diagnosing-oms-high-cpu-utilization\/","title":{"rendered":"Grid Control 11g: Diagnosing OMS High CPU Utilization"},"content":{"rendered":"

Beginning with 11g Oracle Enterprise Manager Grid Control is using Weblogic instead of Oracle Application Server. I am currently experiencing very sluggish Grid Control Console performance and high CPU Utilization of EMGC_OMS1 weblogic server. This article shows how you can diagnose the problem and “assist” Oracle support in helping to solve the issue.<\/p>\n

First step to find out what\u00b4s going on is to use “top”.<\/p>\n

top - 08:29:12 up 4 days, 20:56,  3 users,  load average: 1.47, 1.60, 1.61\r\nTasks: 532 total,   1 running, 531 sleeping,   0 stopped,   0 zombie\r\nCpu(s):  1.3%us,  0.7%sy,  0.0%ni, 97.8%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st\r\nMem:  132033652k total, 20255244k used, 111778408k free,   728860k buffers\r\nSwap: 20971512k total,        0k used, 20971512k free, 15391704k cached\r\n\r\n  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\r\n30181 gc11g    15   0 1778m 1.1g  49m S 99.7  0.9 626:54.30 java\r\n<\/pre>\n
PID 30181 is using 100% (and has been using it for a very long time)<\/p>\n
It\u00b4s the EMGC_OMS1 server:<\/p>\n
\r\ngc11g   30181 30129  9 Nov06 ?        10:27:20 \/oracle\/jdk\/jdk1.6.0_21\/bin\/java -client -Xms256m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=512m -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=\/oracle\/grid\/middleware\/wlserver_10.3\/server\/lib\/weblogic.policy -Dweblogic.system.BootIdentityFile=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/servers\/EMGC_OMS1\/data\/nodemanager\/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DINSTANCE_HOME=\/oracle\/grid\/gc_inst\/em\/EMGC_OMS1 -DORACLE_HOME=\/oracle\/grid\/middleware\/oms11g -Ddomain.home=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Docm.repeater.home=\/oracle\/grid\/middleware\/oms11g -Djava.security.egd=file:\/\/\/dev\/urandom -Xverify:none -da -Dplatform.home=\/oracle\/grid\/middleware\/wlserver_10.3 -Dwls.home=\/oracle\/grid\/middleware\/wlserver_10.3\/server -Dweblogic.home=\/oracle\/grid\/middleware\/wlserver_10.3\/server -Ddomain.home=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain -Dcommon.components.home=\/oracle\/grid\/middleware\/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=\/oracle\/grid\/middleware\/oracle_common\/modules\/oracle.jrf_11.1.1\/jrocket_optfile.txt -Doracle.domain.config.dir=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/config\/fmwconfig -Doracle.server.config.dir=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/config\/fmwconfig\/servers\/EMGC_OMS1 -Doracle.security.jps.config=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/config\/fmwconfig\/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/config\/fmwconfig\/carml -Digf.arisidstack.home=\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/config\/fmwconfig\/arisidprovider -Dweblogic.alternateTypesDirectory=\/oracle\/grid\/middleware\/oracle_common\/modules\/oracle.ossoiap_11.1.1,\/oracle\/grid\/middleware\/oracle_common\/modules\/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=false -Dweblogic.management.server=https:\/\/t44grid.wwk-group.com:7101 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=\/oracle\/grid\/middleware\/patch_wls1032\/profiles\/default\/sysext_manifest_classpath weblogic.Server\r\noel5n1:\/home\/gc11g >\r\n<\/pre>\n
Next step is to find out which thread is consuming 100% of cpu.<\/p>\n
oel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30181   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:03  0.0 java            30182   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30183   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30184   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30185  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30186  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30187   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30188  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30189   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30190   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30191  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30192   1\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30193   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30194  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            30195   5\r\n30181 gc11g   1821640 1167200 R 25 0 08:30:49  7.6 java            30196   2       <<<<<<\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30197  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:04  0.0 java            30198   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30199   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:58  0.0 java            30200  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:57  0.0 java            30201   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30202   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30203  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30204  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30496   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:18:50  0.2 java            30497  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30498  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30499   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30504   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30505  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:11  0.0 java            30510  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:38  0.0 java            30513  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:38  0.0 java            30514  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:37  0.0 java            30517  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:39  0.0 java            30518  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30529   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30533   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30588   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:01  0.0 java            30589   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30590  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30591   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30592   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30899   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30901   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30904   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30907  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            30910  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:02  0.0 java            31205   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:29:45  0.4 java            31526   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            31568  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            31593   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:12:02  0.1 java            31600  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:08:27  0.1 java            31601  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32157   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:01  0.0 java            32160   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32161  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32163   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32166  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32168   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32169  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32170  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32171  10\r\n30181 gc11g   1821640 1167200 S 16 0 00:00:00  0.0 java            32172   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32173  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32174   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32175  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32176  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32177   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32178   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32179  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32180   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:01  0.0 java            32181   6\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32182   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32183   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32264  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:02  0.0 java            32265   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32269   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:12  0.0 java            32270   1\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:14  0.0 java            32271   6\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32275  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:15  0.0 java            32280  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32283  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:10:45  0.1 java            32286  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:47  0.0 java            32289  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:02:57  0.0 java            32295   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:03  0.0 java            32297  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32301   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32302  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32305   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32306  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:09  0.0 java            32307   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32308   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32309   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32310   4\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32311   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32326  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:14  0.0 java            32327  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32336  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:01  0.0 java            32337   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:29  0.0 java            32338   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:05  0.0 java            32339   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32340  10\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:18  0.0 java            32341  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:02  0.0 java            32342   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:29  0.0 java            32343   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:10  0.0 java            32344   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:05  0.0 java            32351   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:02  0.0 java             1943   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:01:19  0.0 java             1944   6\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java             1945  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:06  0.0 java             1946  11\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java             2890  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:02:56  0.0 java             7839  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            10208   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:02  0.0 java            16818  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            18310   2\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:09  0.0 java              642  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java              690   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            27517   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            27518   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:01  0.0 java            29228   3\r\n30181 gc11g   1821640 1167200 S 15 0 00:02:46  0.1 java            12075  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            12160   1\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            12829  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:02  0.0 java             2773   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14748  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14749   6\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14750   1\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14751   0\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14752   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14753   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            14860  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            16706   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            17312   1\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            17313  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            17425  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            17426  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            19993  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            19994  15\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            19995  12\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java             3575   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            16593  14\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            18488   8\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            18489   5\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            21901   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java             5572   7\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java             7993   9\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            27543  13\r\n30181 gc11g   1821640 1167200 S 15 0 00:00:00  0.0 java            32075  15\r\n<\/pre>\n
So, it is thread 30196 of PID 30181. Let\u00b4s remove the \" sleeping \" threads with grep.<\/p>\n
\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 25 0 08:31:37  7.7 java            30196   6\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 25 0 08:31:39  7.7 java            30196   2\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 25 0 08:31:40  7.7 java            30196   2\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 18 0 08:31:40  7.7 java            30196   2\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 18 0 08:31:41  7.7 java            30196  15\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 19 0 08:31:41  7.7 java            30196  15\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 20 0 08:31:42  7.7 java            30196  13\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 20 0 08:31:42  7.7 java            30196   9\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 21 0 08:31:42  7.7 java            30196   9\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 21 0 08:31:43  7.7 java            30196   9\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 23 0 08:31:43  7.7 java            30196   9\r\noel5n1:\/home\/gc11g >  ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr | grep 30181 | grep -v \" S \"\r\n30181 gc11g   1821640 1167200 R 24 0 08:31:43  7.7 java            30196   9\r\n<\/pre>\n
It is always the same thread which is using 7.7% (~1\/16 cores) of the all the CPU\u00b4s on the system. <\/p>\n
let\u00b4s take a pstack:<\/p>\n
\r\noel5n1:\/home\/gc11g > pstack 30181 \r\n<\/pre>\n
let\u00b4s grep for the LWP we know:<\/p>\n
\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\n\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):                           <<<<<<<<<<<<<<<<<<\r\n#0  0x00002b7471e17d3f in instanceKlass::oop_follow_contents ()\r\n#1  0x00002b747203e6c3 in objArrayKlass::oop_follow_contents ()\r\n#2  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#3  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#4  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471e17d3f in instanceKlass::oop_follow_contents ()\r\n#1  0x00002b747203e6c3 in objArrayKlass::oop_follow_contents ()\r\n#2  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#3  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#4  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7472046d27 in objArrayKlass::oop_adjust_pointers ()\r\n#1  0x00002b747209549e in PSMarkSweepDecorator::adjust_pointers ()\r\n#2  0x00002b74720967f0 in PSOldGen::adjust_pointers ()\r\n#3  0x00002b7472094c3a in PSMarkSweep::mark_sweep_phase3 ()\r\n#4  0x00002b74720938f6 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471fee9d3 in MarkSweep::IsAliveClosure::do_object_b ()\r\n#1  0x00002b7471dfad18 in Hashtable::unlink ()\r\n#2  0x00002b74720949ef in PSMarkSweep::mark_sweep_phase1 ()\r\n#3  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#4  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#5  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#6  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#7  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#8  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#9  0x00002b747219c583 in VMThread::loop ()\r\n#10 0x00002b747219c08e in VMThread::run ()\r\n#11 0x00002b74720560df in java_start ()\r\n#12 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#13 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471fee706 in MarkSweep::AdjustPointerClosure::do_oop ()\r\n#1  0x00002b7471dfadb1 in Hashtable::oops_do ()\r\n#2  0x00002b7472094bf9 in PSMarkSweep::mark_sweep_phase3 ()\r\n#3  0x00002b74720938f6 in PSMarkSweep::invoke_no_policy ()\r\n#4  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#5  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#6  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#7  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#8  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#9  0x00002b747219c583 in VMThread::loop ()\r\n#10 0x00002b747219c08e in VMThread::run ()\r\n#11 0x00002b74720560df in java_start ()\r\n#12 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#13 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471e179e5 in instanceKlass::adjust_static_fields ()\r\n#1  0x00002b7471e234ad in instanceKlassKlass::oop_adjust_pointers ()\r\n#2  0x00002b747209549e in PSMarkSweepDecorator::adjust_pointers ()\r\n#3  0x00002b74720967f0 in PSOldGen::adjust_pointers ()\r\n#4  0x00002b7472094c3a in PSMarkSweep::mark_sweep_phase3 ()\r\n#5  0x00002b74720938f6 in PSMarkSweep::invoke_no_policy ()\r\n#6  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#7  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#8  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#9  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#10 0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#11 0x00002b747219c583 in VMThread::loop ()\r\n#12 0x00002b747219c08e in VMThread::run ()\r\n#13 0x00002b74720560df in java_start ()\r\n#14 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#15 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471e17e54 in instanceKlass::oop_follow_contents ()\r\n#1  0x00002b747203e6c3 in objArrayKlass::oop_follow_contents ()\r\n#2  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#3  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#4  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b747203e640 in objArrayKlass::oop_follow_contents ()\r\n#1  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#2  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#3  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#4  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#5  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#6  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#7  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#8  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#9  0x00002b747219c583 in VMThread::loop ()\r\n#10 0x00002b747219c08e in VMThread::run ()\r\n#11 0x00002b74720560df in java_start ()\r\n#12 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#13 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\n\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471d50445 in ConstantPoolCacheEntry::follow_contents ()\r\n#1  0x00002b7471d4f4f6 in constantPoolCacheKlass::oop_follow_contents ()\r\n#2  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#3  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#4  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n\r\noel5n1:\/home\/gc11g > pstack 30181 | grep -A 20 30196\r\nThread 135 (Thread 0x4024f940 (LWP 30196)):\r\n#0  0x00002b7471f8af28 in klassKlass::oop_follow_contents ()\r\n#1  0x00002b7471e226cc in instanceKlassKlass::oop_follow_contents ()\r\n#2  0x00002b7471fee5e3 in MarkSweep::follow_stack ()\r\n#3  0x00002b7472094964 in PSMarkSweep::mark_sweep_phase1 ()\r\n#4  0x00002b7472093859 in PSMarkSweep::invoke_no_policy ()\r\n#5  0x00002b74720a0737 in PSScavenge::invoke ()\r\n#6  0x00002b747206814e in ParallelScavengeHeap::failed_mem_allocate ()\r\n#7  0x00002b747218fbc9 in VM_ParallelGCFailedAllocation::doit ()\r\n#8  0x00002b747219ccfa in VM_Operation::evaluate ()\r\n#9  0x00002b747219c312 in VMThread::evaluate_operation ()\r\n#10 0x00002b747219c583 in VMThread::loop ()\r\n#11 0x00002b747219c08e in VMThread::run ()\r\n#12 0x00002b74720560df in java_start ()\r\n#13 0x0000003130e064a7 in start_thread () from \/lib64\/libpthread.so.0\r\n#14 0x00000031302d3c2d in clone () from \/lib64\/libc.so.6\r\n<\/pre>\n
I\u00b4m not much of a Java guy, but this tells me that the thread is doing ParallelGC. So, the issue might be related to a configuration of too small Heap size. (Xmx)  I will talk with support to decide whether they support to have Xmx increased to 1G or 2G. In case the cpu consumption is not for Garbage Collection but some other OEM code, then there is a command \"emctl dump omsthread\", which can be used:<\/p>\n
\r\noel5n1:\/home\/gc11g > emctl dump omsthread\r\nOracle Enterprise Manager 11g Release 1 Grid Control\r\nCopyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.\r\nThread dumped successfully for process '30181' to log file \r\n\/oracle\/grid\/gc_inst\/user_projects\/domains\/GCDomain\/servers\/EMGC_OMS1\/logs\/EMGC_OMS1.out\r\n<\/pre>\n
If we calculate LWP 30196 from dec to hex, we get 75f5, this is NID from OMS thread dump.<\/p>\n
\r\n\"Reference Handler\" daemon prio=10 tid=0x000000004fdff000 nid=0x75f5 waiting for monitor entry [0x0000000040725000]\r\n   java.lang.Thread.State: BLOCKED (on object monitor)\r\n        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:108)\r\n        - waiting to lock <0x00002aaacf4967a0> (a java.lang.ref.Reference$Lock)\r\n<\/pre>\n
Update 26.11.2010:<\/strong><\/p>\n
Oracle support has recommended to increase Xms\/Xmx  to 1024m and check whether that is sufficient. This has to be changed in file gc_inst\/user_projects\/domains\/GCDomain\/bin\/setDomainEnv.sh and in those variables for Linux 64bit:<\/p>\n
\n
XMS_SUN_64BIT=\"1024\"\r\nXMX_SUN_64BIT=\"1024\"<\/pre>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"
Beginning with 11g Oracle Enterprise Manager Grid Control is using Weblogic instead of Oracle Application Server. I am currently experiencing very sluggish Grid Control Console performance and high CPU Utilization of EMGC_OMS1 weblogic server. This article shows how you can diagnose the problem and “assist” Oracle support in helping to solve the issue. First step […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[44,6],"tags":[],"class_list":["post-958","post","type-post","status-publish","format-standard","hentry","category-linux-unix-2","category-oracle-enterprise-manager"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/958","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/comments?post=958"}],"version-history":[{"count":7,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/958\/revisions"}],"predecessor-version":[{"id":963,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/958\/revisions\/963"}],"wp:attachment":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/media?parent=958"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/categories?post=958"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/tags?post=958"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}