{"id":435,"date":"2008-11-10T21:21:29","date_gmt":"2008-11-10T19:21:29","guid":{"rendered":"http:\/\/www.ora-solutions.net\/web\/?p=435"},"modified":"2008-11-10T23:21:39","modified_gmt":"2008-11-10T21:21:39","slug":"listener-coredumps-on-heavy-load-system","status":"publish","type":"post","link":"https:\/\/www.ora-solutions.net\/web\/2008\/11\/10\/listener-coredumps-on-heavy-load-system\/","title":{"rendered":"Listener Coredumps on heavy load system"},"content":{"rendered":"

Recently I have come across a system which experiences listener crashes (core dumps) every couple of days. The listener logfile shows errors shortly before core-dumping:<\/p>\n

29-SEP-2008 03:49:07 * (CONNECT_DATA=(SID=MDDB1)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.1)(PORT
\n=1398)) * establish * MDDB1 * 12518
\nTNS-12518: TNS:listener could not hand off client connection
\nTNS-12571: TNS:packet writer failure
\nTNS-12560: TNS:protocol adapter error
\nTNS-00530: Protocol adapter error
\nLinux IA64 Error: 104: Connection reset by peer<\/code><\/p><\/blockquote>\n

After analysing the core dump with gdb, the stack points to malloc() calls, which mean that the listener requests memory from the OS.<\/p>\n

gdb \/oracle\/ora10\/bin\/tnslsnr core.311
\nGNU gdb Red Hat Linux (6.3.0.0-1.153.el4rh)
\nCopyright 2004 Free Software Foundation, Inc.
\nGDB is free software, covered by the GNU General Public License, and you are
\nwelcome to change it and\/or distribute copies of it under certain conditions.
\nType \"show copying\" to see the conditions.
\nThere is absolutely no warranty for GDB. Type \"show warranty\" for details.
\nThis GDB was configured as \"ia64-redhat-linux-gnu\"...(no debugging symbols found)
\nUsing host libthread_db library \"\/lib\/tls\/libthread_db.so.1\".<\/p>\n

Reading symbols from shared object read from target memory...(no debugging symbols found)...done.
\nLoaded system supplied DSO at 0xa000000000000000
\nCore was generated by `\/oracle\/ora10\/bin\/tnslsnr listenerv -inherit'.
\nProgram terminated with signal 11, Segmentation fault.<\/p>\n

#0 0x20000000027ee220 in malloc_consolidate ()
\nfrom \/lib\/tls\/libc.so.6.1
\n(gdb) bt
\n#0 0x20000000027ee220 in malloc_consolidate () from \/lib\/tls\/libc.so.6.1
\n#1 0x20000000027f0e30 in _int_malloc () from \/lib\/tls\/libc.so.6.1
\n#2 0x20000000027f4b50 in malloc () from \/lib\/tls\/libc.so.6.1
\n#3 0x40000000000079f0 in nsglconcrt ()
\n#4 0x4000000000011a00 in nsglhc ()
\n#5 0x4000000000019690 in nsglhe ()
\n#6 0x400000000001b980 in nsglma ()
\n#7 0x4000000000007770 in main ()
\n(gdb) quit<\/p>\n

<\/code><\/p><\/blockquote>\n

After contacting Oracle Support with this stack, they confirmed it to be Bug #6752308 which was closed as Duplicate of Bug 6139856. There is patch for 10.2.0.3 available and they also recommend to implement hugepages. By the way, there is an interesting article on the effect of utilizing – or not utilizing – hugepages here.<\/a><\/p>\n

6139856 - TT11.1VALGRIND: FMR (FREE MEMORY READ\/WRITE) IN NSEV.C<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"

Recently I have come across a system which experiences listener crashes (core dumps) every couple of days. The listener logfile shows errors shortly before core-dumping: 29-SEP-2008 03:49:07 * (CONNECT_DATA=(SID=MDDB1)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.1)(PORT =1398)) * establish * MDDB1 * 12518 TNS-12518: TNS:listener could not hand off client connection TNS-12571: TNS:packet writer failure TNS-12560: TNS:protocol adapter error TNS-00530: […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[13,49,5,12],"tags":[],"class_list":["post-435","post","type-post","status-publish","format-standard","hentry","category-10g","category-linux-itanium","category-oracle-database","category-unix"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/435","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/comments?post=435"}],"version-history":[{"count":8,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/435\/revisions"}],"predecessor-version":[{"id":494,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/435\/revisions\/494"}],"wp:attachment":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/media?parent=435"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/categories?post=435"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/tags?post=435"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}