I recently had to troubleshoot a hung instance in a 2 node RAC system. 4 months earlier, the system was reinstalled in a rolling fashion due to the requirement of Linux Upgrade from Oracle Linux 5 to Oracle Linux 7. This was required because of lack of certification for a storage migration to an AllFlash Storage. The system has been stable when running with Oracle Linux 5 for several years. Around 4 months after the reinstallation, one node got hung with and traces showed these error messages:<\/p>\n
\r\nMon Feb 26 08:53:37 2018\r\nskgxpvfynet: mtype: 61 process 15801 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)\r\nErrors in file \/u01\/app\/oracle\/diag_p\/diag\/rdbms\/prod\/PROD2\/trace\/PROD2_ora_15801.trc (incident=480004):\r\nORA-00603: ORACLE server session terminated by fatal error\r\nORA-27504: IPC error creating OSD context\r\nORA-27300: OS system dependent operation:sendmsg failed with status: 105\r\nORA-27301: OS failure message: No buffer space available\r\nORA-27302: failure occurred at: sskgxpsnd2\r\nIncident details in: \/u01\/app\/oracle\/diag_p\/diag\/rdbms\/prod\/PROD2\/incident\/incdir_480004\/PROD2_ora_15801_i480004.trc\r\n<\/pre>\nThis was strange because OS \/proc\/meminfo was showing huge amounts of free memory for this physical host with 512GB of RAM.<\/p>\n
\r\n[root@node17 ~]# cat \/proc\/meminfo \r\nzzz ***Mon Feb 26 08:53:09 CET 2018\r\nMemTotal: 528028424 kB\r\nMemFree: 14593828 kB\r\nMemAvailable: 78305772 kB\r\nBuffers: 28009752 kB\r\nCached: 46896496 kB\r\nSwapCached: 0 kB\r\nActive: 22627436 kB\r\nInactive: 66945168 kB\r\nActive(anon): 14315300 kB\r\nInactive(anon): 2105748 kB\r\nActive(file): 8312136 kB\r\nInactive(file): 64839420 kB\r\nUnevictable: 363996 kB\r\nMlocked: 364020 kB\r\nSwapTotal: 33554428 kB\r\nSwapFree: 33554428 kB\r\nDirty: 404 kB\r\nWriteback: 0 kB\r\nAnonPages: 15768480 kB\r\nMapped: 709896 kB\r\nShmem: 945384 kB\r\nSlab: 2462644 kB\r\nSReclaimable: 2092232 kB\r\nSUnreclaim: 370412 kB\r\nKernelStack: 28336 kB\r\nPageTables: 395568 kB\r\nNFS_Unstable: 0 kB\r\nBounce: 0 kB\r\nWritebackTmp: 0 kB\r\nCommitLimit: 87853440 kB\r\nCommitted_AS: 22087384 kB\r\nVmallocTotal: 34359738367 kB\r\nVmallocUsed: 1106260 kB\r\nVmallocChunk: 34085810172 kB\r\nHardwareCorrupted: 0 kB\r\nAnonHugePages: 0 kB\r\nCmaTotal: 16384 kB\r\nCmaFree: 0 kB\r\nHugePages_Total: 204800\r\nHugePages_Free: 20214\r\nHugePages_Rsvd: 6266\r\nHugePages_Surp: 0\r\nHugepagesize: 2048 kB\r\nDirectMap4k: 36827756 kB\r\nDirectMap2M: 78444544 kB\r\nDirectMap1G: 423624704 kB\r\n<\/pre>\nOracle Support then referred to this MOS Note:<\/p>\n
ORA-27301: OS Failure Message: No Buffer Space Available \/ ORA-27302: failure occurred at: sskgxpsnd2 ( Doc ID 2322410.1 ) <\/p><\/blockquote>\n
It turned out that on systems with a lot of physical memory and on Oracle Linux 7, the MTU size of loopback adapter lo0 has to be reduced from the default value of 64k to 16436 bytes to avoid memory fragmentation in RAC. The note also mentioned that the parameter vm.min_free_kbytes should be set to physmem * 0,004 * (here 2). This requirement was already considered during installation of the system, so the only thing missing was the MTU size.<\/p>\nI was very surprised that neither Cluster Verification Utility (CVU), nor orachk in most recent version did catch this problem at the point of installation. In my opinion, if default value of MTU size of loopback interface on Oracle Linux 7 has the potential to cause an outage, then this must be pre-checked by CVU at installation time or at least integrated into orachk. Unfortunately, this is not the case. It seems that in July we will know if the on-prem release 18.3.0 eventually will catch and enforce this configuration requirement during installation time.<\/p>\n","protected":false},"excerpt":{"rendered":"
I recently had to troubleshoot a hung instance in a 2 node RAC system. 4 months earlier, the system was reinstalled in a rolling fashion due to the requirement of Linux Upgrade from Oracle Linux 5 to Oracle Linux 7. This was required because of lack of certification for a storage migration to an AllFlash […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[57,44,5],"tags":[],"class_list":["post-1594","post","type-post","status-publish","format-standard","hentry","category-11gr2","category-linux-unix-2","category-oracle-database"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1594","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/comments?post=1594"}],"version-history":[{"count":9,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1594\/revisions"}],"predecessor-version":[{"id":1603,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1594\/revisions\/1603"}],"wp:attachment":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/media?parent=1594"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/categories?post=1594"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/tags?post=1594"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}