After patching a customer environment from 19.28.0 to 19.29.0, it immediately became clear that there is something wrong. A couple of days after the patching, the diagnostic_dest Filesystem was filling up with gigabytes of trace files, containing “Dump of memory” output like this:<\/p>\n
\r\n*** 2025-11-13T20:56:09.238188+01:00 (DEMO(5))\r\n*** SESSION ID:(403.58882) 2025-11-13T20:56:09.238214+01:00\r\n*** CLIENT ID:() 2025-11-13T20:56:09.238217+01:00\r\n*** SERVICE NAME:(demo) 2025-11-13T20:56:09.238220+01:00\r\n*** MODULE NAME:(SQL*Plus) 2025-11-13T20:56:09.238224+01:00\r\n*** ACTION NAME:() 2025-11-13T20:56:09.238227+01:00\r\n*** CLIENT DRIVER:(SQL*PLUS) 2025-11-13T20:56:09.238229+01:00\r\n*** CONTAINER ID:(5) 2025-11-13T20:56:09.238233+01:00\r\n\r\nDump of memory from 0x0000000075F97860 to 0x0000000075F97960\r\n075F97860 00000000 00000000 00000000 00000000 [................]\r\n075F97870 0000000D 00000000 41524150 4554454D [........PARAMETE]\r\n075F97880 54435F52 00000058 00000000 00000000 [R_CTX...........]\r\n075F97890 00000000 00000000 00000000 00000000 [................]\r\n Repeat 6 times\r\n075F97900 00000009 00000000 49524156 454C4241 [........VARIABLE]\r\n075F97910 00000031 00000000 00000000 00000000 [1...............]\r\n075F97920 00000000 00000000 00000000 00000000 [................]\r\n Repeat 3 times\r\nDump of memory from 0x00000000678C8798 to 0x00000000678C87E8\r\n0678C8790 00656E6F 00000000 [one.....]\r\n0678C87A0 00000000 00000000 00000000 00000000 [................]\r\n Repeat 2 times\r\n0678C87D0 00000041 00B38F00 678C8780 00000000 [A..........g....]\r\n0678C87E0 154CEE98 00000000 [..L.....]\r\n<\/pre>\nThe string “_CTX” was pointing to application contexts which are heavily used by this application. Next step was to enable 10046 trace to get also information about SQL Statements that are causing this. So we found some application PL\/SQL calls that were manipulating global contexts, but first we were unable to manually reproduce the issue ourselves in SQL*Plus.<\/p>\n
On a DEV environment, I was able to use GDB (gnu debugger) to set a breakpoint to oracle database kernel function that is responsible for writing to diag trace files: ksdwrf. After the breakpoint is set, I execute “c” for continue and when it stops, I execute “bt” for backtrace which gives call stack when ksdwrf is called to print lines to trace file. This is then showing kernel functions kzctx… which again point to application contexts.<\/p>\n
gdb -p <PID of oracle server process that is streaming these messages to trace file><\/p>\n
Example:<\/p>\n
\r\n\r\nas root:\r\n\r\n gdb -p 244012\r\nGNU gdb (Red Hat Enterprise Linux) 14.2-4.1.0.1.el9_6\r\nCopyright (C) 2023 Free Software Foundation, Inc.\r\nLicense GPLv3+: GNU GPL version 3 or later \r\nThis is free software: you are free to change and redistribute it.\r\nThere is NO WARRANTY, to the extent permitted by law.\r\nType \"show copying\" and \"show warranty\" for details.\r\nThis GDB was configured as \"x86_64-redhat-linux-gnu\".\r\nType \"show configuration\" for configuration details.\r\nFor bug reporting instructions, please see:\r\n for more, q to quit, c to continue without paging--c\r\nMissing separate debuginfos, use: dnf debuginfo-install glibc-2.34-168.0.1.el9_6.23.x86_64 libaio-0.3.111-13.el9.x86_64 l ibnsl-2.34-168.0.1.el9_6.23.x86_64 numactl-libs-2.0.19-1.el9.x86_64\r\n\r\n\r\n(gdb) break ksdwrf\r\nBreakpoint 1 at 0xe1d570\r\n\r\n(gdb) c\r\nContinuing.\r\n\r\nBreakpoint 1, 0x0000000000e1d570 in ksdwrf ()\r\n(gdb) bt\r\n#0 0x0000000000e1d570 in ksdwrf ()\r\n#1 0x0000000000e1f022 in ksddmp2 ()\r\n#2 0x000000000c84b32a in kzctxgset ()\r\n#3 0x000000000c846e47 in kzctxgscv ()\r\n#4 0x000000000c828f49 in kzctxesc ()\r\n#5 0x0000000014449dc3 in pevm_icd_call_common ()\r\n#6 0x0000000014441190 in pfrinstr_ICAL ()\r\n#7 0x000000001443ffd4 in pfrrun_no_tool ()\r\n#8 0x000000001443e938 in pfrrun ()\r\n#9 0x000000001444a710 in plsql_run ()\r\n#10 0x0000000014438ec7 in peicnt ()\r\n#11 0x00000000053570ba in kkxexe ()\r\n#12 0x000000000376ca4d in opiexe ()\r\n#13 0x000000001412d56d in kpoal8 ()\r\n#14 0x0000000014104440 in opiodr ()\r\n#15 0x000000001439a577 in ttcpip ()\r\n#16 0x000000000374df15 in opitsk ()\r\n#17 0x0000000003753e27 in opiino ()\r\n#18 0x0000000014104440 in opiodr ()\r\n#19 0x0000000003749be6 in opidrv ()\r\n#20 0x0000000004872215 in sou2o ()\r\n#21 0x0000000000e0f2a6 in opimai_real ()\r\n#22 0x000000000487ea71 in ssthrdmain ()\r\n#23 0x0000000000e0f0d0 in main ()\r\n(gdb) c\r\nContinuing.\r\n\r\nBreakpoint 1, 0x0000000000e1d570 in ksdwrf ()\r\n(gdb) bt\r\n#0 0x0000000000e1d570 in ksdwrf ()\r\n#1 0x0000000000e1f022 in ksddmp2 ()\r\n#2 0x000000000c84e04b in kzctxglhu ()\r\n#3 0x000000000c84b4d4 in kzctxgset ()\r\n#4 0x000000000c846e47 in kzctxgscv ()\r\n#5 0x000000000c828f49 in kzctxesc ()\r\n#6 0x0000000014449dc3 in pevm_icd_call_common ()\r\n#7 0x0000000014441190 in pfrinstr_ICAL ()\r\n#8 0x000000001443ffd4 in pfrrun_no_tool ()\r\n#9 0x000000001443e938 in pfrrun ()\r\n#10 0x000000001444a710 in plsql_run ()\r\n#11 0x0000000014438ec7 in peicnt ()\r\n#12 0x00000000053570ba in kkxexe ()\r\n#13 0x000000000376ca4d in opiexe ()\r\n#14 0x000000001412d56d in kpoal8 ()\r\n#15 0x0000000014104440 in opiodr ()\r\n#16 0x000000001439a577 in ttcpip ()\r\n#17 0x000000000374df15 in opitsk ()\r\n#18 0x0000000003753e27 in opiino ()\r\n#19 0x0000000014104440 in opiodr ()\r\n#20 0x0000000003749be6 in opidrv ()\r\n#21 0x0000000004872215 in sou2o ()\r\n#22 0x0000000000e0f2a6 in opimai_real ()\r\n#23 0x000000000487ea71 in ssthrdmain ()\r\n#24 0x0000000000e0f0d0 in main ()\r\n\r\n\r\n(gdb) c\r\nContinuing.\r\n<\/pre>\nSo, now we had the call stack of the point in time when the write to the trace file was triggered. Maybe we can even get line numbers of these functions when executing “oradebug short_stack” while in breakpoint? So, after giving “c” (continue) and when the code again jumped to “Breakpoint 1, 0x0000000000e1d570 in ksdwrf ()”, in a new terminal, I executed<\/p>\n
\r\nsqlplus \/ as sysdba\r\nalter session set container = ...;\r\noradebug setospid 244012\r\noradebug short_stack\r\n\r\nsedsts()+426\r\n<-ksdxfstk()+63\r\n<-ksdxcb()+872\r\n<-sspuser()+243\r\n<-__sighandler()\r\n<-ksdwrf() <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< call to function to write to trace\r\n<-ksddmp2()+66 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< call to Dump function \r\n<-kzctxglhu()+1403 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< global context function \r\n<-kzctxgset()+6580\r\n<-kzctxgscv()+2263\r\n<-kzctxesc()+9561\r\n<-pevm_icd_call_common()+835\r\n<-pfrinstr_ICAL()+128\r\n<-pfrrun_no_tool()+52\r\n<-pfrrun()+904\r\n<-plsql_run()+752\r\n<-peicnt()+279\r\n<-kkxexe()+1178\r\n<-opiexe()+18973\r\n<-kpoal8()+2301\r\n<-opiodr()+1264\r\n<-ttcpip()+1255\r\n<-opitsk()+1925\r\n<-opiino()+935\r\n<-opiodr()+1264\r\n<-opidrv()+1094\r\n<-sou2o()+165\r\n<-opimai_real()+422\r\n<-ssthrdmain()+417\r\n<-main()+256\r\n<-__libc_start_call_main()+128 \r\n<\/pre>\nSo, obviously the Oracle developer of functions kzctx* introduced some debug output of memory regions into trace files and forgot to remove this debug code in 19.29.0. Unfortunately, even with exact line numbers, it is still difficult to convince the first level support engineers that this is a problem. In order to receive a bugfix, we need to provide a reproducible testcase. After a while, I was successful with the following code:<\/p>\n
\r\nCREATE OR REPLACE CONTEXT parameter_ctx USING context_api accessed globally;\r\n\r\nCREATE OR REPLACE PACKAGE context_api AS\r\n PROCEDURE set_parameter(p_name IN VARCHAR2,\r\n p_value IN VARCHAR2);\r\n PROCEDURE clear_context (p_name IN VARCHAR2);\r\n PROCEDURE clear_all_context;\r\nEND context_api;\r\n\/\r\n\r\nCREATE OR REPLACE PACKAGE BODY context_api IS\r\n PROCEDURE set_parameter (p_name IN VARCHAR2,\r\n p_value IN VARCHAR2) IS\r\n BEGIN\r\n sys.DBMS_SESSION.set_context('parameter_ctx', p_name, p_value);\r\n END set_parameter;\r\n\r\n PROCEDURE clear_context (p_name IN VARCHAR2) IS\r\n BEGIN\r\n sys.DBMS_SESSION.clear_context('parameter_ctx', attribute => p_name);\r\n END clear_context;\r\n\r\n PROCEDURE clear_all_context IS\r\n BEGIN\r\n sys.DBMS_SESSION.clear_all_context('parameter_ctx');\r\n END clear_all_context;\r\nEND context_api;\r\n\/\r\n\r\nEXEC context_api.set_parameter('variable1', 'one');\r\n<\/pre>\nBeginning in 19.29.0, the call to context_api.set_parameter, which in turn executes DBMS_SESSION.SET_CONTEXT, creates a trace file with these lines:<\/p>\n
\r\n*** 2025-11-13T20:56:09.238188+01:00 (DEMO(5))\r\n*** SESSION ID:(403.58882) 2025-11-13T20:56:09.238214+01:00\r\n*** CLIENT ID:() 2025-11-13T20:56:09.238217+01:00\r\n*** SERVICE NAME:(demo) 2025-11-13T20:56:09.238220+01:00\r\n*** MODULE NAME:(SQL*Plus) 2025-11-13T20:56:09.238224+01:00\r\n*** ACTION NAME:() 2025-11-13T20:56:09.238227+01:00\r\n*** CLIENT DRIVER:(SQL*PLUS) 2025-11-13T20:56:09.238229+01:00\r\n*** CONTAINER ID:(5) 2025-11-13T20:56:09.238233+01:00\r\n\r\nDump of memory from 0x0000000075F97860 to 0x0000000075F97960\r\n075F97860 00000000 00000000 00000000 00000000 [................]\r\n075F97870 0000000D 00000000 41524150 4554454D [........PARAMETE]\r\n075F97880 54435F52 00000058 00000000 00000000 [R_CTX...........]\r\n075F97890 00000000 00000000 00000000 00000000 [................]\r\n Repeat 6 times\r\n075F97900 00000009 00000000 49524156 454C4241 [........VARIABLE]\r\n075F97910 00000031 00000000 00000000 00000000 [1...............]\r\n075F97920 00000000 00000000 00000000 00000000 [................]\r\n Repeat 3 times\r\nDump of memory from 0x00000000678C8798 to 0x00000000678C87E8\r\n0678C8790 00656E6F 00000000 [one.....]\r\n0678C87A0 00000000 00000000 00000000 00000000 [................]\r\n Repeat 2 times\r\n0678C87D0 00000041 00B38F00 678C8780 00000000 [A..........g....]\r\n0678C87E0 154CEE98 00000000 [..L.....]\r\n*\/\r\n<\/pre>\nAnd if this code is executed millions of times, you end up with gigabytes of trace data in a couple of days. Now, I hope that Oracle Support will open a bug for this and that we get a fix soon. In the meantime, we have to delete tons of trace files every day.<\/p>\n
\nUpdate 17.11.2025:<\/b>
\nOracle has finally opened a bug for this:
\nBug 38658309 - HUGE TRACE FILES WITH \"DUMP OF MEMORY\" AFTER UPGRADING FROM 19.28.0 TO 19.29 ||APPLICATION CONTEXT<\/p>\n","protected":false},"excerpt":{"rendered":"
After patching a customer environment from 19.28.0 to 19.29.0, it immediately became clear that there is something wrong. A couple of days after the patching, the diagnostic_dest Filesystem was filling up with gigabytes of trace files, containing “Dump of memory” output like this: *** 2025-11-13T20:56:09.238188+01:00 (DEMO(5)) *** SESSION ID:(403.58882) 2025-11-13T20:56:09.238214+01:00 *** CLIENT ID:() 2025-11-13T20:56:09.238217+01:00 *** […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[72],"tags":[],"class_list":["post-1824","post","type-post","status-publish","format-standard","hentry","category-19c"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1824","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/comments?post=1824"}],"version-history":[{"count":7,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1824\/revisions"}],"predecessor-version":[{"id":1831,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/posts\/1824\/revisions\/1831"}],"wp:attachment":[{"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/media?parent=1824"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/categories?post=1824"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ora-solutions.net\/web\/wp-json\/wp\/v2\/tags?post=1824"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}