How to Prevent Linux 7 Housekeeping from Breaking Oracle

by Sean Scott on Aug 8, 2019 12:00:00 AM

Linux 7 introduced multiple changes and enhancements to the kernel. Among them was automated housekeeping for temporary files. In version 6, tmpwatch is available to manage unused files in temp directories. It can be scheduled via cron to automatically keep temp files in check. As of version 7, this functionality moves to systemd-tmpfiles and became a background service largely invisible to users and admins. Like tmpwatch, systemd-tmpfiles removes files and directories based on their access, modified, or created times. The default is to delete files and directories in /tmp that haven't been accessed for 10 days, and the ones in /var/tmp that haven't been accessed for 30 days. That probably seems like a good idea. In many cases, it is. However, it turns out to be a very bad thing for an Oracle server. Oracle uses the hidden /var/tmp/.oracle directory to store socket files. These are a special type of file that enable interprocess connections (IPC) on a system. They're created by Oracle when a connection is established via the Listener and used by ASM in addition to CRS, CSS, and EVM daemons. There are multiple Oracle Support documents that describe the consequences of deleting these files, including 370605.1, 391790.1, 1322234.1, 2099377.1, and 2492508.1. The symptoms are all... well, bad! These include the inability to connect to or start clusterware or a database, ORA-600/ORA-7445 errors, GI crashes, and SCAN failures. Resolutions nearly always requires downtime for the database or cluster. I just encountered a situation where temp file deletion by the systemd-tmpfiles resulted in excessive virtual memory consumption and CPU load as Oracle ASM processes attempted to read deleted socket files. In this example, I was able to solve things without a restart, but the host did come close to crashing from resource starvation. We were first alerted to growing virtual memory use on the system. By looking at the current and previous day's swapping with sar -S, I saw the system normally used 10-12% of its swap space. Over the course of a few hours, that jumped to over 50%. I reviewed the processes using the most swap:

# List info for processes using more than 10k swap space:
 for p in $(grep "VmSwap" /proc/*/status | grep -vi " 0 kb" | cut -d: -f1,3 | awk '$2 >= 10000' | sort -n -k 2 | cut -d\/ -f3)
 do
 egrep "^Name:|^State:|^Pid:|^Vm|switches:" /proc/$p/status
 done

This showed the four biggest consumers were ASMCMD daemon processes. Each was consuming over 20G of swap:

Name: asmcmd daemon
 State: S (sleeping)
 Pid: 7533
 VmPeak: 23526424 kB
 VmSize: 23526424 kB
 VmLck: 0 kB
 VmPin: 0 kB
 VmHWM: 21691104 kB
 VmRSS: 20972500 kB
 VmData: 23254580 kB
 VmStk: 140 kB
 VmExe: 1912 kB
 VmLib: 78352 kB
 VmPTE: 45168 kB
 VmSwap: 2014504 kB
 voluntary_ctxt_switches: 6557964
 nonvoluntary_ctxt_switches: 63522372

Running strace on these processes showed Failed to connect to ASM instance as well as a number of invalid calls to socket files that didn't exist. The associated processes were also showing up in the ASMCMD log files. Messages for Invalid file handle for pipe /tmp/asmcmd_fg_7533 were appearing several times a second. Further analysis showed these processes were systematically grabbing additional swap from the system, with accompanying CPU increases. The database eventually began reporting ORA-700 for "[kskvmstatact: excessive swapping observed]". In this situation, these were foreground sessions and were killed without any impact to the stability or availability of any Oracle components. The memory captured by the processes was immediately returned and load dropped to normal levels. There was no indication any user had removed files and the directory itself was still there, but all atime/ctime/mtimes on the files were under the 10-day threshold. While not an indictment of systemd-tmpfiles directly, the system is running Linux 7 and the service is active, which are good indicators that it "helpfully" removed the socket files. Here's another example of what can happen when files in /var/tmp/.oracle are deleted. Fortunately, there's an easy fix as described in Document 2498572.1. Simply tell systemd-tmpfiles to ignore the hidden .oracle directories by adding the following to the systemd-tmpfiles configuration file located at /usr/lib/tmpfiles.d/tmp.conf:

x /tmp/.oracle*
 x /var/tmp/.oracle*
 x /usr/tmp/.oracle*

Then, restart the service with systemctl restart systemd-tmpfiles-clean.timer. Note that on Exadata, /var/tmp is linked to /tmp which means the 10-day access threshold on /tmp is enforced. This is a good thing to remember when installing or working with Oracle on RHEL/CentOS/OEL 7 systems.

How to Prevent Linux 7 Housekeeping from Breaking Oracle

No Comments Yet

Subscribe by email

Share this

How to Prevent Linux 7 Housekeeping from Breaking Oracle

Share this

No Comments Yet

Subscribe by email