Linux Patching and Oracle: Detect RPM conflicts before they happen

by Jared Still on Nov 15, 2011 12:00:00 AM

A common scenario in the life of a DBA on a linux server looks something like this:

From: Sue-the-sysadamin To:the-dba Date: 2011-10-28 17:54:34 Dear DBA, We will be patching the linux systems this Friday night, October 28th at 21:00. The list of patches is attached. Let us know if there are any conflicts with Oracle. BTW, we need to know by 14:00 tomorrow. Sue

Yikes! The SysAdmin wants to apply these patches this week, and needs to know if there are any conflicts this afternoon. So you open up the list of patches. When you see the lengthy list of patches you suddenly realize the long lunch you had planned with your sweetheart is going to be rescheduled. I recently received several lists of patches that were to be applied to a number of different linux servers that were all running oracle, which led to this blog entry.

Matching Oracle shared lib dependencies to RPM files

Being faced with the task of determining which, if any, components of Oracle would possibly be affected by the list of RPMs to be upgraded across several servers with is a daunting task. One quick method to deal with this is to look for glibc in the list of RPMs. The list I received contained the following:

glibc-2.5-58.el5_6.4.i686
glibc-2.5-58.el5_6.4.x86_64
glibc-common-2.5-58.el5_6.4.x86_64
glibc-devel-2.5-58.el5_6.4.i386
glibc-devel-2.5-58.el5_6.4.x86_64
glibc-headers-2.5-58.el5_6.4.x86_64

Since nearly everything will be affected by an update to glibc, it would be a good idea to test these new RPMs out on the test server for that purpose before deploying them in productions. What if however you need a more exacting list of which Oracle components are affected by which packages? This information is a little harder to come by. While Oracle does supply a list of RPMs that must be installed on a Linux system, that information does not help you in determining which Oracle files are dependent on the RPMs in question. If the list of RPMs to be updated is small, it may even be that that few, if any of the Oracle files on a server may be affected. Here's how that can be determined. This may not be complete, but I think its a pretty good start.

Find the dependency files

Stepping through this manually, we can check an oracle executable file for the list of other files that is dependent on with ldd, which will display shared library dependencies.

[oracle@oravm01 patch_check]$ ldd $ORACLE_HOME/bin/oracle
  libodm11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libodm11.so (0x00002ab54b700000)
  libcell11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libcell11.so (0x00002ab54b802000)
  libskgxp11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxp11.so (0x00002ab54b958000)
  librt.so.1 => /lib64/librt.so.1 (0x0000003ed8c00000)
  libnnz11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so (0x00002ab54bb2c000)
  libclsra11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libclsra11.so (0x00002ab54bef4000)
  libdbcfg11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libdbcfg11.so (0x00002ab54c00d000)
  libhasgen11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libhasgen11.so (0x00002ab54c130000)
  libskgxn2.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxn2.so (0x00002ab54c66e000)
  libocr11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocr11.so (0x00002ab54c771000)
  libocrb11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrb11.so (0x00002ab54c90b000)
  libocrutl11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrutl11.so (0x00002ab54caa5000)
  libasmclnt11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libasmclnt11.so (0x00002ab54cbb1000)
  libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002ab54cd33000)
  libdl.so.2 => /lib64/libdl.so.2 (0x0000003ed8000000)
  libm.so.6 => /lib64/libm.so.6 (0x0000003ed7c00000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ed8400000)
  libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003edb400000)
  libc.so.6 => /lib64/libc.so.6 (0x0000003ed7800000)
  /lib64/ld-linux-x86-64.so.2 (0x0000003ed7400000)

Here we can see that the oracle binary is dependent on several library files. Many of them are owned by oracle, but some of them are files that are part of the Linux system.

Find the RPM

The next step is find which RPM the files are part of. We can use the rpm command for this. The -q option is for 'query', while the '-f' causes rpm to report which package a file is associated with. Choosing one of the linux owned files as an example:

[oracle@oravm01 patch_check]$ rpm -qf /lib64/libm.so.6
 glibc-2.5-49

What happens if rpm -qf is run against one of the oracle owned dependency shared library files?

[oracle@oravm01 patch_check]$ rpm -qf /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so
 file /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so is not owned by any package

As you might expect, rpm has no knowledge of oracle owned files, and reports that fact.

Does the patch affect Oracle?

The final step is to compare the current package with the patch list. In this case there is a match - glibc-2.5-49 is to be replaced with glibc-2.5.58. Now all that you need to do is repeat this process for all oracle binary files. This obviously does not work well as a manual process. Fortunately this is not too difficult a task to automate. Those readers already familiar with me will not be surprised to learn that this automation involves the use of Perl.

The Patch Check Script

Or rather, scripts. The Perl script pc.pl is complemented by the shell script pc.sh. The reason for using a driver shell script is that there may be more than one Oracle home directory on the server that you need to check. One of the few limitations of using Perl with Oracle involves the LD_LIBRARY_PATH variable. This variable cannot be set from within a Perl script, as the Perl compiler loads in the value of LD_LIBRARY_PATH before running your script, so subsequent attempts to change it are futile. There are some hacks that will work on some platforms that allow LD_LIBRARY_PATH to be changed at runtime, but I would rather not go that route. The limitations of using LD_LIBARY_PATH are pretty well known, so it is best just to work around that particular issue.

The RPM patch list

The list of RPM should bin a file, one RPM per line. The file names should not contain the .rpm extension. If the extension is included, you will need to remove it. Either that, or you will need to change the script to work with the .rpm extension. Why does the list of RPMs not include the .rpm extension? Simple, that's how I received the list. Here is a sample of files from a list of RPMs to be updated:

[oracle@oravm01 patch_check]$ head -5 patch_list.txt
 amtu-1.0.6-2.el5.x86_64
 audit-1.7.18-2.el5.x86_64
 audit-libs-1.7.18-2.el5.i386
 audit-libs-1.7.18-2.el5.x86_64
 audit-libs-python-1.7.18-2.el5.x86_64

Which Files are searched?

Not all files in an Oracle home are processed. The find command is used with the -perm /u+x option to find all files that have the executable bit set for the file owner. These files are then processed to determine RPM dependencies.

The pc.pl Perl script

Let's invoke pc.pl to see what it does when run against a single Oracle home with the standard setting for LD_LIBRARY_PATH and other oracle environment variables. Here's the current environment:

[oracle@oravm01 patch_check]$ set | grep -E 'ORA|LD_LIB'
 LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/vmdb01/lib
 ORACLE_BASE=/u01/app/oracle
 ORACLE_HOME=/u01/app/oracle/product/11.2.0/vmdb01
 ORACLE_HOSTNAME=oraexp01.jks.com
 ORACLE_SID=vmdb01
 ORACLE_TERM=xterm
 ORAENV_ASK=NO
 ORAHOME=/u01/app/oracle/product/11.2.0/vmdb01
 ORA_NLS33=/u01/app/oracle/product/11.2.0/vmdb01/ocommon/nls/admin/data

Now run the script. Notice that STDERR has been redirected to a file - more on that later

[oracle@oravm01 patch_check]$ ./pc.pl -verbosity 0 -oracle_home $ORACLE_HOME -linux_patch_list patch_list.txt 2> pc.err
 
 #### Possible Conflicts Found #####
 --------------------
 --- gdbm-1.8.0-26.2.1.el5_6.1.x86_64
 old: 1.8.020 26.220
 new: 1.8.020 26.2.1.el5_6.120
 --------------------
 --- glibc-2.5-58.el5_6.4.x86_64
 old: 2.520 4920
 new: 2.520 58.el5_6.420
 --------------------
 --- libgcc-4.1.2-50.el5.x86_64
 old: 4.1.220 4820
 new: 4.1.220 50.el520
 --------------------
 --- libstdc++-4.1.2-50.el5.x86_64
 old: 4.1.220 4820
 new: 4.1.220 50.el520

pc.pl has found 4 RPM packages that are scheduled to be updated and that Oracle is dependent on. This new information may cause the patch schedule to be pushed back until it is determined that these updates will have no adverse affects on Oracle. Now let's consider the file pc.err, where all error messages were redirected to. We see there 336 lines in the error file.

[oracle@oravm01 patch_check]$ wc -l pc.err
 336 pc.err

The majority of the errors are from rpm -qf - as seen earlier the RPM database knows nothing of files owned by Oracle. As such, these are errors we are not really concerned with. There may however be errors hiding in the file that we are concerned about.

[oracle@oravm01 patch_check]$ head -7 pc.err
 ###########################
 working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies
 libawt_g.so => not found
 ###########################
 working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies
 libjava_g.so => not found
 ###########################

Following is another error - this one is not an rpm error, so it may be something to be concerned about.

##### ldd error ############
 error encountered with /usr/bin/ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so
 run the command manually to see the error
 ###########################

Running the command manually results in a segmentation fault.

[oracle@oravm01 patch_check]$ ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so
 /usr/bin/ldd: line 116: 5216 Segmentation fault LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"

A quick google search on 'ldd segmentation fault' suggests this is a linux bug. We will not be investigating this further, as we aren't too concerned with that at this time. With the large number of errors that we do not really care about being reported by the rpm -qf command, how can we reduce the number of errors?

The pc.sh shell script

The pc.sh script serves two purposes. It reads the file oracle_homes.txt and runs pc.pl for each ORACLE_HOME. The pc.sh script also sets the LD_LIBARY_PATH variable to eliminate many of the false positives seen in pc.err previously. Following is the entire pc.sh script. You may want to adjust the values used for LD_LIBARY_PATH to suit your environment. #!/bin/sh # set the environment to call pc.pl. (patch check) # the LD_LIBRARY_PATH OHOMES_FILE=oracle_homes.txt PATCH_LIST_FILE=patch_list.txt [ -r "$OHOMES_FILE" ] || { echo The file $OHOMES_FILE is missing exit 1 } while read ohome do sid=$(basename $ohome) LOGFILE=patch_check_${sid}.log ERRFILE=patch_check_${sid}.err echo $ohome

# you may have to experiment a bit to get the needed paths in LD_LIBRARY_PATH # in 11g there will probably still be a few files that need to be checked manually export LD_LIBRARY_PATH=$ohome/lib:$ohome/jdk/jre/lib/amd64:$ohome/jdk/jre/lib/amd64/server

./pc.pl -verbosity 0 -oracle_home $ohome -linux_patch_list $PATCH_LIST_FILE >$LOGFILE 2>$ERRFILE done < $OHOMES_FILE Lets run the script and then check the error logs. [oracle@oravm01 patch_check]$ ./pc.sh /u01/app/oracle/product/11.2.0/grid /u01/app/oracle/product/11.2.0/vmdb01 Checking on the output files shows that the error logs are much smaller than what was seen previously. [oracle@oravm01 patch_check]$ ls -ltar *.log *.err -rw-r--r-- 1 oracle oinstall 455 Nov 14 13:18 patch_check_grid.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_grid.log -rw-r--r-- 1 oracle oinstall 296 Nov 14 13:18 patch_check_vmdb01.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_vmdb01.log [oracle@oravm01 patch_check]$ wc -l patch*.err 9 patch_check_grid.err 6 patch_check_vmdb01.err 15 total By including some more library directories in LD_LIBRARY_PATH, rpm -qf was able to resolve many more dependencies, greatly reducing the false positives that were included in the error files. Here are the total contents of the error files:

[oracle@oravm01 patch_check]$ cat *.err Use of uninitialized value in hash element at ./pc.pl line 393. Use of uninitialized value in hash element at ./pc.pl line 393. ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found rpmq: no arguments given for query ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found

The 'uninitialized value' errors are from Perl - this is likely a bug in the script. Looking at the script I can see that while it is a script in the bug, it is not causing any incorrect reporting.

[oracle@oravm01 patch_check]$ grep -n . pc.pl | grep -3 393
 389: ($dummy1,$dummy2,$file) = split(/\s+/,$line);
 390: }
 392: #print " ###### FILE: $file\n";
 393: push @{$self->{olibDeps}->{$file}{dependents}}, $exe;
 394: }
 395: }
 397: # all library dependencies now in %olibDepend

Line 393 should be executed only if there is a value defined for $file. Changing it as follows this fixed the bug:

push @{$self->{olibDeps}->{$file}{dependents}}, $exe if $file;

The small number of lines now found in the .err files makes it simple to determine if these errors are an issue to be concerned with.

Download the scripts

The zip file containing all scripts can be downloaded here - Patch Check Repo on Github

Topics: Linux Technical Track Sql Server On Linux Sql On Linux

No Comments Yet

Let us know what you think

Share this