Patching fun: GI and database patches

by Jared Still on Sep 9, 2015 12:00:00 AM

So, it has been some time since I have patched a database. Patching seemed necessary in a 12c database used for testing, as it was having some crashing issues. As the database and GI home were both un-patched, it seemed worthwhile to just apply the latest set of patches rather than spend (more) time troubleshooting the issue. This is a 12.1.0.2 CDB database on Oracle Linux 6.5, 64 bit. This is a standalone database with Oracle Restart and no RAC. After downloading the appropriate patches, I unzipped the files to a local directory, installed OPatch and then started in on the patch documentation. After perusing the docs it appeared that patch would be fairly easy with this command: [code lang="bash"]# opatchauto apply <UNZIPPED_PATCH_LOCATION>/20996835 -ocmrf <ocm response file>[/code] One problem though. What is an OCM response file? As this was my first time using Opatchauto, I wanted to make this work. The documentation did include a reference to the following helpful Support Note: How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1) Creating the response file was quite straightforward, since the Oracle user the following command took care of it: [code lang="bash"]cd /u01/app/oracle/patches/20996835 $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output ./unconfig.rsp[/code] With that out of the way it was now time to try the patch. This was done as root, as in instructed in the patch documentation. [code lang="bash"]/u01/app/grid/product/12.1.0/grid/OPatch/opatchauto apply /u01/app/oracle/patches/20996835 -ocmrf /u01/app/oracle/patches/20996835/unconfig.rsp[/code] All went well for a few moments, but then (gasp) a failure. Please excuse the sarcasm; I may not have patched a database recently, yet I'm hardly new to the process. Patching working properly on the first attempt would have been quite a surprise. Here is what appeared in the Opatch log file:

2015-09-03_17-30-37 :
 Clusterware is either not running or not configured. You have the following 2 options:
 1. Configure and start the Clusterware on this node and re-run the tool
 2. Run the tool with '-oh <GI_HOME>' to first patch the Grid Home, then invoke tool with '-database <oracle database name>' or '-oh <RAC_HOME>' to patch the RAC home
 Parameter Validation: FAILED

Apparently I should not have shutdown GI before starting. Rather than restarting GI, I chose door #2, that is patch GI and DB homes separately. Seems simple enough. Here's the command I used: [code lang="bash"]/u01/app/grid/product/12.1.0/grid/OPatch/opatchauto apply /u01/app/oracle/patches/20996835 -oh /u01/app/grid/product/12.1.0/grid -ocmrf /u01/app/oracle/patches/20996835/unconfig.rsp[/code] Again, all went well for a few moments, and then another failure. This time a file could not be found. This is from the log file:

Patch 20831113:
 onewaycopyAction : Source File "/u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl" does not exists or is not readable
 'oracle.crs, 12.1.0.2.0': Cannot copy file from 'dropdb.pl' to '/u01/app/grid/product/12.1.0/grid/crs/install/dropdb.pl'

That seemed rather odd. The patch is running as root, and the file is definitely readable by root: [code lang="bash"][oracle@ora12102a 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl -rwx------ 1 oracle asmdba 3541 Jul 12 04:04 /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl[/code] Thinking this might just be a one of error, I changed the permissions on this file to global read: [code lang="bash"][oracle@ora12102a 20996835]$ chmod +r /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl [oracle@ora12102a 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl -rwxr--r-- 1 oracle asmdba 3541 Jul 12 04:04 /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl[/code] ... and then re-ran the patch command. Unfortunately yet another failure. Similar to the previous error, but of course this time a different file:

Patch 19769480:
 onewaycopyAction : Source File "/u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch" does not exists or is not readable
 'oracle.rdbms, 12.1.0.2.0': Cannot copy file from 'sqlpatch' to '/u01/app/grid/product/12.1.0/grid/sqlpatch/sqlpatch'

Checking on the file, again readable by root: [code lang="bash"][oracle@ora12102a 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch -rwx--x--- 1 oracle asmdba 2808 Jul 12 04:03 /u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch[/code] Searching the Oracle Support site on this issue led to this note: OPatch fails with error "sqlpatch does not exists or is not readable" (Doc ID 2026594.1) The solution as per this note is as follows:

Make sure to download the patch as Oracle Software owner and install with same user account.

As a workaround, to fix this error, we have to give the read permission to the file which is issue

Executing the patch as the Oracle software owner however simply does not work: [code lang="bash"][grid@ora12102a ~]$ $ORACLE_HOME/OPatch/opatchauto apply /u01/app/oracle/patches/20996835/ -oh $ORACLE_HOME -ocmrc /u01/app/oracle/patches/20996835/unconfig.rsp Cannot run as 'grid'. Please run as root user.[/code] Ok, so what is the real solution? It was apparent that the reason for these errors is that at some point the patching process is doing an su (or something similar) to the software owners account, in this case the grid user. And grid cannot read that file. The simplest solution seemed to be this: Make all files and directories in the patch directory globally readable. [code lang="bash"][oracle@ora12102a ~]$ cd /u01/app/oracle/patches/20996835/ [oracle@ora12102a 20996835]$ chmod -R +r . [/code] Yes, it was just that easy. Too bad the patch files do not have the correct permissions right out of the box. Following this the patch succeeded for the GI home, and then also for the DB home. And yes, I did run datapatch as well. Has the 12c database stopped crashing? I don't know yet, but will report back later.

References:

Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets (Doc ID 1454618.1)
OPatch fails with error "sqlpatch does not exists or is not readable" (Doc ID 2026594.1)
How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)
Master Note For OPatch (Doc ID 293369.1)

Patching fun: GI and database patches

References:

Patches:

No Comments Yet

Subscribe by email

Share this

Patching fun: GI and database patches

References:

Patches:

Share this

No Comments Yet

Subscribe by email