Applying Oracle 11.2 April 2010 PSU for Single-Instance ASM and DBMS

When news of the April 2010 PSU for Oracle 11.2 came out, I was excited to see it, since it marked the first non-one-off patch release for the 11.2 database software. I happened to have an 11gR2 test system running on 11gR2 ASM via standalone Grid Infrastructure. I applied PSU 9352237 to the DBMS home and fired it up, only to see the folly of my ways when any ASM file operations like disk resizing (or auto-extending) failed with ORA-1653. This was due to the DBMS component now having a higher version number than the ASM component, which ASM does not allow. The Grid Infrastucture PSU would need to be applied to bring the ASM component up to snuff, but that patch (9343627) was, at that time, only “announced” with no ETA. Alas, the patch was rolled back and we continued testing without it.

Then this week I checked again and saw that PSU 9343627 was released and gave it a whirl. I was a little confused when the README seemed to contain a lot of instructions that always assumed it to be on a clustered, RAC install. My setup was a single-instance Grid Infrastructure installation just to provide ASM. I soon met problem upon problem when going through first this setup step:

# GRID_HOME/crs/install/rootcrs.pl -unlock
2010-05-03 11:40:42: Parsing the host name
2010-05-03 11:40:42: Checking for super user privileges
2010-05-03 11:40:42: User has super user privileges
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
The Oracle Clusterware stack failed to stop.
You should stop the stack with 'crsctl stop crs' and rerun the command

I decided to try the PSU anyway, thinking Oracle would certainly have noted in the README if this was a problem for single-node installations. Turns out I was wrong:

$ GRID_HOME/OPatch/opatch napply -local -oh GRID_HOME -id 9343627,9352237
Invoking OPatch 11.2.0.1.2
... snip ...
Running prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:
Patch 9343627:
Copy Action: Desctination File "GRID_HOME/bin/oradnssd" is not writeable.
'oracle.crs, 11.2.0.1.0': Cannot copy file from 'oradnssd' to 'GRID_HOME/bin/oradnssd'
UtilSession failed: Prerequisite check "CheckApplicable" failed.
OPatch failed with error code 73

I created an SR with Oracle support, and after reviewing my case agreed that the instructions were not right for single-instance Grid Infrastructure installations. They will be updating the README, but in the meantime I’d like to share the revised instructions that got me through two successful GI/DBMS PSU applications today.

Naming Conventions
First a few naming conventions to avoid confusion:

• DBMS_HOME – The Oracle DBMS installation directory.
• GRID_HOME – The Oracle Grid Infrastructure installation directory.

I’ve replaced all references to the directories in my examples with DBMS_HOME and GRID_HOME. Whenever you see those strings, substitute the actual path to your DBMS install or GRID install, whichever the case may be.

Download the PSU and OPatch 11.2
First, you’ll obviously need to download patch 9343627 that contains the PSU for your platform. In my case that’s Linux x86_64. As of this writing, I believe it’s only available for Linux x86 and x86_64. Note that patch 9343627 also includes patch 9352237. You do not have to, and should not, download that PSU separately. Everything you need is in the 9343627 download.

Second, choose a working directory to unzip that file. In my case, this is /home/oracle/software/11gR2/psu/gi, which looked like this after unzipping the PSU zip file:

$ ls
9343627  9352237  p9343627_112010_Linux-x86-64.zip

Unlike normal one-off patches, where you would descend into the individual patch-number-named directories, you’ll stay at this level.

Note also that you’ll need OPatch 11.2 or higher. See MOS Doc ID 6880880 and get the latest version of OPatch for your platform. Then just unzip the file in the DBMS_HOME and GRID_HOME directories, overwriting the old OPatch directory.

Pre-Patch Steps
First I’m assuming that you have shut down all DBMS instances on this server. We will shut down ASM later in this process, so be sure to shutdown the DBMS instances now.

Now we’ll start the pre-patch steps. First we stop the database on this server, saving the state of the database configuration in /tmp/oracle_home.stat:

$ DBMS_HOME/bin/srvctl stop home -o DBMS_HOME -s /tmp/oracle_home.stat

Repeat this for any additional Oracle database home installs on the server.

Next we save the database home configuration. This only needs to be done if we’re also going to patch the DBMS home in addition to the GI home. This is done from the directory where we unzipped the PSU:

$ ./9343627/custom/server/9343627/custom/scripts/prepatch.sh -dbhome DBMS_HOME
./9343627/custom/server/9343627/custom/scripts/prepatch.sh completed successfully.

After that, we shut down ASM as one would normally do it:

$ . oraenv
+ASM
$ sqlplus / as sysasm
SQL> shutdown immediate;
SQL> quit

Then, as root, we unlock and shutdown HAS (not CRS, as the original README would have you do):

# GRID_HOME/crs/install/roothas.pl -unlock
2010-05-04 17:14:08: Checking for super user privileges
2010-05-04 17:14:08: User has super user privileges
2010-05-04 17:14:08: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock GRID_HOME

Now we’re ready to do some patching.

Applying PSU 9343627 and 9352237 to Oracle 11.2 Grid Infrastructure
As I mentioned earlier, the PSU contains both 9343627 and 9352237, and it is advised that you apply both. I did so successfully with these commands:

$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9343627
$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9352237

For each one, you’ll go through the normal opatch [y|n] confirmation prompts, and also prompted for an email address. I entered in my MOS login email, and left the password blank (it is optional). The rest was automatic and finished without error for both cases.

Applying PSU 9343627 and 9352237 to Oracle 11.2 DBMS
Here is something slightly different when applying the 9343627 patch to the DBMS_HOME, we specify a subdirectory after the “napply” parameter:

$ DBMS_HOME/OPatch/opatch napply 9343627/custom/server/ -oh DBMS_HOME -id 9343627

For patch 9352237, we use the regular syntax:

$ DBMS_HOME/OPatch/opatch napply -oh DBMS_HOME -id 9352237

Note that I use OPatch from DBMS_HOME when patching DBMS_HOME, and from GRID_HOME when patching GRID_HOME. I can’t say if it is actually necessary, but I like to keep things tidy. This is why I advised installing the new OPatch into both homes earlier.

As of this moment, both homes are patched, but we aren’t done yet!

Post-Patch Steps
First we run the postpatch.sh script included in patch 9343627 to reset some file permissions on the DBMS_HOME:

$ ./9343627/custom/server/9343627/custom/scripts/postpatch.sh -dbhome DBMS_HOME

Now we re-lock the GRID_HOME and restart the GI stack. As root:

# GRID_HOME/crs/install/roothas.pl -patch
2010-05-04 17:33:41: Checking for super user privileges
2010-05-04 17:33:41: User has super user privileges
2010-05-04 17:33:41: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.

Verify that the services we need are started (again as root):

# crsctl check has
CRS-4638: Oracle High Availability Services is online
# crsctl check css
CRS-4529: Cluster Synchronization Services is online

If you don’t see CSS started, then you aren’t alone. On both of my servers, CSS was not set to auto-start from OHASD. This was fixed by running these commands as the GRID_HOME owner (oracle):

$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"

Then, as root:

# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.
# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
# crsctl check has
CRS-4638: Oracle High Availability Services is online
# crsctl check css
CRS-4529: Cluster Synchronization Services is online

Moving on in the post-patch steps, we start the DBMS_HOME from the state file we created earlier:

$ DBMS_HOME/bin/srvctl start home -o DBMS_HOME -s /tmp/oracle_home.stat

At this point, we are done. You can start ASM and the DBMS instances. For some affirmation, run this for both GRID_HOME and DBMS_HOME (just using GRID_HOME in this example)

$ GRID_HOME/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.2
Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.
Oracle Home       : GRID_HOME
Central Inventory : /opt/oracle/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : GRID_HOME/oui
Log file location : GRID_HOME/cfgtoollogs/opatch/opatch2010-05-04_18-36-23PM.log
Patch history file: GRID_HOME/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : GRID_HOME/cfgtoollogs/opatch/lsinv/lsinventory2010-05-04_18-36-23PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch  9352237      : applied on Tue May 04 18:22:41 EDT 2010
Unique Patch ID:  12381846
   Created on 25 Mar 2010, 00:05:17 hrs PST8PDT
   Bugs fixed:
     8661168, 8769239, 8898852, 8801119, 9054253, 8706590, 8725286, 8974548
     8778277, 8780372, 8769569, 9027691, 9454036, 9454037, 9454038, 8761974
     7705591, 8496830, 8702892, 8639114, 8723477, 8729793, 8919682, 8818983
     9001453, 8475069, 9328668, 8891929, 8798317, 8820324, 8733749, 8702535
     8565708, 9036013, 8735201, 8684517, 8870559, 8773383, 8933870, 8812705
     8405205, 8822365, 8813366, 8761260, 8790767, 8795418, 8913269, 8897784
     8760714, 8717461, 8671349, 8775569, 8898589, 8861700, 8607693, 8642202
     8780281, 9369797, 8780711, 8784929, 8834636, 9015983, 8891037, 8828328
     8570322, 8832205, 8665189, 8717031, 8685253, 8718952, 8799099, 8633358
     9032717, 9321701, 8588519, 8783738, 8796511, 8782971, 8756598, 9454385
     8856497, 8703064, 9066116, 9007102, 8721315, 8818175, 8674263, 9352237
     8753903, 8720447, 9057443, 8790561, 8733225, 9197917, 8928276, 8991997, 8837736
Patch  9343627      : applied on Tue May 04 18:18:31 EDT 2010
Unique Patch ID:  12381846
   Created on 15 Apr 2010, 11:28:38 hrs PST8PDT
   Bugs fixed:
     9343627, 9262748, 9262722
--------------------------------------------------------------------------------
OPatch succeeded.

As I said, Oracle support will be updating the README for PSU 9343627, but in the meantime I hope this guide helps you as much as it helped me. I’d like to thank my Pythian colleague Alex Gorbachev for his help in diagnosing some of the GI problems after the first broken patching, and also Esteban B. at Oracle Support for working closely with us to get a new single-node action plan.