Exadata Patching Overview

Hello everyone !

For my first time posting here on the Pythian Blog, I would like to share some of my tips/notes about patching Oracle Exadata, based on my experiences and not less important, research and googling =).

First, the Oracle Exadata Patch has 3 different components that should be patched. As we know about Oracle Exadata, the Exadata rack has a different components, like Cisco Switch, KVM, Power Distribuion Unit, etc… and we only are responsible for patching the Database Servers (usually referenced as compute nodes), Storage Servers (usually referenced as cell nodes) and the Infiniband Switches.

We can divide the patches in 3 different parts:

Storage Server Patch
Database Server Patch
Infiniband Switches Patch

Before starting, I would like to share and note here 2 documents from My Oracle Support, aka metalink. These notes must be the first place that you need to go to review before patching the Exadata environment.

Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions (Doc ID. 888828.1)
– This is for the second and third generation (V2 and X2) for Oracle Exadata, using Sun hardware.

Database Machine and Exadata Storage Server 11g Release 1 (11.1) Supported Versions (Doc ID. 835032.1)
– This is for the first generation (V1) for Oracle Exadata, using HP hardware.

Oracle usually updates these documents for every patch that is released, including different information about that.

1) Exadata Storage Server Patch (cell nodes):

The Storage Server Patch is responsible for keeping our cell nodes always up-to-date, fixing possibles problems and this patch includes different component patches, like kernel patches, firmware, operation system, etc… for the Storage Server.
These patches are usually lunched from any Database Server (typically using the compute node 1) using dcli and ssh to remotely patch each cell node.
All the Storage Server patch came with what the Oracle calls as “Database Minimal Pack” or “Database Convenience Pack”. These packs are included in a .zip file inside the storage patch software and must be applied in all the Database Servers (compute nodes), because it includes, as in the storage patch, kernel, firmware, and operation system patches necessary.
Applying these patches in the Storage Server, will also change the Linux version for each server, but for the Database Minimal Pack, it will NOT change the linux version.

SOME USEFUL COMMANDS:

— Rolling Patch Example
./patchmgr -cells <cells group> -patch_check_prereq -rolling
./patchmgr -cells <cells group> -patch -rolling

— Non-Rolling Patch Example
./patchmgr -cells <cells group> -patch_check_prereq
./patchmgr -cells <cells group> -patch

— Information about the versions in the kernel.
— run in the database server, also the status of the patch.
imageinfo

— Logs to check
/var/log/cellos/validations.log
/var/log/cellos/vldrun*.log

— File with all the cell nodes names, used in the <cells group> tag
cat /opt/Oracle.SupportTools/onecommand/cell_group

HOW ROLLING PATCH WORKS:

Since version 11.2.1.3.1, Oracle have introduced the feature “Cell Rolling Apply” in order to simplify the install and of course, to reduce downtime. You must be aware about that sometimes is necessary some patches in the compute nodes BEFORE using -rolling option in the cells nodes.

Below is a brief overview in how it works:

Preparation of the cell node X
Loop
Turn all Grid Disks offline in the cell node X
Patch cell node X
Grid Disks online
End Loop

2) Exadata Database Server Patch (compute nodes):

This the patch that we are used to work as an Oracle DBA. These patches are being released specifically for Exadata, so you should not upgrade one Database to a version that is not support to the Exadata, so always check in the documentation if the release that you want was already published. The software to patches already contain upgrades for the Oracle Database and Oracle Clusterware.

All the patches provided as a bundle patches are comulative, and it also may include the most recent Patch Set Update (PSU) and/or Critical Patch Update (CPU).

a) So, for example, if you are in the 11.2.0.2.0 BP 1 (Bundle Patch 1) and would like to migrate to BP 7 (Bundle Patch 7), you just need to upgrade directly, without applying the 2, 3, 4, 5 and 6.
b) As the Bundle Patches have the last PSU and CPU, its not necessary to apply PSU/CPU in the Exadata environment, because its already included in the Bundle Patches. Always check the README file in order to check which PSU/CPU is included or not in the Bundle Patch.

— Bundle Patches Bug List:
11.2.0.2 Patch Bundles for Oracle Exadata Database Machine (Doc ID. 1314319.1)
11.2.0.1 Patch Bundles for Oracle Exadata Database Machine (Doc ID. 1316026.1)

3) Infiniband Switches Patch:

Most of Exadata Machines have 2 leafs of Infiniband switches and 1 spine. The position of the 2 Leafs is usually in U20 and U24 and the Spine is located in the very bottom position, U1. Actually, there are some Exadata Machines that have the Spine switch in the Half or Quarter Rack, but usually only the Full rack should have it.
In order to patch the switches, you must apply the patch in the SPINE switch first, rebooting and waiting it to back online BEFORE proceeding to the next switches (leafs).

All the Infiniband Switchs Patch are not CUMULATIVE, its necessary to upgrade version by version, for example, if I want to migrate the version 1.0.1-1 to 1.3.3-2, I must first upgrade 1.0.1-1 to 1.1.3-2 and after 1.1.3-2 to 1.3.3-2.

4) Order of Patching

This is an overview in how usually you should patch all the components, you may also want to wait a few days to progress to different components, in order to be aware of which component you had patched before starting facing one bug/error.

Infinibands Switchs

• Spine
• Leaf

Storage Server Software

• Cell nodes
• Database Minimal Pack in compute nodes

Database Bundle Patch

• Grid Home (if applicable)
• Oracle Home

CRS MLR patches (11.2.0.1.0)

• Grid Home
• Oracle Home (if applicable)

In order to finish this post, I would like to remember you that you MUST upgrade your TEST/QA/DEV environments before patching Production, I believe that everyone already knows that, but just a reminder =). Always plan the migration keeping all the recommendations in place, try getting some days between patching different components, it will avoid a lot of problems.

I hope that this post helps anyone understand the overview of patching Exadata =).

Have a great day.
cya ;) …

Fábio Galera