Pythian Blog: Technical Track

Recovering from quarterly PSU patching disaster

Applying GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.181016(Patch 28429134) for AIX (64-Bit) was a disaster for Oracle Restart environment, as described in a previous post.
# $GRID_HOME/OPatch/opatch auto $GI_PSU -ocmrf /tmp/ocm.rsp
 
 patch /stage/psu/11g/GI/2018OCT/28429134/27735020/custom/server/27735020 apply successful for home /u01/app/oracle/product/11.2.0/db_1
 patch /stage/psu/11g/GI/2018OCT/28429134/28204707 apply failed for home /u01/app/oracle/product/11.2.0/db_1
 
 patch /stage/psu/11g/GI/2018OCT/28429134/27735020 apply successful for home /u01/11.2.0/grid
 patch /stage/psu/11g/GI/2018OCT/28429134/28204707 apply failed for home /u01/11.2.0/grid
 
After failed patching, the following commands failed; hence, a rollback was not a feasible option and we needed to restore GI and DB Home.
# crsctl config has
 exec(): 0509-036 Cannot load program crsctl.bin because of the following errors:
 rtld: 0712-001 Symbol ztca_Shutdown was referenced
  from module crsctl.bin(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced
  from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-001 Symbol ztpk_DestroyKey was referenced
  from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-001 Symbol ztpk_Sign was referenced
  from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-001 Symbol ztpk_Verify was referenced
  from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-002 fatal error: exiting.
 
 $ srvctl config database -d $ORACLE_SID
 exec(): 0509-036 Cannot load program getcrshome because of the following errors:
 rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced
  from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 rtld: 0712-001 Symbol ztpk_DestroyKey was referenced
  from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition
  of the symbol was not found.
 
Below are the high level steps we implemented to restore, reconfigure, and relink GI and DB Home. These may vary by environment.
Restore GI HOME:
 cd $GRID_HOME
 cp /backup/GIHome.tar .
 tar -xvpf GIHome.tar
 $GRID_HOME/crs/install/roothas.pl -unlock
 cd $GRID_HOME/rdbms/lib/
 ls -l config.o
 mv config.o config.o.bkup
 $GRID_HOME/bin/relink all
 
 # ./rootadd_rdbms.sh
 # ./roothas.pl -patch
 
 Restore DB HOME:
 cd $ORACLE_HOME
 cp /backup/DBHome.tar .
 tar -xvpf DBHome.tar
 $ORACLE_HOME/bin/relink all
 
In conclusion, don't depend solely on rollback to recover from failed patching. It might be a good idea to back up binaries. Some useful references: How To Relink The Oracle Grid Infrastructure Standalone (Restart) Installation Or Oracle Grid Infrastructure RAC/Cluster Installation (11.2 or 12c). ( Doc ID 1536057.1 ) How to backup a Grid Infrastructure installation (Doc ID 1482803.1) Relinking Oracle Home FAQ ( Frequently Asked Questions) (Doc ID 1467060.1) It's nice to see Oracle's updated Patch 28429134 has been withdrawn. Master Note for Database Proactive Patch Program (Doc ID 756671.1) GI PSU 11.2.0.4.181016 Patch 28429134 All except HP-UX PA-RISC (64-Bit), AIX (64-Bit) ETA: 13-November-2018

No Comments Yet

Let us know what you think

Subscribe by email