Exadata: How to manage OS dependencies when upgrading to 19c

Exadata 19c comes with Linux 7 and Exadata 12c and 18c come with Linux 6. There will then be an upgrade to Linux 7 when upgrading from 12c or 18c to Exadata 19c or above. In this specific case, you may face specific dependencies issues which I will explain how to deal with. This can be split into two more general cases:

• When there is no OS upgrade, I have documented the way of dealing with dependencies in this blog.
• If you have an OS upgrade - if you are upgrading from 12c or 18c to 19c and above - you will have to follow a specific way of dealing with this due to the upgrade to Linux 7 which I describe below.

When running the usual database servers pre-requisites while upgrading to 19c, the dependencies error message will be slightly different than the usual one (when no OS upgrade happens):

[root@exacel01 dbserver_patch_19.190411]# ./patchmgr -dbnodes ~/dbs_group -precheck -iso_repo /tmp/SAVE/p29542809_*_Linux-x86-64.zip -target_version 19.2.1.0.0.190410 -allow_active_network_mounts
 ************************************************************************************************************
 NOTE patchmgr release: 19.190411 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
 NOTE
 WARNING Do not interrupt the patchmgr session.
 WARNING Do not resize the screen. It may disturb the screen layout.
 WARNING Do not reboot database nodes during update or rollback.
 WARNING Do not open logfiles in write mode and do not try to alter them.
 ************************************************************************************************************
 2019-05-07 19:49:41 -0500 :Working: DO: Initiate precheck on 4 node(s)
 2019-05-07 19:50:33 -0500 :Working: DO: Check free space and verify SSH equivalence for the root user to node(s)
 2019-05-07 19:50:46 -0500 :SUCCESS: DONE: Check free space and verify SSH equivalence for the root user to node(s)
 2019-05-07 19:52:02 -0500 :Working: DO: dbnodeupdate.sh running a precheck on node(s).
 2019-05-07 19:57:29 -0500 :ERROR : dbnodeupdate.sh precheck failed on one or more nodes
  SUMMARY OF WARNINGS AND ERRORS FOR exadb01:
  . . .
  exadb01: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb01:/var/log/cellos/unkown_packages-rpt.070519194937.txt for more details.
  exadb01: WARNING: This Exadata update includes an Oracle Linux 6 to Oracle Linux 7 update.
  exadb01: WARNING: Although the system was analyzed for custom rpms, any other custom installed software (such as tar-balls) cannot be detected.
  exadb01: WARNING: After updating to another major OS it may be necessary to update any script used for mounting DBFS so that it can correctly determine whether or not DBFS has been successfully mounted since the expected output from the 'stat' command, which is used to determine the status of the DBFS mount point, has changed. If an early version of mount-dbfs.sh from Note 1054431.1 has been used or extended, download and use/extend the new version that checks for 'fuseblk' as the output of the 'stat' command. Failure to update the script will lead to incorrect status being returned for DBFS, with the script reporting that DBFS has not been mounted successfully when it has. Any CRS resource that makes use of such a script will also show the incorrect status!
  exadb01: The following known issues will be checked for but require manual follow-up:
  exadb01: (*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
  exadb01: (*) - DBFS mount scripts may fail after a major os upgrade
 

And when looking at the *unkown_packages-rpt* file:

[root@exadb01]# cat /var/log/cellos/unkown_packages-rpt.070519194937.txt
 #################################################################################
 # File initialized at 070519_195509 (runid :070519194937) by dbnodeupdate.sh 19.190411
 # NOTE: This list contains rpms which are seen as custom rpms
 #################################################################################
 . . .
 list of customs RPMs here
 . . .
 #################################################################################
 # ALERT : Custom packages found (see above)
 # These custom packages MUST be removed before proceeding a major Oracle Linux upgrade.
 # Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed - OR -
 # Run /var/log/cellos/remove_unkown_packages.070519194937.sh manually as root before re-running prereq check.
 #
 # RECOMMENDED : Let dbnodeupdate.sh remove the packages right after the mandatory backup by using the -R flag
 # WARNING : Removing custom packages may impact functionality
 # NOTE : When removed functionality is still required after the update, reinstall the rpm after the update
 #################################################################################
 [root@exadb01]#
 

So in this case, clearly the customs RPMs MUST be removed to be able to upgrade to 19c, and then to Linux 7. We have a few ways of achieving this:

• Using a generated script (/var/log/cellos/remove_unkown_packages.070519194937.sh) right now
• Using a generated script (/var/log/cellos/remove_unkown_packages.070519194937.sh) just before the upgrade
• Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed -- this is the patchmgr option -force_remove_custom_rpms

I personally recommend going with the patchmgr option -force_remove_custom_rpms which would do everything for us at upgrade time. One thing: if we check the patchmgr documentation, -force_remove_custom_rpms is still to be used to upgrade from Linux 5 to Linux 6:

 -force_remove_custom_rpms
  Remove any custom rpms when db node update goes from OL5 to OL6.
 

Oracle may have missed updating the patchmgr help. You can indeed use -force_remove_custom_rpms from OL6 to OL7 as well. So . . . in case of an upgrade to Exadata 19c, you just have to add -force_remove_custom_rpms to your patchmgr command line when patching:

[root@exacel01 dbserver_patch_19.190411]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/SAVE/p29542809_*_Linux-x86-64.zip -target_version 19.2.1.0.0.190410 -allow_active_network_mounts -force_remove_custom_rpms -rolling &
 

And you are good to go! (Also, you have to organize the customs RPMs re-installation after patching or you may use this opportunity for a bit of a cleanup!) A general rule to keep in mind for the dependencies issues when patching Exadata is:

• When there is no OS upgrade, follow this blog.
• When there is an OS upgrade, add -force_remove_custom_rpms to your patchmgr command line when patching.