While Upgrading from GI 12c to 19c, +asm1(19c) Cannot Detect +asm2(12c)

A few weeks ago, I was working on a GI upgrade from 12.2 to 19.6. After running the upgrade on Node 1, an issue occurred where +asm1(19c) could not detect +asm2(12c). The information below describes the actions I performed to resolve the problem.

First, I ran the pre-checks. Everything passed successfully.

[grid@vxe-dev-rac-01 bin]$ ./cluvfy stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.2.0.1/grid -dest_crshome /u01/app/19.6.0.0/grid -dest_version 19.1.0.0.0 -fixupnoexec -verbose
 ...
  Node Name Required Running? Comment
  ------------ ------------------------ ------------------------ ----------
  reneace02 no yes passed
  reneace01 no yes passed
 Verifying Node Application Existence ...PASSED
 Verifying Check incorrectly sized ASM Disks ...PASSED
 Verifying ASM disk group free space ...PASSED
 Verifying Network configuration consistency checks ...PASSED
 Verifying File system mount options for path GI_HOME ...PASSED
 Verifying /boot mount ...PASSED
 Verifying OLR Integrity ...PASSED
 Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...PASSED
 Verifying User Equivalence ...PASSED
 Verifying RPM Package Manager database ...INFORMATION (PRVG-11250)
 Verifying Network interface bonding status of private interconnect network interfaces ...PASSED
 Verifying /dev/shm mounted as temporary file system ...PASSED
 Verifying File system mount options for path /var ...PASSED
 Verifying DefaultTasksMax parameter ...PASSED
 Verifying zeroconf check ...PASSED
 Verifying ASM Filter Driver configuration ...PASSED
 Verifying Systemd login manager IPC parameter ...PASSED
 Verifying Kernel retpoline support ...PASSED
 
 Pre-check for cluster services setup was successful.
 Verifying RPM Package Manager database ...INFORMATION
 PRVG-11250 : The check "RPM Package Manager database" was not performed because
 it needs 'root' user privileges.
 
 
 CVU operation performed: stage -pre crsinst
 Date: Feb 20, 2020 11:28:43 AM
 CVU home: /u01/temp/19.6.0.0/cvu/bin/../
 User: grid

Then, I created the following response file so that I could run the GI upgrade in silent mode:

[grid@reneace01 grid]$ cat /tmp/gridresponse.rsp | grep -v "#"
 oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
 INVENTORY_LOCATION=/u01/app/oraInventory
 oracle.install.option=UPGRADE
 ORACLE_BASE=/u01/app/grid
 oracle.install.asm.OSDBA=
 oracle.install.asm.OSOPER=
 oracle.install.asm.OSASM=
 oracle.install.crs.config.scanType=LOCAL_SCAN
 oracle.install.crs.config.SCANClientDataFile=
 oracle.install.crs.config.gpnp.scanName=
 oracle.install.crs.config.gpnp.scanPort=
 oracle.install.crs.config.ClusterConfiguration=STANDALONE
 oracle.install.crs.config.configureAsExtendedCluster=false
 oracle.install.crs.config.memberClusterManifestFile=
 oracle.install.crs.config.clusterName=vxe-dev-cluster
 oracle.install.crs.config.gpnp.configureGNS=false
 oracle.install.crs.config.autoConfigureClusterNodeVIP=false
 oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
 oracle.install.crs.config.gpnp.gnsClientDataFile=
 oracle.install.crs.config.gpnp.gnsSubDomain=
 oracle.install.crs.config.gpnp.gnsVIPAddress=
 oracle.install.crs.config.sites=
 oracle.install.crs.config.clusterNodes=reneace01:reneace01-vip,reneace02:reneace02-vip
 oracle.install.crs.config.networkInterfaceList=
 oracle.install.crs.configureGIMR=false
 oracle.install.asm.configureGIMRDataDG=false
 oracle.install.crs.config.storageOption=
 oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=
 oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=
 oracle.install.crs.config.useIPMI=false
 oracle.install.crs.config.ipmi.bmcUsername=
 oracle.install.crs.config.ipmi.bmcPassword=
 oracle.install.asm.SYSASMPassword=
 oracle.install.asm.diskGroup.name=ORACRS
 oracle.install.asm.diskGroup.redundancy=
 oracle.install.asm.diskGroup.AUSize=
 oracle.install.asm.diskGroup.FailureGroups=
 oracle.install.asm.diskGroup.disksWithFailureGroupNames=
 oracle.install.asm.diskGroup.disks=
 oracle.install.asm.diskGroup.quorumFailureGroupNames=
 oracle.install.asm.diskGroup.diskDiscoveryString=
 oracle.install.asm.monitorPassword=
 oracle.install.asm.gimrDG.name=
 oracle.install.asm.gimrDG.redundancy=
 oracle.install.asm.gimrDG.AUSize=4
 oracle.install.asm.gimrDG.FailureGroups=
 oracle.install.asm.gimrDG.disksWithFailureGroupNames=
 oracle.install.asm.gimrDG.disks=
 oracle.install.asm.gimrDG.quorumFailureGroupNames=
 oracle.install.asm.configureAFD=false
 oracle.install.crs.configureRHPS=false
 oracle.install.crs.config.ignoreDownNodes=false
 oracle.install.config.managementOption=NONE
 oracle.install.config.omsHost=
 oracle.install.config.omsPort=
 oracle.install.config.emAdminUser=
 oracle.install.config.emAdminPassword=
 oracle.install.crs.rootconfig.executeRootScript=false
 oracle.install.crs.rootconfig.configMethod=
 oracle.install.crs.rootconfig.sudoPath=
 oracle.install.crs.rootconfig.sudoUserName=
 oracle.install.crs.config.batchinfo=
 oracle.install.crs.app.applicationAddress=
 oracle.install.crs.deleteNode.nodes=

I then executed the upgrade. As shown below, there were no failures after executing the rootupgrade.sh script:

[grid@reneace01 grid]$ pwd
 /u01/app/19.6.0.0/grid
 [grid@reneace01 grid]$ ./gridSetup.sh -silent -responseFile /tmp/gridresponse.rsp -applyRU /u06/patches/oracle19c/30501910
 Preparing the home to patch...
 Applying the patch /u06/patches/oracle19c/30501910...
 Successfully applied the patch.
 The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/installerPatchActions_2020-02-26_07-27-27PM.log
 Launching Oracle Grid Infrastructure Setup Wizard...
 
 [WARNING] [INS-13014] Target environment does not meet some optional requirements.
  CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log
  ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
 The response file for this session can be found at:
  /u01/app/19.6.0.0/grid/install/response/grid_2020-02-26_07-27-27PM.rsp
 
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log
 
 
 As a root user, execute the following script(s):
  1. /u01/app/19.6.0.0/grid/rootupgrade.sh
 
 Execute /u01/app/19.6.0.0/grid/rootupgrade.sh on the following nodes:
 [reneace01, reneace02]
 
 Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.
 
 Successfully Setup Software with warning(s).
 As install user, execute the following command to complete the configuration.
  /u01/app/19.6.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/gridresponse.rsp [-silent]
 
 
 [grid@reneace01 GridSetupActions2020-02-26_07-27-27PM]$ tail -f /u01/app/19.6.0.0/grid/install/root_reneace01.network_2020-02-26_20-09-57-369528654.log
  ORACLE_HOME= /u01/app/19.6.0.0/grid
  Copying dbhome to /usr/local/bin ...
  Copying oraenv to /usr/local/bin ...
  Copying coraenv to /usr/local/bin ...
 
 Entries will be added to the /etc/oratab file as needed by
 Database Configuration Assistant when a database is created
 Finished running generic part of root script.
 Now product-specific root actions will be performed.
 Relinking oracle with rac_on option
 Using configuration parameter file: /u01/app/19.6.0.0/grid/crs/install/crsconfig_params
 The log of current session can be found at:
  /u01/app/grid/crsdata/reneace01/crsconfig/rootcrs_reneace01_2020-02-26_08-10-30PM.log
 2020/02/26 20:11:02 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
 2020/02/26 20:11:02 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
 2020/02/26 20:11:02 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
 2020/02/26 20:11:09 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
 2020/02/26 20:11:09 CLSRSC-464: Starting retrieval of the cluster configuration data
 2020/02/26 20:11:46 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
 2020/02/26 20:14:49 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
 2020/02/26 20:17:53 CLSRSC-693: CRS entities validation completed successfully.
 2020/02/26 20:18:02 CLSRSC-515: Starting OCR manual backup.
 2020/02/26 20:18:18 CLSRSC-516: OCR manual backup successful.
 2020/02/26 20:18:28 CLSRSC-486:
  At this stage of upgrade, the OCR has changed.
  Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
 2020/02/26 20:18:28 CLSRSC-541:
  To downgrade the cluster:
  1. All nodes that have been upgraded must be downgraded.
 2020/02/26 20:18:28 CLSRSC-542:
  2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
 2020/02/26 20:18:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
 2020/02/26 20:18:39 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
 2020/02/26 20:18:40 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
 2020/02/26 20:18:46 CLSRSC-363: User ignored prerequisites during installation
 2020/02/26 20:19:02 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
 2020/02/26 20:19:02 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.
 2020/02/26 20:22:26 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
 2020/02/26 20:22:26 CLSRSC-482: Running command: '/u01/app/12.2.0.1/grid/bin/crsctl start rollingupgrade 19.0.0.0.0'
 CRS-1131: The cluster was successfully set to rolling upgrade mode.
 2020/02/26 20:22:31 CLSRSC-482: Running command: '/u01/app/19.6.0.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.2.0.1/grid -oldCRSVersion 12.2.0.1.0 -firstNode true -startRolling false '
 
 ASM configuration upgraded in local node successfully.
 
 2020/02/26 20:25:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
 2020/02/26 20:25:34 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
 2020/02/26 20:26:08 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
 2020/02/26 20:26:11 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
 2020/02/26 20:26:12 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
 2020/02/26 20:26:30 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
 2020/02/26 20:26:30 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
 2020/02/26 20:26:41 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
 2020/02/26 20:26:51 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
 2020/02/26 20:26:51 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
 2020/02/26 20:29:52 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
 2020/02/26 20:33:58 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
 2020/02/26 20:34:07 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
 2020/02/26 21:22:11 CLSRSC-343: Successfully started Oracle Clusterware stack
 clscfg: EXISTING configuration version 5 detected.
 Successfully taken the backup of node specific configuration in OCR.
 Successfully accumulated necessary OCR keys.
 Creating OCR keys for user 'root', privgrp 'root'..
 Operation successful.
 2020/02/26 21:22:36 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
 2020/02/26 21:22:41 CLSRSC-474: Initiating upgrade of resource types
 2020/02/26 22:07:33 CLSRSC-475: Upgrade of resource types successfully initiated.
 2020/02/26 22:07:48 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
 2020/02/26 22:07:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

However, my team and I noticed that the +asm1 instance was not detecting the +asm2 instance. While troubleshooting the issue, we discovered that Node 1 and Node 2 had different netmasks:

# Node 1
 eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 169.254.13.192 netmask 255.255.224.0 broadcast 169.254.31.255
 ether 00:50:56:8e:21:77 txqueuelen 1000 (Ethernet)
 
 # Node 2
 eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 169.254.60.115 netmask 255.255.0.0 broadcast 169.254.255.255
 ether 00:50:56:8e:b4:4a txqueuelen 1000 (Ethernet)

After consulting with Oracle, we were able to confirm that this is a known issue with 19c. Note that the documents and bugs below are internal:

• ASM Fails to Start After Running "rootupgrade.sh" on First Node While Upgrading to 19c Grid Infrastructure (Doc ID 2606735.1)
• Bug 30265357 - FAILS TO START ORA.ASM WHEN UPGRADE FROM 12.1.0.2 TO 19.3.0
• Bug 30452852 - DURING UPGRADE FROM 12C TO 19C +ASM1(19C) FAILS TO JOIN ALREADY RUNNING +ASM2(12C)
• Bug 29379299 - HAIP SUBNET FLIPS UPON BOTH LINK DOWN/UP EVENT

To resolve this issue, do the following:

1. Shut down Node 1 (crsctl stop crs).
2. Run rootupgrade.sh in Node 2.
3. After the rootupgrade.sh script completes in Node 2, start up both Node 1 and Node 2.
4. To finish the upgrade in Node 1, run /u01/app/19.6.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/gridresponse.rsp -silent.

Make sure you are aware of this bug if you are preparing to perform a GI upgrade from 12.x to 19.6. Otherwise, the upgrade causes a complete outage (not a rolling upgrade).

Note: This post was originally published at rene-ace.com.