Pythian Blog: Technical Track

How to Upgrade Grid Infrastructure to 18c

by Fred Denis on Jul 10, 2019 12:00:00 AM

Now that Oracle Grid Infrastructure (GI) 19c has been released, it is time to upgrade your GI to 18c! I am presenting here a well-tested procedure to upgrade your Grid Infrastructure to 18c. It has been designed and applied on many Exadatas and the procedure can also be applied on non-Exadata systems. You will notice that it is very close to upgrade your Grid Infrastructure to 12c.  

0/ Preparation

Here are a few things that are good to know and read before starting upgrading a GI to 18c:
  • Download the GI 18c image from edelivery: V978971-01.zip.
  • Download the RU you want to apply like Patch 28828717: GI RELEASE UPDATE 18.5.0.0.0; As I use to do this kind of maintenance when I patch the Exadata stack, I usually use the GI RU provided in the Exadata bundle like 28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717 from the Jan 19 Bundle. You can do either way, it is the same RU.
  • GI 18c will be installed on /u01/app/18.1.0.0/grid.
  • GI 12.1 is running from /u01/app/12.1.0.2/grid on the systems I work on.
  • This procedure has been successfully applied on many Exadata's half rack and full rack; it also applies to non-Exadata GI system.
  • I use the rac-status.sh script to check the status of all the resources of my cluster before and after the maintenance to avoid any unpleasantness
  • Check your oratab entries to avoid having them deleted during the upgrade as explained in this post

Versions naming:

It is not a secret to anyone, Oracle's version naming has always been changed in a not-really-consistent manner across the years making it not really easy to follow. Then this new style of yearly release also came with its new version numbering. Indeed, we used to name our directories with the version number like 12.1.0.2 and then create a new directory like 12.2.0.1 when we would perform an out-of-place upgrade to a new major version.
Nowadays, even if 18c is actually version 12.2.0.2, the database came with an 18.1 version and GI came with an 18.0 version. Each RU will increment the second number of the version: January 19 RU (patch 28828717) being GI version 18.5, and so on.
I had a discussion with some friends about how to name the GI directory knowing that and also that nowadays, Oracle promotes the Rapid Home Provisionning tool for an "always out of place" patch strategy. We decided that we wouldn't go with RHP nor an out-of-place patching for each RU but for every major release as we were doing before. It would then have made no sense to name our directory /u01/app/18.5.0.0/grid for GI upgrade to 18c + Jan 19 Bundle as April 19 Bundle would upgrade the version to 18.6; we then have decided to name our GI 18c directory with the original version number /u01/app/18.1.0.0/grid which makes it clear to everyone (18.0 looked weird to me, indeed, a "0" version doesn't make much sense to me). Also, we kept four numbers in the naming to stay consistent with what we were doing before and with the databases homes.
And the day you need to double check what exact patch is installed here, we would rely on something like lspatches, which you would also do in any case, as one-off patches may have been installed on your homes and you would not have renamed the directory.
Having said that, you can name it as you wish :)
 
 

1/ Install GI 18c from the gold image

Let's enjoy this super new feature and quickly install GI 18c from a gold image: 
-- Create the target directories for GI 18c
 sudo su -
 dcli -g ~/dbs_group -l root "ls -ltr /u01/app/18.1.0.0/grid"
 dcli -g ~/dbs_group -l root mkdir -p /u01/app/18.1.0.0/grid
 dcli -g ~/dbs_group -l root chown grid:oinstall /u01/app/18.1.0.0/grid
 dcli -g ~/dbs_group -l root "ls -altr /u01/app/18.1.0.0/grid"
 
 -- Install GI using this gold image : /patches/V978971-01.zip
 sudo su - grid
 unzip -q /patches/V978971-01.zip -d /u01/app/18.1.0./grid
 

2/ Pre requisites

2.1/ Upgrade opatch

As usual, it is recommended to patch opatch before starting any patching activity. If you work with Exadata, you may have a look at this post where I show how to quickly upgrade opatch with dcli.
 

2.2/ ASM spfile and password file

Check that the ASM passwordfile and the ASM spfile are located under ASM to avoid issues during the upgrade: 
[grid@exadb01]$ asmcmd spget
 +DATA/mycluster/ASMPARAMETERFILE/registry.253.909449003
 [grid@exadb01]$ asmcmd pwget --asm
 +DATA/orapwASM
 [grid@exadb01]$
If you don't, you may face the below error and ASM won't restart after being upgraded: 
Verifying Verify that the ASM instance was configured using an existing ASM
 parameter file. ...FAILED
 PRCT-1011 : Failed to run "asmcmd". Detailed error:
 ASMCMD-8001: diskgroup 'u01' does not exist or is not mounted
Please find a quick procedure to move the ASM passwordfile from a FileSystem to ASM: 
[grid@exadb01]$ asmcmd pwcopy /u01/app/12.1.0.2/grid/dbs/orapw+ASM +DBFS_DG/orapwASM
 [grid@exadb01]$ asmcmd pwset --asm +DBFS_DG/orapwASM
 [grid@exadb01]$ asmcmd pwget --asm
 

2.3/ Prepare a responsefile such as this one

[grid@exadb01]$ egrep -v "^#|^$" /tmp/giresponse.rsp | head -10
 oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v18.1.0
 INVENTORY_LOCATION=
 oracle.install.option=UPGRADE
 ORACLE_BASE=/u01/app/oracle
 oracle.install.asm.OSDBA=dba
 oracle.install.asm.OSOPER=dba
 oracle.install.asm.OSASM=dba
 oracle.install.crs.config.gpnp.scanName=
 oracle.install.crs.config.gpnp.scanPort=
 oracle.install.crs.config.ClusterConfiguration=
 [grid@exadb01]$
 

2.4/ System prerequisites

Check these system prerequisites: 
-- a 10240 limits for the "soft stack" (if not, set it, log off and log on)
 [root@exadatadb01]# dcli -g ~/dbs_group -l root grep stack /etc/security/limits.conf | grep soft
 exadatadb01: * soft stack 10240
 exadatadb02: * soft stack 10240
 exadatadb03: * soft stack 10240
 exadatadb04: * soft stack 10240
 [root@exadatadb01]#
 
 -- at least 1500 huge pages free
 [root@exadatadb01]# dcli -g ~/dbs_group -l root grep -i huge /proc/meminfo
 ....
 AnonHugePages: 0 kB
 HugePages_Total: 200000
 HugePages_Free: 132171
 HugePages_Rsvd: 38338
 HugePages_Surp: 0
 Hugepagesize: 2048 kB
 ....
 [root@exadatadb01]#
 

2.5/ Run the prerequisites

This step is very important and the logs need to be checked closely for any error: 
[grid@exadb01]$ cd /u01/app/18.1.0.0/grid
 [grid@exadb01]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verbose
Below an output which requires a fixup: 
CVU operation performed: stage -pre crsinst
 Date: Jun 18, 2019 9:47:15 PM
 CVU home: /u01/app/18.1.0.0/grid/
 User: grid
 ******************************************************************************************
 Following is the list of fixable prerequisites selected to fix in this session
 ******************************************************************************************
 -------------- --------------- ------------- ------------- 
 Check failed. Failed on nodes Reboot Re-Login 
  required? required? 
 -------------- --------------- ------------- ------------- 
 Group Membership: asmoper exadb01,exadb02 no no 
 Execute "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" as root user on nodes "exadb01,exadb02" to perform the fix up operations manually
 Press ENTER key to continue after execution of "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" has completed on nodes "exadb01,exadb02"
Which I executed on both nodes (do not do that if your a fixup requires a reboot!): 
[root@exadb01 ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh
 All Fix-up operations were completed successfully.
 [root@exadb01 ~]# ssh exadb02
 [root@exadb02 ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh
 All Fix-up operations were completed successfully.
 [root@exadb02 ~]#
Once the fixup script has been executed, restart the prerequisites to be sure that everything is now good: 
[grid@exadb01]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verbose
 . . .
 Pre-check for cluster services setup was successful. 
 CVU operation performed: stage -pre crsinst
 Date: Jun 18, 2019 9:53:40 PM
 CVU home: /u01/app/18.1.0.0/grid/
 User: grid
 [grid@exadb01]$
 

3/ Upgrade to GI 18c

Now that all the prerequisites are successful, we can upgrade GI to 18c.

3.0/ A status before starting the upgrade

I strongly recommend keeping a status of all the resources across your cluster before starting the maintenance to avoid any unpleasantness after the maintenance. 
[grid@exadb01]$ ./rac-status.sh -a -w0 | tee status_before_GI_upgrade_to_18
 
  Cluster exadata is a X5-2 Elastic Rack HC 8TB
 
  Listener | Port | db01 | db02 | db03 | db04 | Type |
 -------------------------------------------------------------------------------------------------------------------
  LISTENER | TCP:1551 | Online | Online | Online | Online | Listener |
  LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener |
  LISTENER_SCAN1| TCP:1551,1561 | - | - | Online | - | SCAN |
  LISTENER_SCAN2| TCP:1551,1561 | - | Online | - | - | SCAN |
  LISTENER_SCAN3| TCP:1551,1561 | Online | - | - | - | SCAN |
 -------------------------------------------------------------------------------------------------------------------
 
  DB | Service | db01 | db02 | db03 | db04 |
 ----------------------------------------------------------------------------------------------------
  db01 | proddb_1_bkup | Online | - | - | - |
  | proddb_2_bkup | - | Online | - | - |
  | proddb_3_bkup | - | - | Online | - |
  | proddb_4_bkup | - | - | - | Online |
  db02 | db02svc1_bkup | - | - | Online | - |
  | db02svc2_bkup | - | - | Online | - |
  db03 | db03svc1_bkup | Online | - | - | - |
  | db03svc2_bkup | Online | - | - | - |
  db04 | db04svc1_bkup | Online | - | - | - |
  | db04svc2_bkup | - | Online | - | - |
  | db04svc3_bkup | - | - | Online | - |
  | db04svc4_bkup | - | - | - | Online |
 ----------------------------------------------------------------------------------------------------
 
  DB | Version | db01 | db02 | db03 | db04 | DB Type |
 -------------------------------------------------------------------------------------------------------------------
  db01 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) |
  db02 | 12.1.0.2 (1) | - | - | Open | Open | RAC (P) |
  db03 | 12.1.0.2 (1) | Open | Open | - | - | RAC (P) |
  db04 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) |
 -------------------------------------------------------------------------------------------------------------------
 
  ORACLE_HOME references listed in the Version column : Primary : White and (P)
  Standby : Red and (S)
  1 : /u01/app/oracle/product/12.1.0.2/dbhome_1
 [grid@exadb01]$

3.1/ ASM memory setting

Some recommended memory settings have to be set at ASM instance level: 
[grid@exadb01]$ sqlplus / as sysasm
 SQL> alter system set sga_max_size = 3G scope=spfile sid='*';
 SQL> alter system set sga_target = 3G scope=spfile sid='*';
 SQL> alter system set memory_target=0 sid='*' scope=spfile;
 SQL> alter system set memory_max_target=0 sid='*' scope=spfile /* required workaround */;
 SQL> alter system reset memory_max_target sid='*' scope=spfile;
 SQL> alter system set use_large_pages=true sid='*' scope=spfile /* 11.2.0.2 and later(Linux only) */;
 

3.2/ Reset miscount to default

The miscount parameter is the maximum time, in seconds, that a network heartbeat can be missed before a node eviction occurs. It needs to be reset to default before upgrading. It has to be done as the GI owner. 
[grid@exadb01]$ . oraenv <<< +ASM1
 [grid@exadb01]$ crsctl unset css misscount
 

3.3/ gridSetup.sh

We will be using a script named gridSetup.sh to initiate the GI upgrade to 18c Please find below a whole output: 
[grid@exadb01]$ cd /u01/app/18.1.0.0/grid
 [grid@exadb01]$ ./gridSetup.sh -silent -responseFile /tmp/giresponse.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.crs.enableRemoteGIMR=false -applyRU /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717
 Preparing the home to patch...
 Applying the patch /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717...
 Successfully applied the patch.
 The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/installerPatchActions_2018-11-18_05-11-43PM.log
 Launching Oracle Grid Infrastructure Setup Wizard...
 . . .
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/gridSetupActions2018-11-18_05-11-43PM.log
 
 As a root user, execute the following script(s):
  1. /u01/app/18.1.0.0/grid/rootupgrade.sh
 
 Execute /u01/app/18.1.0.0/grid/rootupgrade.sh on the following nodes:
 [exadb01, exadb02, exadb04, exadb03]
 
 Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.
 
 Successfully Setup Software.
 As install user, execute the following command to complete the configuration.
  /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp [-silent]
 
 [grid@exadb01]$
Above are some ignorable warnings about OS groups. Is also described the next step which is to start rootupgrade.sh on each node.
 

3.4/ rootupgrade.sh

As specified by gridSetup.sh in the previous step, we now need to run rootupgrade.sh on each node knowing that you can start rootupgrade.sh in parallel except for the first and the last node. Below is an example with a half rack (four nodes):
  1. Start rootupgrade.sh on the node 1
  2. Start rootupgrade.sh in parallel on the nodes 2 and 3
  3. Start rootupgrade.sh on the node 4
Note that CRS will be stopped on the node you apply rootupgrade.sh on so your instances will suffer an outage during this operation. Think about rebalancing your services accordingly to avoid any application downtime. Here is a sample output; note that rootupgrade.sh is very silent, all logs go to the log file specified: 
[root@exadb01]# /u01/app/18.1.0.0/grid/rootupgrade.sh
 Check /u01/app/18.1.0.0/grid/install/root_exadb01_2019-06-29_11-14-09-408446489.log for the output of root script
 [root@exadb01]#
An interesting thing to note here after a node is patched is that the softwareversion is now the target one (18.5 and not that here CRS shows 18.0.0.0.0) but the activeversion is still the old one (12.1). The activeversion will be changed to 18.5 when applying rootupgrade.sh on the last node. 
[root@exadb01]# . oraenv <<< +ASM1
 [root@exadb01]# crsctl query crs softwareversion
 Oracle Clusterware version on node [exadb01] is [18.0.0.0.0]
 [root@exadb01]# crsctl query crs activeversion
 Oracle Clusterware active version on the cluster is [12.1.0.2.0]
 [root@exadatadb01]#
 

3.5/ gridSetup.sh -executeConfigTools

Run the gridSetup.sh -executeConfigTools command: 
[grid@exadb01]$ /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp -silent
 Launching Oracle Grid Infrastructure Setup Wizard...
 
 You can find the logs of this session at:
 /u01/app/oraInventory/logs/GridSetupActions2018-11-18_07-11-22PM
 
 Successfully Configured Software.
 [grid@exadb01]$
 

3.6/ Check that GI is relinked with RDS

It is worth double checking that the new GI Home is properly relinked with RDS to avoid future performance issues (you may want to read this pdf for more information on what RDS is): 
[grid@exadb01]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/skgxpinfo
 exadatadb01: rds
 exadatadb02: rds
 exadatadb03: rds
 exadatadb04: rds
 [grid@exadb01]$
If not, relink the GI Home with RDS: 
dcli -g ~/dbs_group -l oracle "ORACLE_HOME=/u01/app/18.1.0.0/grid; make -C /u01/app/18.1.0.0/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle"
 

3.7/ Check the status of the cluster

Let's have a look at the status of the cluster and the activeversion: 
[grid@exadb01]$ /u01/app/18.1.0.1/grid/bin/crsctl check cluster -all
 **************************************************************
 exadatadb01:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************
 exadatadb02:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************
 exadatadb03:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************
 exadatadb04:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************
 [grid@exadb01]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/crsctl query crs activeversion
 Oracle Clusterware version on node [exadb01] is [18.0.0.0.0]
 Oracle Clusterware version on node [exadb02] is [18.0.0.0.0]
 Oracle Clusterware version on node [exadb03] is [18.0.0.0.0]
 Oracle Clusterware version on node [exadb04] is [18.0.0.0.0]
 [grid@exadb01]$
Let's check the status of all the resources like we did in paragraph 3.0: 
[grid@exadb01]$ ./rac-status.sh -a -w0 | tee status_after_GI_upgrade_to_18
 
  Cluster exadata is a X5-2 Elastic Rack HC 8TB
 
  Listener | Port | db01 | db02 | db03 | db04 | Type |
 -------------------------------------------------------------------------------------------------------------------
  LISTENER | TCP:1551 | Online | Online | Online | Online | Listener |
  LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener |
  LISTENER_SCAN1| TCP:1551,1561 | - | Online | - | - | SCAN |
  LISTENER_SCAN2| TCP:1551,1561 | Online | - | - | - | SCAN |
  LISTENER_SCAN3| TCP:1551,1561 | - | - | Online | - | SCAN |
 -------------------------------------------------------------------------------------------------------------------
 
  DB | Service | db01 | db02 | db03 | db04 |
 ----------------------------------------------------------------------------------------------------
  db01 | proddb_1_bkup | Online | - | - | - |
  | proddb_2_bkup | - | Online | - | - |
  | proddb_3_bkup | - | - | Online | - |
  | proddb_4_bkup | - | - | - | Online |
  db02 | db02svc1_bkup | - | - | Online | - |
  | db02svc2_bkup | - | - | Online | - |
  db03 | db03svc1_bkup | Online | - | - | - |
  | db03svc2_bkup | Online | - | - | - |
  db04 | db04svc1_bkup | Online | - | - | - |
  | db04svc2_bkup | - | Online | - | - |
  | db04svc3_bkup | - | - | Online | - |
  | db04

No Comments Yet

Let us know what you think

See all

See all

Subscribe by email
Follow us
Love Your Data
Pythian Logo

© 2024 All rights reserved.