How to Resize Exadata Grid Disks Without an Outage

As you might know, each Exadata cell contains 12 physical disks with a fixed size. If you want to increase the ASM disk group size, you may need to purchase additional cell servers, which is expensive. In this blog, I’ll show you how to resize the grid disks and move the space between different ASM disk groups.

If you’ve previously deployed Exadata, it’s likely that two ASM disk groups were created. One is +DATA which is for your database data, the other is +RECO, which is for the database archive logs. As you continued to use your system and your data grew, you might have found yourself needing more space in the disk group +DATA but less space in +RECO and wondered if you could transfer space from the latter to the former.

The answer is yes.

Fortunately, you can do it without a database or Exadata Machine outage because it’s done online. Keep in mind that disk rebalancing operations always affect IO performance and this should be taken into consideration.

Before we go further let’s understand the relationship between Exadata physical disks, LUN, cell disks, grid disks and ASM disks.

Disk & LUN: Each Exadata cell contains 12 physical disks and each physical disk is mapped to a logical abstraction called a Logical Unit (LUN). The disk size can vary; in X2 it’s 600 GB, in X9 it’s 18000 GB.
CellDisk: A cell disk is a higher-level abstraction that represents the data storage area on each LUN.
GridDisk: After a cell disk is created, it can be subdivided into one or more grid disks, which are directly exposed to ASM.
ASM Disk: Disks used to create the disk groups.

Now that we understand the disk relationship, we know that we need to resize the Grid Disk first if we want to move the space from the disk group +RECO. We’ve done this (moved from X2 to X8M) many times for our clients—here, the steps are the same.

Here’s an example in X2 Exadata.

Look at the disk group size before we move the space from +RECO to +DATA. As you can see, +DATA has a total of 35532 GB; 34090 GB are in use and only 509 GB are free. However +RECO has 8877 GB in total, with 760 GB in use and 4005 GB of free space:

SQL> select group_number,name,STATE,type,total_mb/1024 "TOTAL_GB", REQUIRED_MIRROR_FREE_MB/1024 "MIRROR GB",(TOTAL_MB-FREE_MB)/1024 "USED_GB",FREE_MB/1024 "FREE_GB", trunc(free_mb/1024 - required_mirror_free_mb/1024)/(case type when 'HIGH' then 3 when 'NORMAL' then 2 else 1 end) "USEABLE_GB", (TOTAL_MB-FREE_MB)/total_mb*100 "USAGE(%)", DATABASE_COMPATIBILITY, COMPATIBILITY from v$asm_diskgroup;

GROUP_NUMBER NAME STATE TYPE TOTAL_GB MIRROR GB USED_GB FREE_GB USEABLE_GB USAGE(%) DATABASE_C COMPATIBIL
------------ ------------------------------ --------------------------------- ------------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 DATA MOUNTED NORMAL 35532 423 34090.1445 1441.85547 509 95.9420931 11.2.0.2.0 18.7.0.0.0
2 RECO MOUNTED NORMAL 8877.75 105.6875 760.671875 8117.07813 4005.5 8.56829574 11.2.0.2.0 18.7.0.0.0

Most cell disks have around 558 GB with no free space. However, the first 2 cell disks have 528 GB only. That’s because they contain a system area that spans multiple disk partitions; there are approximately 29 GB on each disk:

[root@exa1db01 ~]# dcli -g ~/cell_group -l root "cellcli -e list celldisk where name like \'CD.*\' attributes name,size, freespace"
exa1cel01: CD_00cel01 528.734375G 0
exa1cel01: CD_01cel01 528.734375G 0
...
exa1cel01: CD_09cel01 557.859375G 0
exa1cel01: CD_10cel01 557.859375G 0
...
exa1cel07: CD_00cel07 528.734375G 0
exa1cel07: CD_01cel07 528.734375G 0
...
exa1cel07: CD_10cel07 557.859375G 0
exa1cel07: CD_11cel07 557.859375G 0

In each cell disk, 423 GB have been assigned to DATA grid disks:

[root@exa1db01 ~]# dcli -g ~/cell_group -l root "cellcli -e list griddisk where name like \'DATA.*\' attributes name, size"
exa1cel01: DATA_CD_00cel01 423G
...
exa1cel01: DATA_CD_08cel01 423G
exa1cel01: DATA_CD_09cel01 423G
...
exa1cel07: DATA_CD_00cel07 423G
exa1cel07: DATA_CD_01cel07 423G
...
exa1cel07: DATA_CD_10cel07 423G
exa1cel07: DATA_CD_11cel07 423G

Now, 105 GB have been assigned to RECO grid disks:

[root@exa1db01 ~]# dcli -g ~/cell_group -l root "cellcli -e list griddisk where name like \'RECO.*\' attributes name, size"
exa1cel01: RECO_CD_00cel01 105.6875G
exa1cel01: RECO_CD_01cel01 105.6875G
...
exa1cel01: RECO_CD_08cel01 105.6875G
exa1cel01: RECO_CD_09cel01 105.6875G
...
exa1cel07: RECO_CD_00cel07 105.6875G
exa1cel07: RECO_CD_01cel07 105.6875G
...
exa1cel07: RECO_CD_10cel07 105.6875G
exa1cel07: RECO_CD_11cel07 105.6875G

Now, I’ll resize the RECO grid disks to 25 GB so I can move 80 GB from each RECO grid disk to the DATA grid disk. Here are the commands to resize the grid disks on the first cell storage. You’ll need to repeat this step for the other cells:

[root@exa1cel01 ~]# cellcli -e alter griddisk RECO_CD_01cel01,RECO_CD_02cel01,RECO_CD_03cel01,RECO_CD_04cel01,RECO_CD_05cel01,RECO_CD_06cel01,RECO_CD_07cel01,RECO_CD_08cel01,RECO_CD_09cel01,RECO_CD_10cel01,RECO_CD_11cel01 size=25600M;
GridDisk RECO_CD_01cel01 successfully altered
GridDisk RECO_CD_02cel01 successfully altered
...
GridDisk RECO_CD_10cel01 successfully altered
GridDisk RECO_CD_11cel01 successfully altered

[root@exa1cel01 ~]# cellcli -e list griddisk where name like \'RECO.*\' attributes name, size 
RECO_CD_00cel01 25G
RECO_CD_01cel01 25G
...
RECO_CD_10cel01 25G
RECO_CD_11cel01 25G

[root@exa1cel01 ~]# cellcli -e list celldisk where name like \'CD.*\' attributes name,size, freespace
CD_00cel01 528.734375G 80.6875G
CD_01cel01 528.734375G 80.6875G
...
CD_10cel01 557.859375G 80.6875G
CD_11cel01 557.859375G 80.6875G

After we resize all RECO grid disks, we can move all free space from each RECO grid disk to the DATA grid disk. The calculation is 423 GB + 80.6875 G = 503.6875 G. This step needs to be repeated in other cells as well:

[root@exa1cel01 ~]# cellcli -e alter griddisk DATA_CD_00cel01,DATA_CD_01cel01,DATA_CD_02cel01,DATA_CD_03cel01,DATA_CD_04cel01,DATA_CD_05cel01,DATA_CD_06cel01,DATA_CD_07cel01,DATA_CD_08cel01,DATA_CD_09cel01,RECO_CD_10cel01,RECO_CD_11cel01 size=503.6875G
GridDisk DATA_CD_00cel01 successfully altered
GridDisk DATA_CD_01cel01 successfully altered
...
GridDisk DATA_CD_10cel01 successfully altered
GridDisk DATA_CD_11cel01 successfully altered

[root@exa1cel01 ~]# cellcli -e list griddisk where name like \'DATA.*\' attributes name, size
DATA_CD_00cel01 503.6875G
DATA_CD_01cel01 503.6875G
...
DATA_CD_10cel01 503.6875G
DATA_CD_11cel01 503.6875G

After we resize all DATA grid disks, from ASM we can see the disk group +RECO has only 2100 GB in total and total usage is still 760 GB with 657 GB of free space (vs. 8877 GB total with 760 GB in use and 4005 GB free):

GROUP_NUMBER NAME STATE TYPE TOTAL_GB MIRROR GB USED_GB FREE_GB USEABLE_GB USAGE(%) DATABASE_C COMPATIBIL
------------ ------------------------------ ----------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 DATA MOUNTED NORMAL 35532 423 34162.1758 1369.82422 473 96.1448153 11.2.0.2.0 18.7.0.0.0
2 RECO MOUNTED NORMAL 2100 25 760.671875 1339.32813 657 36.2224702 11.2.0.2.0 18.7.0.0.0

Next, let’s resize the disk group +DATA to maximize its capacity:

SQL> conn / as sysasm
SQL> alter diskgroup DATA resize all rebalance power 32;

After rebalancing, we can see the disk group +DATA space have been increased. The total size is 42309 GB, with  34162 GB in use and 3821 GB free (vs. 35532 GB total, 34090 GB in use and only 509 GB of free):

GROUP_NUMBER NAME STATE TYPE TOTAL_GB MIRROR GB USED_GB FREE_GB USEABLE_GB USAGE(%) DATABASE_C COMPATIBIL
------------ ------------------------------ ----------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 DATA MOUNTED NORMAL 42309.75 503.6875 34162.1758 8147.57422 3821.5 80.7430339 11.2.0.2.0 18.7.0.0.0
2 RECO MOUNTED NORMAL 2100 25 760.671875 1339.32813 657 36.2224702 11.2.0.2.0 18.7.0.0.0

I hope you enjoyed this post. Feel free to ask me questions in the comments.

If you haven’t already, sign up for my next post here.