Troubleshooting RAC GoldenGate relocation

by Michael Dinh on Aug 17, 2018 12:00:00 AM

During RAC online patching, GoldenGate resource is required to be relocated to surviving node. When relocation fails, there are a few logs which can be useful for troubleshooting. Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware (XAG) was not installed and using existing version from $GRID_HOME. Unfortunately, I don't have the exact version and would suspect the version is probably same as:

[oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/agctl query releaseversion
 The Oracle Grid Infrastructure Agents release version is 3.1.0
 
 [oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/srvctl -V
 srvctl version: 12.1.0.2.0
 [oracle@racnode-dc1-1 ~]$

The name of the GoldenGate instance/resource (instance_name): gg_xx GoldenGate resources can be determined using the following command:

$GRID_HOME/bin/crsctl stat res -t|grep -A2 xag
 xag.gg_xx-vip.vip
  1 ONLINE ONLINE racnode-dc1-2 STABLE
 xag.gg_xx.goldengate
  1 ONLINE ONLINE racnode-dc1-2 STABLE

Here is the output for GoldenGate relocation and can be performed from any node.

$GRID_HOME/bin/agctl relocate goldengate gg_xx --node racnode-dc1-1
 
 CRS-2673: Attempting to stop 'xag.gg_xx.goldengate' on 'racnode-dc1-2'
 CRS-2677: Stop of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded
 CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-2'
 CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded
 CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-1'
 CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded
 CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-1'
 CRS-2674: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' failed
 CRS-2679: Attempting to clean 'xag.gg_xx.goldengate' on 'racnode-dc1-1'
 CRS-2681: Clean of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' succeeded
 
 CRS-2564: Failed to relocate resource 'xag.gg_xx.goldengate'. Will attempt to restore it on 'racnode-dc1-2' now.
 
 CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-1'
 CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded
 CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-2'
 CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded
 CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-2'
 CRS-2676: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded
 CRS-4000: Command Relocate failed, or completed with errors.
 
 $GRID_HOME/bin/agctl status goldengate gg_xx
 Goldengate instance 'gg_xx' is running on racnode-dc1-2

Here are the logs to check in order of personal preference. Check GoldenGate Error Log: $GG_HOME/ggserr.log Check XAG Agent trace file: GoldenGate user is ggsuser $ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/crsd_scriptagent_ggsuser.trc Check alert.log for crs: $ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/alert.log In summary, there are multiple logs to check and when checking the right one first will make troubleshooting more efficient.

No Comments Yet

Let us know what you think

See all

Share this

Troubleshooting RAC GoldenGate relocation

Share this

No Comments Yet

Subscribe by email