ORA-16069? You May Need A New Standby Controlfile

On a recent Monday, I had to perform an emergency Oracle standby switchover for a client whose primary instance host had mysteriously rebooted itself over the previous day. Confidence in that host was, understandably, shaken.

The Oracle Data Guard configuration is a 3-instance setup using Data Guard Broker: one primary, we’ll call it OraA, feeding two standby instances, OraB and OraC. In this particular configuration, we perform switchovers between OraA and OraC. Caught in the middle is OraB, which is on a 60-minute standby delay.

After this particular switchover, OraB started complaining with this message in the alert log:

ORA-16069: Archive Log standby database activation identifier mismatch

We had seen this occasionally in prior switchovers, and the problem would fix itself once the standby delay passed and the OraB standby would process the log notifying it of the switchover. This time, however, recovery was stopped and more than enough time had elapsed. OraA and OraC were performing perfectly fine.

Much of the reference searching I saw suggested that the standby instance would have to be completely rebuilt. Not an appetizing option. A search of metalink turned up Bug 4048687, which seemed to demonstrate a similar problem, although on a different OS/Platform. That solution was to recreate the standby controlfile. Trust me, it sounds more drastic than it is!

Here’s how to do it in just 6 easy steps!

1. Shutdown the misbehaving standby.
2. Copy one of the current standby controlfiles for safekeeping (just in case).
3. On the primary instance, create a new standby controlfile: alter database create standby controlfile as '/tmp/stdby.ctl';
4. Transfer that new standby controlfile to the standby host.
5. Copy the new controlfile to the controlfile location(s) used by the instance (you have more than one, right?).
6. STARTUP MOUNT the standby instance. If you use the Data Guard Broker, it should automatically begin recovery for you; otherwise restart managed recovery with alter database recover managed standby database disconnect;

Voila. Standby recovery should resume nicely, assuming logs are there to apply.