Oracle-Data-Guard-12c-Redo-Gap-Resolutions-Part-I (1)

Note: On the primary database during redo transport LWGR, LNS,NSS, ARCH after Oracle Database 11g R2 for best practice corruption detection/protection parameter is enabled using DB_ULTRA_SAFE =TRUE. On the standby database during redo apply RFS, ARCH, MRP, DBWR after Oracle Database 11g R2 for best practice corruption detection/prevention parameters is enabled using DB_BLOCK_CHECKSUM=FULL and DB_LOST_WRITE_PROTECT=TYPICAL.

Demo 2:

Redo Gap Resolution using FAL: In this case we corrupted and removed some “not applied” archived redo logs in standby database server and analyzing alert logs for behavior databases in this case.

1. Stop Redo Apply process on standby database.

[oracle@oel62-stbdb-12c ~]$ export ORACLE_SID=stbcdb [oracle@oel62-stbdb-12c ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Thu Aug 1 16:22:27 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> alter database recover managed standby database cancel;

Database altered.

2.Transport is running and sending redo to standby database.

[oracle@oel62-stbdb-12c ~]$ export ORACLE_SID=stbcdb [oracle@oel62-stbdb-12c ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Thu Aug 1 16:24:36 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select max(sequence#) from v$archived_log;

MAX(SEQUENCE#)

--------------------------

261

SQL> select max(sequence#) from v$archived_log where applied='YES';

11 Oracle Database 12c - Redo Gap Resolutions (Part I)

Joel Perez (Oracle ACED), Mahir M. Quluzade (OCP) & Carlos H. Y. Furushima (OCE)

MAX(SEQUENCE#)

--------------------------

258

Standby database received additionally 3 archived log file from primary. Redo Apply process stopped, this 3 archived log file is not applied to standby database.

3. Remove and make corrupt not applied archived redo log files.

[oracle@oel62-stbdb-12c ~]$ cd /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/ [oracle@oel62-stbdb-12c 2014_08_01]$ ls -l

[oracle@oel62-stbdb-12c 2014_08_01]$ rm -fr o1_mf_1_259_8znkb877_.arc [oracle@oel62-stbdb-12c 2014_08_01]$ cat >> o1_mf_1_260_8znkbbyp_.arc Corruption on 260 sequence#

[oracle@oel62-stbdb-12c 2014_08_01]$ cat o1_mf_1_260_8znkbbyp_.arc

" eX}|{z " 0 PRMCDB \ Thread 0001, Seq# 0000000260, SCN 0x0000002316130x00000023161aL 0?# 1# 1?L 0# 1 ! /1 0 ڃ' * Y" * Y <O ; 1 " h# -#)47# 1k k k " p h"#D\Q-# 1k k k Corruption on 260 sequence#

Removed archive log sequence is 259 and corrupted archived log sequence is 260.

4. Start real-time redo apply process on standby database.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

Alert log on standby database:

…

MRP0: Background Managed Standby Recovery process started (stbcdb) Thu Aug 01 16:32:14 2014

Serial Media Recovery started

Managed Standby Recovery starting Real Time Apply Thu Aug 01 16:32:16 2014

Waiting for all non-current ORLs to be archived...

12 Oracle Database 12c - Redo Gap Resolutions (Part I)

Joel Perez (Oracle ACED), Mahir M. Quluzade (OCP) & Carlos H. Y. Furushima (OCE)

Thu Aug 01 16:32:16 2014

All non-current ORLs have been archived. Thu Aug 01 16:32:16 2014

Media Recovery Log /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_259_8znkb877_.arc

Error opening /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_259_8znkb877_.arc Attempting refetch

Media Recovery Waiting for thread 1 sequence 259

Fetching gap sequence in thread 1, gap sequence 259-259

Completed: alter database recover managed standby database disconnect from session Thu Aug 01 16:32:17 2014

RFS[6]: Assigned to RFS process (PID:4915)

RFS[6]: Allowing overwrite of partial archivelog for thread 1 sequence 259

RFS[6]: Opened log for thread 1 sequence 259 dbid 2485119180 branch 820252236

Thu Aug 01 16:32:19 2014

Archived Log entry 12 added for thread 1 sequence 259 rlc 820252236 ID 0x94205ccc dest 2: Thu Aug 01 16:32:20 2014

Media Recovery Log /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_259_8znkt266_.arc

Thu Aug 01 16:32:21 2014 Media Recovery Log

/u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_260_8znkbbyp_.arc

Error opening /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_260_8znkbbyp_.arc Attempting refetch

Media Recovery Waiting for thread 1 sequence 260

Fetching gap sequence in thread 1, gap sequence 260-260

Thu Aug 01 16:32:21 2014

RFS[6]: Allowing overwrite of partial archivelog for thread 1 sequence 260

RFS[6]: Opened log for thread 1 sequence 260 dbid 2485119180 branch 820252236

Thu Aug 01 16:32:22 2014

Archived Log entry 13 added for thread 1 sequence 260 rlc 820252236 ID 0x94205ccc dest 2: Thu Aug 01 16:32:22 2014

Media Recovery Log /u01/app/oracle/fast_recovery_area/STBCDB/archivelog/2014_08_01/o1_mf_1_260_8znkt5t8_.arc

Thu Aug 01 16:32:24 2014

…

As it is seen from alert log, when media recovery process can’t open archived log, then fetching gap sequence. RFS request to primary for gap sequence. If there have archived log in primary side, then RFS allowing overwrite of partial archived log, primary ARCH process sending archive log again.

There have two nuances redo gap resolution process. When occurs archive gap on standby database, before connection between primary and standby database restore, may be not shipped archived redo logs of standby database is removed or corrupted on primary side.

We discussed this nuances in second part of this article. We invite read second part of this article.

References:

Oracle Database Backup and Recovery User's Guide 12c Release 1 (12.1)

13 Oracle Database 12c - Redo Gap Resolutions (Part I)

Joel Perez (Oracle ACED), Mahir M. Quluzade (OCP) & Carlos H. Y. Furushima (OCE)

Oracle Data Guard Concepts and Administration 12c Release 1 (12.1)

14 Oracle Database 12c - Redo Gap Resolutions (Part I)

Joel Perez (Oracle ACED), Mahir M. Quluzade (OCP) & Carlos H. Y. Furushima (OCE)