TABLE ACCESS FULL on NULL column

TABLE ACCESS FULL on NULL column

Issue :

We had a new query performing a join on few tables and users were reporting performance issue.

Upon checking the plan, we found that the query was doing a full table scan on one of the tables.

Here’s the execution plan.

Execution Plan

----------------------------------------------------------

Plan hash value: 2413029803

------------------------------------------------------------------------------------------------------------------------------------

| Id  | Operation                               | Name                     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |

------------------------------------------------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                        |                          |     1 |   127 | 36234   (1)| 00:07:15 |       |       |

|   1 |  SORT ORDER BY                          |                          |     1 |   127 | 36234   (1)| 00:07:15 |       |       |

|*  2 |   HASH JOIN OUTER                       |                          |     1 |   127 | 36233   (1)| 00:07:15 |       |       |

|   3 |    NESTED LOOPS                         |                          |       |       |            |          |       |       |

|   4 |     NESTED LOOPS                        |                          |     1 |   106 |    27   (0)| 00:00:01 |       |       |

|   5 |      PARTITION RANGE ALL                |                          |     1 |    97 |    26   (0)| 00:00:01 |     1 |    24 |

|   6 |       PARTITION HASH SINGLE             |                          |     1 |    97 |    26   (0)| 00:00:01 |     2 |     2 |

|   7 |        TABLE ACCESS BY LOCAL INDEX ROWID| SELLER_OFFERING          |     1 |    97 |    26   (0)| 00:00:01 |       |       |

|*  8 |         INDEX RANGE SCAN                | IX1_SELLER_CONFIGURATION |     1 |       |    25   (0)| 00:00:01 |       |       |

|*  9 |      INDEX UNIQUE SCAN                  | PK_SALE_CONFIGURATION    |     1 |       |     0   (0)| 00:00:01 |       |       |

|  10 |     TABLE ACCESS BY INDEX ROWID         | SALE_CONFIGURATION       |     1 |     9 |     1   (0)| 00:00:01 |       |       |

|  11 |    PARTITION RANGE ALL                  |                          |  7152K|   143M| 36184   (1)| 00:07:15 |     1 |    24 |

|  12 |     PARTITION HASH ALL                  |                          |  7152K|   143M| 36184   (1)| 00:07:15 |     1 |     8 |

|  13 |      TABLE ACCESS FULL                  | PURCHASE                 |  7152K|   143M| 36184   (1)| 00:07:15 |     1 |   192 |

------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------

   2 - access("PR"."SELLER_OFFERING_ID"(+)="SO"."SELLER_OFFERING_ID")

   8 - access("SO"."SERVICE_ORDER_ID"=10145663)

   9 - access("SC"."SALE_CONFIGURATION_ID"="SO"."SALE_CONFIGURATION_ID")

We found that there was a newly added column on the PURCHASE table and it is not NOT NULL.

Upon investigating, we came to know that 99% of the existing records were null on this field. The field will be getting records going forward.

There was a btree index on the column but it was not being used by the optimizer.

So, the join was on a column whose majority of the records were NULL as of now and hence the full table scans.

Resolution:

There were some debates on what is the best way to address this. Can we update the field with some default value and then the index will come into play? There were few other suggestions. Well, one workaround worked.

As a temporary solution, I dropped the regular index on the column and created a function based index using NVL function.

The NVL function replaces a NA value or an empty string with a string.

Syntax

NVL (exp , replacement-exp)

SQL> drop index APPL.IX4_PURCHASE;

Index dropped.

SQL> create index APPL.IX4_PURCHASE ON APPL.PURCHASE nvl(SELLER_OFFERING_ID,null);

Index created.

This is the new execution plan and the query executes in sub seconds. The function based index comes into play.

Execution Plan

----------------------------------------------------------

Plan hash value: 3975705781

-----------------------------------------------------------------------------------------------------------------------------------

| Id  | Operation                              | Name                     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |

-----------------------------------------------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                       |                          |     1 |   127 |   129   (1)| 00:00:02 |       |       |

|   1 |  SORT ORDER BY                         |                          |     1 |   127 |   129   (1)| 00:00:02 |       |       |

|   2 |   NESTED LOOPS OUTER                   |                          |     1 |   127 |   128   (0)| 00:00:02 |       |       |

|   3 |    NESTED LOOPS                        |                          |     1 |   106 |    27   (0)| 00:00:01 |       |       |

|   4 |     PARTITION RANGE ALL                |                          |     1 |    97 |    26   (0)| 00:00:01 |     1 |    24 |

|   5 |      PARTITION HASH SINGLE             |                          |     1 |    97 |    26   (0)| 00:00:01 |     2 |     2 |

|   6 |       TABLE ACCESS BY LOCAL INDEX ROWID| SELLER_OFFERING          |     1 |    97 |    26   (0)| 00:00:01 |       |       |

|*  7 |        INDEX RANGE SCAN                | IX1_SELLER_CONFIGURATION |     1 |       |    25   (0)| 00:00:01 |       |       |

|   8 |     TABLE ACCESS BY INDEX ROWID        | SALE_CONFIGURATION       |     1 |     9 |     1   (0)| 00:00:01 |       |       |

|*  9 |      INDEX UNIQUE SCAN                 | PK_SALE_CONFIGURATION    |     1 |       |     0   (0)| 00:00:01 |       |       |

|  10 |    TABLE ACCESS BY GLOBAL INDEX ROWID  | PURCHASE                 |     1 |    21 |   101   (0)| 00:00:02 | ROWID | ROWID |

|* 11 |     INDEX RANGE SCAN                   | IX4_PURCHASE             |   105 |       |     1   (0)| 00:00:01 |       |       |

-----------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------

   7 - access("SO"."SERVICE_ORDER_ID"=10145663)

   9 - access("SC"."SALE_CONFIGURATION_ID"="SO"."SALE_CONFIGURATION_ID")

  11 - access("PR"."SELLER_OFFERING_ID"(+)="SO"."SELLER_OFFERING_ID")

I’m wondering if there could be noticeable performance issue for dml’s. So far I have not heard of any issues. We may move away from the function based index. But so far, it did the job.

There could be other better solutions and I’d like to hear them all. Thanks.

Issue during RMAN duplicate database; ORA-17628: Oracle error 19505 returned by remote Oracle server

Issue during RMAN duplicate database;

ORA-17628: Oracle error 19505 returned by remote Oracle server

Error :

The following error occurred when attempting to duplicate database from one RAC cluster to another.

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35

ORA-17628: Oracle error 19505 returned by remote Oracle server

continuing other job steps, job failed will not be re-run

output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:28

output file name=/u31_ARCHIVE/MAINS/5_518_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:26

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 11/09/2013 02:37:00

RMAN-05501: aborting duplication of target database

RMAN-03015: error occurred in stored script Memory Script

Version:

11.2.0.2

Duplicate Process:

Parameter File

> cat initMAINS.ora

DB_NAME='MAINS'

diagnostic_dest='/u01/app/oracle'

DB_FILE_name_CONVERT=('/u10_MAINS/oradata','/u10_MAINS/oradata')

LOG_FILE_NAME_CONVERT=('/u20_MAINS/redo1','/u20_MAINS/redo1','/u21_MAINS/redo2','/u21_MAINS/redo2')

SGA_TARGET=1G

CONTROL_FILES='/u20_MAINS/redo1/control01.ctl','/u20_MAINS/redo1/control02.ctl','/u21_MAINS/redo2/control03.ctl'

COMPATIBLE= 11.2.0.2

cluster_database=false

log_archive_dest_1='LOCATION=/u30_ARCHIVE/MAINS/'

log_archive_dest_2='LOCATION=/u31_ARCHIVE/MAINS/'

local_listener='(address=(protocol=tcp)(host=myracn01v.mycomp.com)(port=1527))'

Static listener configuration

LISTENER_MAINS =

  (DESCRIPTION_LIST =

    (DESCRIPTION =

      (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_MAINS))

      (ADDRESS = (PROTOCOL = TCP)(HOST = myracn01v.mycomp.com)(PORT = 1527))

    )

  )

SID_LIST_LISTENER_MAINS =

  (SID_LIST =

    (SID_DESC =

      (GLOBAL_DBNAME = MAINS)

      (ORACLE_HOME = /u01/app/oracle/product/11.2.0.2)

      (SID_NAME = MAINS)

    )

  )

Added oratab entry, set the environment and started the instance in nomount

SQL> startup nomount

ORACLE instance started.

Total System Global Area 1068937216 bytes

Fixed Size                  2233336 bytes

Variable Size             281021448 bytes

Database Buffers          775946240 bytes

Redo Buffers                9736192 bytes

RMAN Duplication

> rman

Recovery Manager: Release 11.2.0.2.0 - Production on Sat Nov 9 02:24:15 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prodblr10v.mycomp.com)(PORT=1561)))(CONNECT_DATA=(SID=PBPROD5)))

target database Password:

connected to target database: PBPROD (DBID=2131291108)

RMAN> connect auxiliary sys@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=myracn01v.mycomp.com)(PORT=1527)))(CONNECT_DATA=(SID=MAINS)))

auxiliary database Password:

connected to auxiliary database: MAINS (not mounted)

RMAN> run

{

CONFIGURE DEVICE TYPE disk PARALLELISM 4;

duplicate database to MAINS from active database NOFILENAMECHECK;

}

2> 3> 4> 5>

using target database control file instead of recovery catalog

new RMAN configuration parameters:

CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;

new RMAN configuration parameters are successfully stored

Starting Duplicate Db at 09-NOV-13

allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=169 device type=DISK

allocated channel: ORA_AUX_DISK_2

channel ORA_AUX_DISK_2: SID=175 device type=DISK

allocated channel: ORA_AUX_DISK_3

channel ORA_AUX_DISK_3: SID=181 device type=DISK

allocated channel: ORA_AUX_DISK_4

channel ORA_AUX_DISK_4: SID=187 device type=DISK

contents of Memory Script:

{

   sql clone "create spfile from memory";

}

executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:

{

   shutdown clone immediate;

   startup clone nomount;

}

executing Memory Script

Oracle instance shut down

connected to auxiliary database (not started)

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

contents of Memory Script:

{

   sql clone "alter system set  db_name =

 ''PBPROD'' comment=

 ''Modified by RMAN duplicate'' scope=spfile";

   sql clone "alter system set  db_unique_name =

 ''MAINS'' comment=

 ''Modified by RMAN duplicate'' scope=spfile";

   shutdown clone immediate;

   startup clone force nomount

   backup as copy current controlfile auxiliary format  '/u20_MAINS/redo1/control01.ctl';

   restore clone controlfile to  '/u20_MAINS/redo1/control02.ctl' from

 '/u20_MAINS/redo1/control01.ctl';

   restore clone controlfile to  '/u21_MAINS/redo2/control03.ctl' from

 '/u20_MAINS/redo1/control01.ctl';

   alter clone database mount;

}

executing Memory Script

sql statement: alter system set  db_name =  ''PBPROD'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''MAINS'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

Starting backup at 09-NOV-13

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=544 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: SID=595 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_3

channel ORA_DISK_3: SID=646 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_4

channel ORA_DISK_4: SID=694 instance=PBPROD5 device type=DISK

channel ORA_DISK_1: starting datafile copy

copying current control file

output file name=/u40_backup/MAINS/ctl/snapcf_PBPROD1.f tag=TAG20131109T022618 RECID=2 STAMP=831003979

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01

Finished backup at 09-NOV-13

Starting restore at 09-NOV-13

allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=169 device type=DISK

allocated channel: ORA_AUX_DISK_2

channel ORA_AUX_DISK_2: SID=175 device type=DISK

allocated channel: ORA_AUX_DISK_3

channel ORA_AUX_DISK_3: SID=181 device type=DISK

allocated channel: ORA_AUX_DISK_4

channel ORA_AUX_DISK_4: SID=187 device type=DISK

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_1: copied control file copy

Finished restore at 09-NOV-13

Starting restore at 09-NOV-13

using channel ORA_AUX_DISK_1

using channel ORA_AUX_DISK_2

using channel ORA_AUX_DISK_3

using channel ORA_AUX_DISK_4

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_1: copied control file copy

Finished restore at 09-NOV-13

database mounted

contents of Memory Script:

{

   set newname for datafile  1 to

 "/u10_MAINS/oradata/system01.dbf";

   set newname for datafile  2 to

 "/u10_MAINS/oradata/sysaux01.dbf";

   set newname for datafile  3 to

 "/u10_MAINS/oradata/undotbs05.dbf";

   set newname for datafile  4 to

 "/u10_MAINS/oradata/users01.dbf";

   set newname for datafile  6 to

 "/u10_MAINS/oradata/MST_01.dbf";

   set newname for datafile  7 to

 "/u10_MAINS/oradata/MSTIDX_01.dbf";

   set newname for datafile  8 to

 "/u10_MAINS/oradata/MSTLOB_01.dbf";

   set newname for datafile  9 to

 "/u10_MAINS/oradata/jiraext01.dbf";

   set newname for datafile  10 to

 "/u10_MAINS/oradata/jira01.dbf";

   set newname for datafile  11 to

 "/u10_MAINS/oradata/undotbs03.dbf";

   set newname for datafile  12 to

 "/u10_MAINS/oradata/undotbs04.dbf";

   backup as copy reuse

   datafile  1 auxiliary format

 "/u10_MAINS/oradata/system01.dbf"   datafile

 2 auxiliary format

 "/u10_MAINS/oradata/sysaux01.dbf"   datafile

 3 auxiliary format

 "/u10_MAINS/oradata/undotbs05.dbf"   datafile

 4 auxiliary format

 "/u10_MAINS/oradata/users01.dbf"   datafile

 6 auxiliary format

 "/u10_MAINS/oradata/MST_01.dbf"   datafile

 7 auxiliary format

 "/u10_MAINS/oradata/MSTIDX_01.dbf"   datafile

 8 auxiliary format

 "/u10_MAINS/oradata/MSTLOB_01.dbf"   datafile

 9 auxiliary format

 "/u10_MAINS/oradata/jiraext01.dbf"   datafile

 10 auxiliary format

 "/u10_MAINS/oradata/jira01.dbf"   datafile

 11 auxiliary format

 "/u10_MAINS/oradata/undotbs03.dbf"   datafile

 12 auxiliary format

 "/u10_MAINS/oradata/undotbs04.dbf"   ;

   sql 'alter system archive log current';

}

executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 09-NOV-13

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

channel ORA_DISK_1: starting datafile copy

input datafile file number=00011 name=/u10_MAINS/oradata/undotbs03.dbf

channel ORA_DISK_2: starting datafile copy

input datafile file number=00006 name=/u10_MAINS/oradata/MST_01.dbf

channel ORA_DISK_3: starting datafile copy

input datafile file number=00007 name=/u10_MAINS/oradata/MSTIDX_01.dbf

channel ORA_DISK_4: starting datafile copy

input datafile file number=00012 name=/u10_MAINS/oradata/undotbs04.dbf

output file name=/u10_MAINS/oradata/MSTIDX_01.dbf tag=TAG20131109T022644

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:04:56

channel ORA_DISK_3: starting datafile copy

input datafile file number=00002 name=/u10_MAINS/oradata/sysaux01.dbf

output file name=/u10_MAINS/oradata/undotbs04.dbf tag=TAG20131109T022644

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:04:57

channel ORA_DISK_4: starting datafile copy

input datafile file number=00010 name=/u10_MAINS/oradata/jira01.dbf

output file name=/u10_MAINS/oradata/MST_01.dbf tag=TAG20131109T022644

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:07:43

channel ORA_DISK_2: starting datafile copy

input datafile file number=00001 name=/u10_MAINS/oradata/system01.dbf

output file name=/u10_MAINS/oradata/jira01.dbf tag=TAG20131109T022644

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:03:03

channel ORA_DISK_4: starting datafile copy

input datafile file number=00008 name=/u10_MAINS/oradata/MSTLOB_01.dbf

output file name=/u10_MAINS/oradata/sysaux01.dbf tag=TAG20131109T022644

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:03:07

channel ORA_DISK_3: starting datafile copy

input datafile file number=00003 name=/u10_MAINS/oradata/undotbs05.dbf

output file name=/u10_MAINS/oradata/MSTLOB_01.dbf tag=TAG20131109T022644

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:09

channel ORA_DISK_4: starting datafile copy

input datafile file number=00004 name=/u10_MAINS/oradata/users01.dbf

output file name=/u10_MAINS/oradata/undotbs05.dbf tag=TAG20131109T022644

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:11

channel ORA_DISK_3: starting datafile copy

input datafile file number=00009 name=/u10_MAINS/oradata/jiraext01.dbf

output file name=/u10_MAINS/oradata/jiraext01.dbf tag=TAG20131109T022644

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:07

output file name=/u10_MAINS/oradata/users01.dbf tag=TAG20131109T022644

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:10

output file name=/u10_MAINS/oradata/system01.dbf tag=TAG20131109T022644

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:58

output file name=/u10_MAINS/oradata/undotbs03.dbf tag=TAG20131109T022644

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:09:04

Finished backup at 09-NOV-13

sql statement: alter system archive log current

contents of Memory Script:

{

   backup as copy reuse

   archivelog like  "/u30_ARCHIVE/MAINS/PBPROD_4_1214_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_4_1214_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf"   archivelog like

 "/u30_ARCHIVE/MAINS/PBPROD_4_1215_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_4_1215_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf"   archivelog like

 "/u30_ARCHIVE/MAINS/PBPROD_5_518_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_518_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_5_518_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_518_723932772.dbf"   ;

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_518_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_518_723932772.dbf";

   switch clone datafile all;

}

executing Memory Script

Starting backup at 09-NOV-13

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

channel ORA_DISK_1: starting archived log copy

input archived log thread=4 sequence=1214 RECID=9133 STAMP=830946634

channel ORA_DISK_2: starting archived log copy

input archived log thread=4 sequence=1214 RECID=9134 STAMP=830946634

channel ORA_DISK_3: starting archived log copy

input archived log thread=4 sequence=1215 RECID=9137 STAMP=831004563

channel ORA_DISK_4: starting archived log copy

input archived log thread=4 sequence=1215 RECID=9138 STAMP=831004563

output file name=/u31_ARCHIVE/MAINS/4_1215_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_3: archived log copy complete, elapsed time: 00:00:25

channel ORA_DISK_3: starting archived log copy

input archived log thread=5 sequence=518 RECID=9139 STAMP=831004565

output file name=/u31_ARCHIVE/MAINS/4_1215_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:25

channel ORA_DISK_4: starting archived log copy

input archived log thread=5 sequence=518 RECID=9140 STAMP=831004565

output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:27

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35

ORA-17628: Oracle error 19505 returned by remote Oracle server

continuing other job steps, job failed will not be re-run

output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:28

output file name=/u31_ARCHIVE/MAINS/5_518_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:26

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 11/09/2013 02:37:00

RMAN-05501: aborting duplication of target database

RMAN-03015: error occurred in stored script Memory Script

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35

ORA-17628: Oracle error 19505 returned by remote Oracle server

Now, when I look this up in the internet, most cases point to ‘Wrong location for controlfile’  and ‘missing parameter - db_file_name_convert in auxiliary pfile’

I rechecked and mine were correct. So, why did the error occur and duplicate fail ?

The first instance of the error was right after this step:

output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:27

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35

ORA-17628: Oracle error 19505 returned by remote Oracle server

I was able to ‘ls’ the file and compared it with source and it appeared to be correct.

After a bit of search, I found that there is a similar issue tied to NFS mounts. We were using netapp filers and we had storage volumes carved and mounted as NFS file system on our RAC.

This happens when duplicate is done to NFS mount point.

To resolve this, ensure that the NFS-filesystem is using the below recommended nfs mount option

rw,bg,nointr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp,noac

My mount options were correct:

> cat /etc/fstab |grep -i MAINS

#MAINS TOTAL

filer-prod02-node1:/vol/MAINS_oradata /u10_MAINS/oradata nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod02-node1:/vol/MAINS_redo1 /u20_MAINS/redo1 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod02-node2:/vol/MAINS_temp /u25_MAINS/temp nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod02-node2:/vol/MAINS_redo2 /u21_MAINS/redo2 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod01-node1:/vol/prod_oracle_backup/MAINS /u40_backup/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod02-node1:/vol/prod_oracle_archive/MAINS /u31_ARCHIVE/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

filer-prod01-node1:/vol/prod_oracle_archive/MAINS /u30_ARCHIVE/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0

So to work around and finish my duplication, I followed note 1401333.1

Put below event parameter in pfile of the auxiliary instance:
event="10298 trace name context forever, level 32"

Above event will instruct Oracle/rman to bypass the nfs checks.

So, I added the parameter to the pfile, restarted the auxiliary instance in nomount mode and retried the duplicate.

> cat initMAINS.ora

DB_NAME='MAINS'

diagnostic_dest='/u01/app/oracle'

DB_FILE_name_CONVERT=('/u10_MAINS/oradata','/u10_MAINS/oradata')

LOG_FILE_NAME_CONVERT=('/u20_MAINS/redo1','/u20_MAINS/redo1','/u21_MAINS/redo2','/u21_MAINS/redo2')

SGA_TARGET=1G

CONTROL_FILES='/u20_MAINS/redo1/control01.ctl','/u20_MAINS/redo1/control02.ctl','/u21_MAINS/redo2/control03.ctl'

COMPATIBLE= 11.2.0.2

cluster_database=false

log_archive_dest_1='LOCATION=/u30_ARCHIVE/MAINS/'

log_archive_dest_2='LOCATION=/u31_ARCHIVE/MAINS/'

local_listener='(address=(protocol=tcp)(host=myracn01v.mycomp.com)(port=1527))'

event="10298 trace name context forever, level 32"

RMAN> run

{

CONFIGURE DEVICE TYPE disk PARALLELISM 4;

duplicate database to MAINS from active database NOFILENAMECHECK;

}2> 3> 4> 5>

using target database control file instead of recovery catalog

old RMAN configuration parameters:

CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;

new RMAN configuration parameters:

CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;

new RMAN configuration parameters are successfully stored

Starting Duplicate Db at 09-NOV-13

allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=169 device type=DISK

allocated channel: ORA_AUX_DISK_2

channel ORA_AUX_DISK_2: SID=175 device type=DISK

allocated channel: ORA_AUX_DISK_3

channel ORA_AUX_DISK_3: SID=181 device type=DISK

allocated channel: ORA_AUX_DISK_4

channel ORA_AUX_DISK_4: SID=187 device type=DISK

contents of Memory Script:

{

   sql clone "create spfile from memory";

}

executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:

{

   shutdown clone immediate;

   startup clone nomount;

}

executing Memory Script

Oracle instance shut down

connected to auxiliary database (not started)

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

contents of Memory Script:

{

   sql clone "alter system set  db_name =

 ''PBPROD'' comment=

 ''Modified by RMAN duplicate'' scope=spfile";

   sql clone "alter system set  db_unique_name =

 ''MAINS'' comment=

 ''Modified by RMAN duplicate'' scope=spfile";

   shutdown clone immediate;

   startup clone force nomount

   backup as copy current controlfile auxiliary format  '/u20_MAINS/redo1/control01.ctl';

   restore clone controlfile to  '/u20_MAINS/redo1/control02.ctl' from

 '/u20_MAINS/redo1/control01.ctl';

   restore clone controlfile to  '/u21_MAINS/redo2/control03.ctl' from

 '/u20_MAINS/redo1/control01.ctl';

   alter clone database mount;

}

executing Memory Script

sql statement: alter system set  db_name =  ''PBPROD'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''MAINS'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

Starting backup at 09-NOV-13

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=544 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: SID=595 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_3

channel ORA_DISK_3: SID=646 instance=PBPROD5 device type=DISK

allocated channel: ORA_DISK_4

channel ORA_DISK_4: SID=694 instance=PBPROD5 device type=DISK

channel ORA_DISK_1: starting datafile copy

copying current control file

output file name=/u40_backup/MAINS/ctl/snapcf_PBPROD1.f tag=TAG20131109T030428 RECID=4 STAMP=831006269

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01

Finished backup at 09-NOV-13

Starting restore at 09-NOV-13

allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=169 device type=DISK

allocated channel: ORA_AUX_DISK_2

channel ORA_AUX_DISK_2: SID=175 device type=DISK

allocated channel: ORA_AUX_DISK_3

channel ORA_AUX_DISK_3: SID=181 device type=DISK

allocated channel: ORA_AUX_DISK_4

channel ORA_AUX_DISK_4: SID=187 device type=DISK

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_1: copied control file copy

Finished restore at 09-NOV-13

Starting restore at 09-NOV-13

using channel ORA_AUX_DISK_1

using channel ORA_AUX_DISK_2

using channel ORA_AUX_DISK_3

using channel ORA_AUX_DISK_4

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found

channel ORA_AUX_DISK_1: copied control file copy

Finished restore at 09-NOV-13

database mounted

contents of Memory Script:

{

   set newname for datafile  1 to

 "/u10_MAINS/oradata/system01.dbf";

   set newname for datafile  2 to

 "/u10_MAINS/oradata/sysaux01.dbf";

   set newname for datafile  3 to

 "/u10_MAINS/oradata/undotbs05.dbf";

   set newname for datafile  4 to

 "/u10_MAINS/oradata/users01.dbf";

   set newname for datafile  6 to

 "/u10_MAINS/oradata/MST_01.dbf";

   set newname for datafile  7 to

 "/u10_MAINS/oradata/MSTIDX_01.dbf";

   set newname for datafile  8 to

 "/u10_MAINS/oradata/MSTLOB_01.dbf";

   set newname for datafile  9 to

 "/u10_MAINS/oradata/jiraext01.dbf";

   set newname for datafile  10 to

 "/u10_MAINS/oradata/jira01.dbf";

   set newname for datafile  11 to

 "/u10_MAINS/oradata/undotbs03.dbf";

   set newname for datafile  12 to

 "/u10_MAINS/oradata/undotbs04.dbf";

   backup as copy reuse

   datafile  1 auxiliary format

 "/u10_MAINS/oradata/system01.dbf"   datafile

 2 auxiliary format

 "/u10_MAINS/oradata/sysaux01.dbf"   datafile

 3 auxiliary format

 "/u10_MAINS/oradata/undotbs05.dbf"   datafile

 4 auxiliary format

 "/u10_MAINS/oradata/users01.dbf"   datafile

 6 auxiliary format

 "/u10_MAINS/oradata/MST_01.dbf"   datafile

 7 auxiliary format

 "/u10_MAINS/oradata/MSTIDX_01.dbf"   datafile

 8 auxiliary format

 "/u10_MAINS/oradata/MSTLOB_01.dbf"   datafile

 9 auxiliary format

 "/u10_MAINS/oradata/jiraext01.dbf"   datafile

 10 auxiliary format

 "/u10_MAINS/oradata/jira01.dbf"   datafile

 11 auxiliary format

 "/u10_MAINS/oradata/undotbs03.dbf"   datafile

 12 auxiliary format

 "/u10_MAINS/oradata/undotbs04.dbf"   ;

   sql 'alter system archive log current';

}

executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 09-NOV-13

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

channel ORA_DISK_1: starting datafile copy

input datafile file number=00011 name=/u10_MAINS/oradata/undotbs03.dbf

channel ORA_DISK_2: starting datafile copy

input datafile file number=00006 name=/u10_MAINS/oradata/MST_01.dbf

channel ORA_DISK_3: starting datafile copy

input datafile file number=00007 name=/u10_MAINS/oradata/MSTIDX_01.dbf

channel ORA_DISK_4: starting datafile copy

input datafile file number=00012 name=/u10_MAINS/oradata/undotbs04.dbf

output file name=/u10_MAINS/oradata/MSTIDX_01.dbf tag=TAG20131109T030458

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:04:25

channel ORA_DISK_3: starting datafile copy

input datafile file number=00002 name=/u10_MAINS/oradata/sysaux01.dbf

output file name=/u10_MAINS/oradata/undotbs04.dbf tag=TAG20131109T030458

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:04:30

channel ORA_DISK_4: starting datafile copy

input datafile file number=00010 name=/u10_MAINS/oradata/jira01.dbf

output file name=/u10_MAINS/oradata/jira01.dbf tag=TAG20131109T030458

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:02:36

channel ORA_DISK_4: starting datafile copy

input datafile file number=00001 name=/u10_MAINS/oradata/system01.dbf

output file name=/u10_MAINS/oradata/MST_01.dbf tag=TAG20131109T030458

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:07:46

channel ORA_DISK_2: starting datafile copy

input datafile file number=00008 name=/u10_MAINS/oradata/MSTLOB_01.dbf

output file name=/u10_MAINS/oradata/MSTLOB_01.dbf tag=TAG20131109T030458

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:15

channel ORA_DISK_2: starting datafile copy

input datafile file number=00003 name=/u10_MAINS/oradata/undotbs05.dbf

output file name=/u10_MAINS/oradata/sysaux01.dbf tag=TAG20131109T030458

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:03:39

channel ORA_DISK_3: starting datafile copy

input datafile file number=00004 name=/u10_MAINS/oradata/users01.dbf

output file name=/u10_MAINS/oradata/undotbs05.dbf tag=TAG20131109T030458

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:17

channel ORA_DISK_2: starting datafile copy

input datafile file number=00009 name=/u10_MAINS/oradata/jiraext01.dbf

output file name=/u10_MAINS/oradata/users01.dbf tag=TAG20131109T030458

channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:15

output file name=/u10_MAINS/oradata/system01.dbf tag=TAG20131109T030458

channel ORA_DISK_4: datafile copy complete, elapsed time: 00:01:12

output file name=/u10_MAINS/oradata/jiraext01.dbf tag=TAG20131109T030458

channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:03

output file name=/u10_MAINS/oradata/undotbs03.dbf tag=TAG20131109T030458

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:08:58

Finished backup at 09-NOV-13

sql statement: alter system archive log current

contents of Memory Script:

{

   backup as copy reuse

   archivelog like  "/u30_ARCHIVE/MAINS/PBPROD_5_519_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_519_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_5_519_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_519_723932772.dbf"   archivelog like

 "/u30_ARCHIVE/MAINS/PBPROD_5_520_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_520_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_5_520_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/5_520_723932772.dbf"   archivelog like

 "/u30_ARCHIVE/MAINS/PBPROD_4_1217_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf"   archivelog like

 "/u31_ARCHIVE/MAINS/PBPROD_4_1217_723932772.arc" auxiliary format

 "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf"   ;

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_519_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_519_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_520_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_520_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf";

   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf";

   switch clone datafile all;

}

executing Memory Script

Starting backup at 09-NOV-13

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

channel ORA_DISK_1: starting archived log copy

input archived log thread=5 sequence=519 RECID=9143 STAMP=831006017

channel ORA_DISK_2: starting archived log copy

input archived log thread=5 sequence=519 RECID=9144 STAMP=831006017

channel ORA_DISK_3: starting archived log copy

input archived log thread=5 sequence=520 RECID=9145 STAMP=831006838

channel ORA_DISK_4: starting archived log copy

input archived log thread=5 sequence=520 RECID=9146 STAMP=831006838

output file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:01

channel ORA_DISK_1: starting archived log copy

input archived log thread=4 sequence=1217 RECID=9147 STAMP=831006840

output file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:01

channel ORA_DISK_2: starting archived log copy

input archived log thread=4 sequence=1217 RECID=9148 STAMP=831006840

output file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_3: archived log copy complete, elapsed time: 00:00:02

output file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:02

output file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:02

output file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=0 STAMP=0

channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:01

Finished backup at 09-NOV-13

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=9145 STAMP=831006846

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=9146 STAMP=831006846

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=9147 STAMP=831006846

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=9148 STAMP=831006847

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=9149 STAMP=831006847

cataloged archived log

archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=9150 STAMP=831006847

datafile 1 switched to datafile copy

input datafile copy RECID=4 STAMP=831006847 file name=/u10_MAINS/oradata/system01.dbf

datafile 2 switched to datafile copy

input datafile copy RECID=5 STAMP=831006847 file name=/u10_MAINS/oradata/sysaux01.dbf

datafile 3 switched to datafile copy

input datafile copy RECID=6 STAMP=831006848 file name=/u10_MAINS/oradata/undotbs05.dbf

datafile 4 switched to datafile copy

input datafile copy RECID=7 STAMP=831006848 file name=/u10_MAINS/oradata/users01.dbf

datafile 6 switched to datafile copy

input datafile copy RECID=8 STAMP=831006848 file name=/u10_MAINS/oradata/MST_01.dbf

datafile 7 switched to datafile copy

input datafile copy RECID=9 STAMP=831006848 file name=/u10_MAINS/oradata/MSTIDX_01.dbf

datafile 8 switched to datafile copy

input datafile copy RECID=10 STAMP=831006849 file name=/u10_MAINS/oradata/MSTLOB_01.dbf

datafile 9 switched to datafile copy

input datafile copy RECID=11 STAMP=831006849 file name=/u10_MAINS/oradata/jiraext01.dbf

datafile 10 switched to datafile copy

input datafile copy RECID=12 STAMP=831006849 file name=/u10_MAINS/oradata/jira01.dbf

datafile 11 switched to datafile copy

input datafile copy RECID=13 STAMP=831006849 file name=/u10_MAINS/oradata/undotbs03.dbf

datafile 12 switched to datafile copy

input datafile copy RECID=14 STAMP=831006850 file name=/u10_MAINS/oradata/undotbs04.dbf

contents of Memory Script:

{

   set until scn  5447248687;

   recover

   clone database

    delete archivelog

   ;

}

executing Memory Script

executing command: SET until clause

Starting recover at 09-NOV-13

using channel ORA_AUX_DISK_1

using channel ORA_AUX_DISK_2

using channel ORA_AUX_DISK_3

using channel ORA_AUX_DISK_4

starting media recovery

archived log for thread 4 with sequence 1217 is already on disk as file /u31_ARCHIVE/MAINS/4_1217_723932772.dbf

archived log for thread 5 with sequence 520 is already on disk as file /u31_ARCHIVE/MAINS/5_520_723932772.dbf

archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf thread=4 sequence=1217

archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf thread=5 sequence=520

media recovery complete, elapsed time: 00:00:00

Finished recover at 09-NOV-13

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

contents of Memory Script:

{

   sql clone "alter system set  db_name =

 ''MAINS'' comment=

 ''Reset to original value by RMAN'' scope=spfile";

   sql clone "alter system reset  db_unique_name scope=spfile";

   shutdown clone immediate;

   startup clone nomount;

}

executing Memory Script

sql statement: alter system set  db_name =  ''MAINS'' comment= ''Reset to original value by RMAN'' scope=spfile

sql statement: alter system reset  db_unique_name scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes

Variable Size                285215752 bytes

Database Buffers             771751936 bytes

Redo Buffers                   9736192 bytes

sql statement: CREATE CONTROLFILE REUSE SET DATABASE "MAINS" RESETLOGS ARCHIVELOG

  MAXLOGFILES     16

  MAXLOGMEMBERS      3

  MAXDATAFILES      100

  MAXINSTANCES     8

  MAXLOGHISTORY      292

 LOGFILE

  GROUP  5 ( '/u21_MAINS/redo2/redo05b.log', '/u20_MAINS/redo1/redo05a.log' ) SIZE 500 M  REUSE,

  GROUP  6 ( '/u20_MAINS/redo1/redo06a.log', '/u21_MAINS/redo2/redo06b.log' ) SIZE 500 M  REUSE

 DATAFILE

  '/u10_MAINS/oradata/system01.dbf'

 CHARACTER SET AL32UTF8

sql statement: ALTER DATABASE ADD LOGFILE

  INSTANCE 'i4'

  GROUP  7 ( '/u20_MAINS/redo1/redo07a.log', '/u21_MAINS/redo2/redo07b.log' ) SIZE 500 M  REUSE,

  GROUP  8 ( '/u20_MAINS/redo1/redo08a.log', '/u21_MAINS/redo2/redo08b.log' ) SIZE 500 M  REUSE

sql statement: ALTER DATABASE ADD LOGFILE

  INSTANCE 'i5'

  GROUP  9 ( '/u20_MAINS/redo1/redo09a.log', '/u21_MAINS/redo2/redo09b.log' ) SIZE 500 M  REUSE,

  GROUP 10 ( '/u21_MAINS/redo2/redo10b.log', '/u20_MAINS/redo1/redo10a.log' ) SIZE 500 M  REUSE

contents of Memory Script:

{

   set newname for tempfile  2 to

 "/u25_TEMP/MAINS/temp01.dbf";

   switch clone tempfile all;

   catalog clone datafilecopy  "/u10_MAINS/oradata/sysaux01.dbf",

 "/u10_MAINS/oradata/undotbs05.dbf",

 "/u10_MAINS/oradata/users01.dbf",

 "/u10_MAINS/oradata/MST_01.dbf",

 "/u10_MAINS/oradata/MSTIDX_01.dbf",

 "/u10_MAINS/oradata/MSTLOB_01.dbf",

 "/u10_MAINS/oradata/jiraext01.dbf",

 "/u10_MAINS/oradata/jira01.dbf",

 "/u10_MAINS/oradata/undotbs03.dbf",

 "/u10_MAINS/oradata/undotbs04.dbf";

   switch clone datafile all;

}

executing Memory Script

executing command: SET NEWNAME

renamed tempfile 2 to /u25_TEMP/MAINS/temp01.dbf in control file

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/sysaux01.dbf RECID=1 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/undotbs05.dbf RECID=2 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/users01.dbf RECID=3 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/MST_01.dbf RECID=4 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/MSTIDX_01.dbf RECID=5 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/MSTLOB_01.dbf RECID=6 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/jiraext01.dbf RECID=7 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/jira01.dbf RECID=8 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/undotbs03.dbf RECID=9 STAMP=831006912

cataloged datafile copy

datafile copy file name=/u10_MAINS/oradata/undotbs04.dbf RECID=10 STAMP=831006912

datafile 2 switched to datafile copy

input datafile copy RECID=1 STAMP=831006912 file name=/u10_MAINS/oradata/sysaux01.dbf

datafile 3 switched to datafile copy

input datafile copy RECID=2 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs05.dbf

datafile 4 switched to datafile copy

input datafile copy RECID=3 STAMP=831006912 file name=/u10_MAINS/oradata/users01.dbf

datafile 6 switched to datafile copy

input datafile copy RECID=4 STAMP=831006912 file name=/u10_MAINS/oradata/MST_01.dbf

datafile 7 switched to datafile copy

input datafile copy RECID=5 STAMP=831006912 file name=/u10_MAINS/oradata/MSTIDX_01.dbf

datafile 8 switched to datafile copy

input datafile copy RECID=6 STAMP=831006912 file name=/u10_MAINS/oradata/MSTLOB_01.dbf

datafile 9 switched to datafile copy

input datafile copy RECID=7 STAMP=831006912 file name=/u10_MAINS/oradata/jiraext01.dbf

datafile 10 switched to datafile copy

input datafile copy RECID=8 STAMP=831006912 file name=/u10_MAINS/oradata/jira01.dbf

datafile 11 switched to datafile copy

input datafile copy RECID=9 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs03.dbf

datafile 12 switched to datafile copy

input datafile copy RECID=10 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs04.dbf

contents of Memory Script:

{

   Alter clone database open resetlogs;

}

executing Memory Script

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 11/09/2013 03:15:59

RMAN-05501: aborting duplication of target database

RMAN-03015: error occurred in stored script Memory Script

RMAN-06136: ORACLE error from auxiliary database: ORA-38856: cannot mark instance UNNAMED_INSTANCE_4 (redo thread 4) as enabled

RMAN> exit

Got the above error during the ‘alter database open resetlogs’. I guess most of us would know how to resolve the above error. Refer note 334899.1 to use “_no_recovery_through_resetlogs=TRUE” and then issue a manual ‘alter database open resetlogs’ via sqlplus. 

At this point we have a cloned database instance. Remove all hidden parameters and add/remove instances as required.