Monday, November 11, 2013

Issue during RMAN duplicate database; ORA-17628: Oracle error 19505 returned by remote Oracle server

Issue during RMAN duplicate database;
ORA-17628: Oracle error 19505 returned by remote Oracle server

Error :

The following error occurred when attempting to duplicate database from one RAC cluster to another.

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:28
output file name=/u31_ARCHIVE/MAINS/5_518_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:26
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/09/2013 02:37:00
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script

Version:
11.2.0.2

Duplicate Process:

Parameter File

> cat initMAINS.ora

DB_NAME='MAINS'
diagnostic_dest='/u01/app/oracle'
DB_FILE_name_CONVERT=('/u10_MAINS/oradata','/u10_MAINS/oradata')
LOG_FILE_NAME_CONVERT=('/u20_MAINS/redo1','/u20_MAINS/redo1','/u21_MAINS/redo2','/u21_MAINS/redo2')
SGA_TARGET=1G
CONTROL_FILES='/u20_MAINS/redo1/control01.ctl','/u20_MAINS/redo1/control02.ctl','/u21_MAINS/redo2/control03.ctl'
COMPATIBLE= 11.2.0.2
cluster_database=false
log_archive_dest_1='LOCATION=/u30_ARCHIVE/MAINS/'
log_archive_dest_2='LOCATION=/u31_ARCHIVE/MAINS/'
local_listener='(address=(protocol=tcp)(host=myracn01v.mycomp.com)(port=1527))'


Static listener configuration

LISTENER_MAINS =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_MAINS))
      (ADDRESS = (PROTOCOL = TCP)(HOST = myracn01v.mycomp.com)(PORT = 1527))
    )
  )

SID_LIST_LISTENER_MAINS =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = MAINS)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0.2)
      (SID_NAME = MAINS)
    )
  )


Added oratab entry, set the environment and started the instance in nomount

SQL> startup nomount
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2233336 bytes
Variable Size             281021448 bytes
Database Buffers          775946240 bytes
Redo Buffers                9736192 bytes


RMAN Duplication

> rman

Recovery Manager: Release 11.2.0.2.0 - Production on Sat Nov 9 02:24:15 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prodblr10v.mycomp.com)(PORT=1561)))(CONNECT_DATA=(SID=PBPROD5)))

target database Password:
connected to target database: PBPROD (DBID=2131291108)

RMAN> connect auxiliary sys@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=myracn01v.mycomp.com)(PORT=1527)))(CONNECT_DATA=(SID=MAINS)))

auxiliary database Password:
connected to auxiliary database: MAINS (not mounted)

RMAN> run
{
CONFIGURE DEVICE TYPE disk PARALLELISM 4;
duplicate database to MAINS from active database NOFILENAMECHECK;
}
2> 3> 4> 5>
using target database control file instead of recovery catalog
new RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters are successfully stored

Starting Duplicate Db at 09-NOV-13
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=169 device type=DISK
allocated channel: ORA_AUX_DISK_2
channel ORA_AUX_DISK_2: SID=175 device type=DISK
allocated channel: ORA_AUX_DISK_3
channel ORA_AUX_DISK_3: SID=181 device type=DISK
allocated channel: ORA_AUX_DISK_4
channel ORA_AUX_DISK_4: SID=187 device type=DISK

contents of Memory Script:
{
   sql clone "create spfile from memory";
}
executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:
{
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''PBPROD'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''MAINS'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u20_MAINS/redo1/control01.ctl';
   restore clone controlfile to  '/u20_MAINS/redo1/control02.ctl' from
 '/u20_MAINS/redo1/control01.ctl';
   restore clone controlfile to  '/u21_MAINS/redo2/control03.ctl' from
 '/u20_MAINS/redo1/control01.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''PBPROD'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''MAINS'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes

Starting backup at 09-NOV-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=544 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=595 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=646 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=694 instance=PBPROD5 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=/u40_backup/MAINS/ctl/snapcf_PBPROD1.f tag=TAG20131109T022618 RECID=2 STAMP=831003979
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 09-NOV-13

Starting restore at 09-NOV-13
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=169 device type=DISK
allocated channel: ORA_AUX_DISK_2
channel ORA_AUX_DISK_2: SID=175 device type=DISK
allocated channel: ORA_AUX_DISK_3
channel ORA_AUX_DISK_3: SID=181 device type=DISK
allocated channel: ORA_AUX_DISK_4
channel ORA_AUX_DISK_4: SID=187 device type=DISK

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 09-NOV-13

Starting restore at 09-NOV-13
using channel ORA_AUX_DISK_1
using channel ORA_AUX_DISK_2
using channel ORA_AUX_DISK_3
using channel ORA_AUX_DISK_4

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 09-NOV-13

database mounted

contents of Memory Script:
{
   set newname for datafile  1 to
 "/u10_MAINS/oradata/system01.dbf";
   set newname for datafile  2 to
 "/u10_MAINS/oradata/sysaux01.dbf";
   set newname for datafile  3 to
 "/u10_MAINS/oradata/undotbs05.dbf";
   set newname for datafile  4 to
 "/u10_MAINS/oradata/users01.dbf";
   set newname for datafile  6 to
 "/u10_MAINS/oradata/MST_01.dbf";
   set newname for datafile  7 to
 "/u10_MAINS/oradata/MSTIDX_01.dbf";
   set newname for datafile  8 to
 "/u10_MAINS/oradata/MSTLOB_01.dbf";
   set newname for datafile  9 to
 "/u10_MAINS/oradata/jiraext01.dbf";
   set newname for datafile  10 to
 "/u10_MAINS/oradata/jira01.dbf";
   set newname for datafile  11 to
 "/u10_MAINS/oradata/undotbs03.dbf";
   set newname for datafile  12 to
 "/u10_MAINS/oradata/undotbs04.dbf";
   backup as copy reuse
   datafile  1 auxiliary format
 "/u10_MAINS/oradata/system01.dbf"   datafile
 2 auxiliary format
 "/u10_MAINS/oradata/sysaux01.dbf"   datafile
 3 auxiliary format
 "/u10_MAINS/oradata/undotbs05.dbf"   datafile
 4 auxiliary format
 "/u10_MAINS/oradata/users01.dbf"   datafile
 6 auxiliary format
 "/u10_MAINS/oradata/MST_01.dbf"   datafile
 7 auxiliary format
 "/u10_MAINS/oradata/MSTIDX_01.dbf"   datafile
 8 auxiliary format
 "/u10_MAINS/oradata/MSTLOB_01.dbf"   datafile
 9 auxiliary format
 "/u10_MAINS/oradata/jiraext01.dbf"   datafile
 10 auxiliary format
 "/u10_MAINS/oradata/jira01.dbf"   datafile
 11 auxiliary format
 "/u10_MAINS/oradata/undotbs03.dbf"   datafile
 12 auxiliary format
 "/u10_MAINS/oradata/undotbs04.dbf"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 09-NOV-13
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
channel ORA_DISK_1: starting datafile copy
input datafile file number=00011 name=/u10_MAINS/oradata/undotbs03.dbf
channel ORA_DISK_2: starting datafile copy
input datafile file number=00006 name=/u10_MAINS/oradata/MST_01.dbf
channel ORA_DISK_3: starting datafile copy
input datafile file number=00007 name=/u10_MAINS/oradata/MSTIDX_01.dbf
channel ORA_DISK_4: starting datafile copy
input datafile file number=00012 name=/u10_MAINS/oradata/undotbs04.dbf
output file name=/u10_MAINS/oradata/MSTIDX_01.dbf tag=TAG20131109T022644
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:04:56
channel ORA_DISK_3: starting datafile copy
input datafile file number=00002 name=/u10_MAINS/oradata/sysaux01.dbf
output file name=/u10_MAINS/oradata/undotbs04.dbf tag=TAG20131109T022644
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:04:57
channel ORA_DISK_4: starting datafile copy
input datafile file number=00010 name=/u10_MAINS/oradata/jira01.dbf
output file name=/u10_MAINS/oradata/MST_01.dbf tag=TAG20131109T022644
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:07:43
channel ORA_DISK_2: starting datafile copy
input datafile file number=00001 name=/u10_MAINS/oradata/system01.dbf
output file name=/u10_MAINS/oradata/jira01.dbf tag=TAG20131109T022644
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:03:03
channel ORA_DISK_4: starting datafile copy
input datafile file number=00008 name=/u10_MAINS/oradata/MSTLOB_01.dbf
output file name=/u10_MAINS/oradata/sysaux01.dbf tag=TAG20131109T022644
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:03:07
channel ORA_DISK_3: starting datafile copy
input datafile file number=00003 name=/u10_MAINS/oradata/undotbs05.dbf
output file name=/u10_MAINS/oradata/MSTLOB_01.dbf tag=TAG20131109T022644
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:09
channel ORA_DISK_4: starting datafile copy
input datafile file number=00004 name=/u10_MAINS/oradata/users01.dbf
output file name=/u10_MAINS/oradata/undotbs05.dbf tag=TAG20131109T022644
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:11
channel ORA_DISK_3: starting datafile copy
input datafile file number=00009 name=/u10_MAINS/oradata/jiraext01.dbf
output file name=/u10_MAINS/oradata/jiraext01.dbf tag=TAG20131109T022644
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:07
output file name=/u10_MAINS/oradata/users01.dbf tag=TAG20131109T022644
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:10
output file name=/u10_MAINS/oradata/system01.dbf tag=TAG20131109T022644
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:58
output file name=/u10_MAINS/oradata/undotbs03.dbf tag=TAG20131109T022644
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:09:04
Finished backup at 09-NOV-13

sql statement: alter system archive log current

contents of Memory Script:
{
   backup as copy reuse
   archivelog like  "/u30_ARCHIVE/MAINS/PBPROD_4_1214_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_4_1214_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf"   archivelog like
 "/u30_ARCHIVE/MAINS/PBPROD_4_1215_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_4_1215_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf"   archivelog like
 "/u30_ARCHIVE/MAINS/PBPROD_5_518_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_518_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_5_518_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_518_723932772.dbf"   ;
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1214_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1215_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_518_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_518_723932772.dbf";
   switch clone datafile all;
}
executing Memory Script

Starting backup at 09-NOV-13
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
channel ORA_DISK_1: starting archived log copy
input archived log thread=4 sequence=1214 RECID=9133 STAMP=830946634
channel ORA_DISK_2: starting archived log copy
input archived log thread=4 sequence=1214 RECID=9134 STAMP=830946634
channel ORA_DISK_3: starting archived log copy
input archived log thread=4 sequence=1215 RECID=9137 STAMP=831004563
channel ORA_DISK_4: starting archived log copy
input archived log thread=4 sequence=1215 RECID=9138 STAMP=831004563
output file name=/u31_ARCHIVE/MAINS/4_1215_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_3: archived log copy complete, elapsed time: 00:00:25
channel ORA_DISK_3: starting archived log copy
input archived log thread=5 sequence=518 RECID=9139 STAMP=831004565
output file name=/u31_ARCHIVE/MAINS/4_1215_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:25
channel ORA_DISK_4: starting archived log copy
input archived log thread=5 sequence=518 RECID=9140 STAMP=831004565
output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:27
RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:28
output file name=/u31_ARCHIVE/MAINS/5_518_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:26
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/09/2013 02:37:00
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script

RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35
ORA-17628: Oracle error 19505 returned by remote Oracle server




Now, when I look this up in the internet, most cases point to ‘Wrong location for controlfile’  and ‘missing parameter - db_file_name_convert in auxiliary pfile’
I rechecked and mine were correct. So, why did the error occur and duplicate fail ?

The first instance of the error was right after this step:

output file name=/u31_ARCHIVE/MAINS/4_1214_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:27
RMAN-03009: failure of backup command on ORA_DISK_3 channel at 11/09/2013 02:36:35
ORA-17628: Oracle error 19505 returned by remote Oracle server

I was able to ‘ls’ the file and compared it with source and it appeared to be correct.

After a bit of search, I found that there is a similar issue tied to NFS mounts. We were using netapp filers and we had storage volumes carved and mounted as NFS file system on our RAC.
This happens when duplicate is done to NFS mount point.

To resolve this, ensure that the NFS-filesystem is using the below recommended nfs mount option
rw,bg,nointr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp,noac

My mount options were correct:

> cat /etc/fstab |grep -i MAINS
#MAINS TOTAL
filer-prod02-node1:/vol/MAINS_oradata /u10_MAINS/oradata nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod02-node1:/vol/MAINS_redo1 /u20_MAINS/redo1 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod02-node2:/vol/MAINS_temp /u25_MAINS/temp nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod02-node2:/vol/MAINS_redo2 /u21_MAINS/redo2 nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod01-node1:/vol/prod_oracle_backup/MAINS /u40_backup/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod02-node1:/vol/prod_oracle_archive/MAINS /u31_ARCHIVE/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0
filer-prod01-node1:/vol/prod_oracle_archive/MAINS /u30_ARCHIVE/MAINS nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0


So to work around and finish my duplication, I followed note 1401333.1

Put below event parameter in pfile of the auxiliary instance:
event="10298 trace name context forever, level 32"

Above event will instruct Oracle/rman to bypass the nfs checks.


So, I added the parameter to the pfile, restarted the auxiliary instance in nomount mode and retried the duplicate.

> cat initMAINS.ora
DB_NAME='MAINS'
diagnostic_dest='/u01/app/oracle'
DB_FILE_name_CONVERT=('/u10_MAINS/oradata','/u10_MAINS/oradata')
LOG_FILE_NAME_CONVERT=('/u20_MAINS/redo1','/u20_MAINS/redo1','/u21_MAINS/redo2','/u21_MAINS/redo2')
SGA_TARGET=1G
CONTROL_FILES='/u20_MAINS/redo1/control01.ctl','/u20_MAINS/redo1/control02.ctl','/u21_MAINS/redo2/control03.ctl'
COMPATIBLE= 11.2.0.2
cluster_database=false
log_archive_dest_1='LOCATION=/u30_ARCHIVE/MAINS/'
log_archive_dest_2='LOCATION=/u31_ARCHIVE/MAINS/'
local_listener='(address=(protocol=tcp)(host=myracn01v.mycomp.com)(port=1527))'
event="10298 trace name context forever, level 32"



RMAN> run
{
CONFIGURE DEVICE TYPE disk PARALLELISM 4;
duplicate database to MAINS from active database NOFILENAMECHECK;
}2> 3> 4> 5>

using target database control file instead of recovery catalog
old RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters are successfully stored

Starting Duplicate Db at 09-NOV-13
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=169 device type=DISK
allocated channel: ORA_AUX_DISK_2
channel ORA_AUX_DISK_2: SID=175 device type=DISK
allocated channel: ORA_AUX_DISK_3
channel ORA_AUX_DISK_3: SID=181 device type=DISK
allocated channel: ORA_AUX_DISK_4
channel ORA_AUX_DISK_4: SID=187 device type=DISK

contents of Memory Script:
{
   sql clone "create spfile from memory";
}
executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:
{
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''PBPROD'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''MAINS'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u20_MAINS/redo1/control01.ctl';
   restore clone controlfile to  '/u20_MAINS/redo1/control02.ctl' from
 '/u20_MAINS/redo1/control01.ctl';
   restore clone controlfile to  '/u21_MAINS/redo2/control03.ctl' from
 '/u20_MAINS/redo1/control01.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''PBPROD'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''MAINS'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes

Starting backup at 09-NOV-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=544 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=595 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=646 instance=PBPROD5 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=694 instance=PBPROD5 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=/u40_backup/MAINS/ctl/snapcf_PBPROD1.f tag=TAG20131109T030428 RECID=4 STAMP=831006269
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 09-NOV-13

Starting restore at 09-NOV-13
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=169 device type=DISK
allocated channel: ORA_AUX_DISK_2
channel ORA_AUX_DISK_2: SID=175 device type=DISK
allocated channel: ORA_AUX_DISK_3
channel ORA_AUX_DISK_3: SID=181 device type=DISK
allocated channel: ORA_AUX_DISK_4
channel ORA_AUX_DISK_4: SID=187 device type=DISK

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 09-NOV-13

Starting restore at 09-NOV-13
using channel ORA_AUX_DISK_1
using channel ORA_AUX_DISK_2
using channel ORA_AUX_DISK_3
using channel ORA_AUX_DISK_4

channel ORA_AUX_DISK_2: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_3: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_4: skipped, AUTOBACKUP already found
channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 09-NOV-13

database mounted

contents of Memory Script:
{
   set newname for datafile  1 to
 "/u10_MAINS/oradata/system01.dbf";
   set newname for datafile  2 to
 "/u10_MAINS/oradata/sysaux01.dbf";
   set newname for datafile  3 to
 "/u10_MAINS/oradata/undotbs05.dbf";
   set newname for datafile  4 to
 "/u10_MAINS/oradata/users01.dbf";
   set newname for datafile  6 to
 "/u10_MAINS/oradata/MST_01.dbf";
   set newname for datafile  7 to
 "/u10_MAINS/oradata/MSTIDX_01.dbf";
   set newname for datafile  8 to
 "/u10_MAINS/oradata/MSTLOB_01.dbf";
   set newname for datafile  9 to
 "/u10_MAINS/oradata/jiraext01.dbf";
   set newname for datafile  10 to
 "/u10_MAINS/oradata/jira01.dbf";
   set newname for datafile  11 to
 "/u10_MAINS/oradata/undotbs03.dbf";
   set newname for datafile  12 to
 "/u10_MAINS/oradata/undotbs04.dbf";
   backup as copy reuse
   datafile  1 auxiliary format
 "/u10_MAINS/oradata/system01.dbf"   datafile
 2 auxiliary format
 "/u10_MAINS/oradata/sysaux01.dbf"   datafile
 3 auxiliary format
 "/u10_MAINS/oradata/undotbs05.dbf"   datafile
 4 auxiliary format
 "/u10_MAINS/oradata/users01.dbf"   datafile
 6 auxiliary format
 "/u10_MAINS/oradata/MST_01.dbf"   datafile
 7 auxiliary format
 "/u10_MAINS/oradata/MSTIDX_01.dbf"   datafile
 8 auxiliary format
 "/u10_MAINS/oradata/MSTLOB_01.dbf"   datafile
 9 auxiliary format
 "/u10_MAINS/oradata/jiraext01.dbf"   datafile
 10 auxiliary format
 "/u10_MAINS/oradata/jira01.dbf"   datafile
 11 auxiliary format
 "/u10_MAINS/oradata/undotbs03.dbf"   datafile
 12 auxiliary format
 "/u10_MAINS/oradata/undotbs04.dbf"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 09-NOV-13
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
channel ORA_DISK_1: starting datafile copy
input datafile file number=00011 name=/u10_MAINS/oradata/undotbs03.dbf
channel ORA_DISK_2: starting datafile copy
input datafile file number=00006 name=/u10_MAINS/oradata/MST_01.dbf
channel ORA_DISK_3: starting datafile copy
input datafile file number=00007 name=/u10_MAINS/oradata/MSTIDX_01.dbf
channel ORA_DISK_4: starting datafile copy
input datafile file number=00012 name=/u10_MAINS/oradata/undotbs04.dbf
output file name=/u10_MAINS/oradata/MSTIDX_01.dbf tag=TAG20131109T030458
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:04:25
channel ORA_DISK_3: starting datafile copy
input datafile file number=00002 name=/u10_MAINS/oradata/sysaux01.dbf
output file name=/u10_MAINS/oradata/undotbs04.dbf tag=TAG20131109T030458
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:04:30
channel ORA_DISK_4: starting datafile copy
input datafile file number=00010 name=/u10_MAINS/oradata/jira01.dbf
output file name=/u10_MAINS/oradata/jira01.dbf tag=TAG20131109T030458
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:02:36
channel ORA_DISK_4: starting datafile copy
input datafile file number=00001 name=/u10_MAINS/oradata/system01.dbf
output file name=/u10_MAINS/oradata/MST_01.dbf tag=TAG20131109T030458
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:07:46
channel ORA_DISK_2: starting datafile copy
input datafile file number=00008 name=/u10_MAINS/oradata/MSTLOB_01.dbf
output file name=/u10_MAINS/oradata/MSTLOB_01.dbf tag=TAG20131109T030458
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_2: starting datafile copy
input datafile file number=00003 name=/u10_MAINS/oradata/undotbs05.dbf
output file name=/u10_MAINS/oradata/sysaux01.dbf tag=TAG20131109T030458
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:03:39
channel ORA_DISK_3: starting datafile copy
input datafile file number=00004 name=/u10_MAINS/oradata/users01.dbf
output file name=/u10_MAINS/oradata/undotbs05.dbf tag=TAG20131109T030458
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:17
channel ORA_DISK_2: starting datafile copy
input datafile file number=00009 name=/u10_MAINS/oradata/jiraext01.dbf
output file name=/u10_MAINS/oradata/users01.dbf tag=TAG20131109T030458
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:15
output file name=/u10_MAINS/oradata/system01.dbf tag=TAG20131109T030458
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:01:12
output file name=/u10_MAINS/oradata/jiraext01.dbf tag=TAG20131109T030458
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:03
output file name=/u10_MAINS/oradata/undotbs03.dbf tag=TAG20131109T030458
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:08:58
Finished backup at 09-NOV-13

sql statement: alter system archive log current

contents of Memory Script:
{
   backup as copy reuse
   archivelog like  "/u30_ARCHIVE/MAINS/PBPROD_5_519_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_519_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_5_519_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_519_723932772.dbf"   archivelog like
 "/u30_ARCHIVE/MAINS/PBPROD_5_520_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_520_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_5_520_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/5_520_723932772.dbf"   archivelog like
 "/u30_ARCHIVE/MAINS/PBPROD_4_1217_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf"   archivelog like
 "/u31_ARCHIVE/MAINS/PBPROD_4_1217_723932772.arc" auxiliary format
 "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf"   ;
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_519_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_519_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_520_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/5_520_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf";
   catalog clone archivelog  "/u31_ARCHIVE/MAINS/4_1217_723932772.dbf";
   switch clone datafile all;
}
executing Memory Script

Starting backup at 09-NOV-13
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
channel ORA_DISK_1: starting archived log copy
input archived log thread=5 sequence=519 RECID=9143 STAMP=831006017
channel ORA_DISK_2: starting archived log copy
input archived log thread=5 sequence=519 RECID=9144 STAMP=831006017
channel ORA_DISK_3: starting archived log copy
input archived log thread=5 sequence=520 RECID=9145 STAMP=831006838
channel ORA_DISK_4: starting archived log copy
input archived log thread=5 sequence=520 RECID=9146 STAMP=831006838
output file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log copy
input archived log thread=4 sequence=1217 RECID=9147 STAMP=831006840
output file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:01
channel ORA_DISK_2: starting archived log copy
input archived log thread=4 sequence=1217 RECID=9148 STAMP=831006840
output file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_3: archived log copy complete, elapsed time: 00:00:02
output file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_4: archived log copy complete, elapsed time: 00:00:02
output file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:02
output file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=0 STAMP=0
channel ORA_DISK_2: archived log copy complete, elapsed time: 00:00:01
Finished backup at 09-NOV-13

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=9145 STAMP=831006846

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/5_519_723932772.dbf RECID=9146 STAMP=831006846

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=9147 STAMP=831006846

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf RECID=9148 STAMP=831006847

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=9149 STAMP=831006847

cataloged archived log
archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf RECID=9150 STAMP=831006847

datafile 1 switched to datafile copy
input datafile copy RECID=4 STAMP=831006847 file name=/u10_MAINS/oradata/system01.dbf
datafile 2 switched to datafile copy
input datafile copy RECID=5 STAMP=831006847 file name=/u10_MAINS/oradata/sysaux01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=6 STAMP=831006848 file name=/u10_MAINS/oradata/undotbs05.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=7 STAMP=831006848 file name=/u10_MAINS/oradata/users01.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=8 STAMP=831006848 file name=/u10_MAINS/oradata/MST_01.dbf
datafile 7 switched to datafile copy
input datafile copy RECID=9 STAMP=831006848 file name=/u10_MAINS/oradata/MSTIDX_01.dbf
datafile 8 switched to datafile copy
input datafile copy RECID=10 STAMP=831006849 file name=/u10_MAINS/oradata/MSTLOB_01.dbf
datafile 9 switched to datafile copy
input datafile copy RECID=11 STAMP=831006849 file name=/u10_MAINS/oradata/jiraext01.dbf
datafile 10 switched to datafile copy
input datafile copy RECID=12 STAMP=831006849 file name=/u10_MAINS/oradata/jira01.dbf
datafile 11 switched to datafile copy
input datafile copy RECID=13 STAMP=831006849 file name=/u10_MAINS/oradata/undotbs03.dbf
datafile 12 switched to datafile copy
input datafile copy RECID=14 STAMP=831006850 file name=/u10_MAINS/oradata/undotbs04.dbf

contents of Memory Script:
{
   set until scn  5447248687;
   recover
   clone database
    delete archivelog
   ;
}
executing Memory Script

executing command: SET until clause

Starting recover at 09-NOV-13
using channel ORA_AUX_DISK_1
using channel ORA_AUX_DISK_2
using channel ORA_AUX_DISK_3
using channel ORA_AUX_DISK_4

starting media recovery

archived log for thread 4 with sequence 1217 is already on disk as file /u31_ARCHIVE/MAINS/4_1217_723932772.dbf
archived log for thread 5 with sequence 520 is already on disk as file /u31_ARCHIVE/MAINS/5_520_723932772.dbf
archived log file name=/u31_ARCHIVE/MAINS/4_1217_723932772.dbf thread=4 sequence=1217
archived log file name=/u31_ARCHIVE/MAINS/5_520_723932772.dbf thread=5 sequence=520
media recovery complete, elapsed time: 00:00:00
Finished recover at 09-NOV-13
Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''MAINS'' comment=
 ''Reset to original value by RMAN'' scope=spfile";
   sql clone "alter system reset  db_unique_name scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''MAINS'' comment= ''Reset to original value by RMAN'' scope=spfile

sql statement: alter system reset  db_unique_name scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2233336 bytes
Variable Size                285215752 bytes
Database Buffers             771751936 bytes
Redo Buffers                   9736192 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "MAINS" RESETLOGS ARCHIVELOG
  MAXLOGFILES     16
  MAXLOGMEMBERS      3
  MAXDATAFILES      100
  MAXINSTANCES     8
  MAXLOGHISTORY      292
 LOGFILE
  GROUP  5 ( '/u21_MAINS/redo2/redo05b.log', '/u20_MAINS/redo1/redo05a.log' ) SIZE 500 M  REUSE,
  GROUP  6 ( '/u20_MAINS/redo1/redo06a.log', '/u21_MAINS/redo2/redo06b.log' ) SIZE 500 M  REUSE
 DATAFILE
  '/u10_MAINS/oradata/system01.dbf'
 CHARACTER SET AL32UTF8

sql statement: ALTER DATABASE ADD LOGFILE

  INSTANCE 'i4'
  GROUP  7 ( '/u20_MAINS/redo1/redo07a.log', '/u21_MAINS/redo2/redo07b.log' ) SIZE 500 M  REUSE,
  GROUP  8 ( '/u20_MAINS/redo1/redo08a.log', '/u21_MAINS/redo2/redo08b.log' ) SIZE 500 M  REUSE
sql statement: ALTER DATABASE ADD LOGFILE


  INSTANCE 'i5'
  GROUP  9 ( '/u20_MAINS/redo1/redo09a.log', '/u21_MAINS/redo2/redo09b.log' ) SIZE 500 M  REUSE,
  GROUP 10 ( '/u21_MAINS/redo2/redo10b.log', '/u20_MAINS/redo1/redo10a.log' ) SIZE 500 M  REUSE

contents of Memory Script:
{
   set newname for tempfile  2 to
 "/u25_TEMP/MAINS/temp01.dbf";
   switch clone tempfile all;
   catalog clone datafilecopy  "/u10_MAINS/oradata/sysaux01.dbf",
 "/u10_MAINS/oradata/undotbs05.dbf",
 "/u10_MAINS/oradata/users01.dbf",
 "/u10_MAINS/oradata/MST_01.dbf",
 "/u10_MAINS/oradata/MSTIDX_01.dbf",
 "/u10_MAINS/oradata/MSTLOB_01.dbf",
 "/u10_MAINS/oradata/jiraext01.dbf",
 "/u10_MAINS/oradata/jira01.dbf",
 "/u10_MAINS/oradata/undotbs03.dbf",
 "/u10_MAINS/oradata/undotbs04.dbf";
   switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 2 to /u25_TEMP/MAINS/temp01.dbf in control file

cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/sysaux01.dbf RECID=1 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/undotbs05.dbf RECID=2 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/users01.dbf RECID=3 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/MST_01.dbf RECID=4 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/MSTIDX_01.dbf RECID=5 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/MSTLOB_01.dbf RECID=6 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/jiraext01.dbf RECID=7 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/jira01.dbf RECID=8 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/undotbs03.dbf RECID=9 STAMP=831006912
cataloged datafile copy
datafile copy file name=/u10_MAINS/oradata/undotbs04.dbf RECID=10 STAMP=831006912

datafile 2 switched to datafile copy
input datafile copy RECID=1 STAMP=831006912 file name=/u10_MAINS/oradata/sysaux01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=2 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs05.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=3 STAMP=831006912 file name=/u10_MAINS/oradata/users01.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=4 STAMP=831006912 file name=/u10_MAINS/oradata/MST_01.dbf
datafile 7 switched to datafile copy
input datafile copy RECID=5 STAMP=831006912 file name=/u10_MAINS/oradata/MSTIDX_01.dbf
datafile 8 switched to datafile copy
input datafile copy RECID=6 STAMP=831006912 file name=/u10_MAINS/oradata/MSTLOB_01.dbf
datafile 9 switched to datafile copy
input datafile copy RECID=7 STAMP=831006912 file name=/u10_MAINS/oradata/jiraext01.dbf
datafile 10 switched to datafile copy
input datafile copy RECID=8 STAMP=831006912 file name=/u10_MAINS/oradata/jira01.dbf
datafile 11 switched to datafile copy
input datafile copy RECID=9 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs03.dbf
datafile 12 switched to datafile copy
input datafile copy RECID=10 STAMP=831006912 file name=/u10_MAINS/oradata/undotbs04.dbf

contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/09/2013 03:15:59
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-38856: cannot mark instance UNNAMED_INSTANCE_4 (redo thread 4) as enabled

RMAN> exit


Got the above error during the ‘alter database open resetlogs’. I guess most of us would know how to resolve the above error. Refer note 334899.1 to use “_no_recovery_through_resetlogs=TRUE” and then issue a manual ‘alter database open resetlogs’ via sqlplus. 

At this point we have a cloned database instance. Remove all hidden parameters and add/remove instances as required.



1 comment:

  1. Hi Nelson,

    helpful research... I am also facing the same problem but I am using ASM. any idea.

    Thanks
    Masood

    ReplyDelete