Troubleshooting RapidClone issues with Oracle Applications 11i (Doc ID 364565.1)
Modified: 23-Mar-2013 Type: HOWTO
In this Document
Goal
Fix
Section 1. Before starting to clone
Section 2. Preparing Source System
Section 3. Copy the files to the Target system
Section 4. Configure Target system
Appendix A: Common Issues when cloning the Database Tier
How to prevent DB recovery issues when running adcfgclone
Troubleshooting DB recovery issues while running adcfgclone
References
APPLIES TO:
Oracle Applications Manager - Version 11.5.2 to 11.5.10.2 [Release 11.5 to 11.5.10]
Information in this document applies to any platform.
Checked for relevance on 23-MAR-2013
GOAL
This Note is NOT a replacement for Note:230672.1.
Note:230672.1 must be followed when performing a clone of Oracle Applications 11i.
This troubleshooting note is intended as a guide to help determine the cause of some problems that may arise during a Rapid Clone.
The idea is to provide a detailed explanation into what each step is trying to achieve making it easier to identify which part of the clone has failed and why.
Locations of log files are listed, which should be uploaded when reporting a cloning issue to Oracle Support.
FIX
Section 1. Before starting to clone
The list of pre-requisite patches can be found on Note:230672.1
The first thing to do is check is that all pre-requisite patches have been applied.
Once this is done, ensure the latest code and templates is being used by applying latest Rapid Clone patch.
Check Note 230672.1 for the latest Rapid Clone patch.
Important: There are several steps to applying a rapid clone patch. Checking the recently applied patch list, or querying the AD_BUGS table, is not sufficient in
verifying that the latest cloning code exists.
Important: When applying a Rapid Clone patch using adpatch, ensure all of the following steps are performed:-
Apply the RapidClone patch using adpatch to Apps Tier nodes.
Run AutoConfig on all Apps Tier nodes
This will ensure the new templates supplied by the patch are instantiated.
Rapid Clone scripts, such as adpreclone.pl, are also instantiated using the latest templates
> $COMMON_TOP/admin/scripts/$CONTEXT_NAME/adautocfg.sh appspass=<appspwd>
Copy the new templates and code from the patch to the DB Tier
Applying the Rapid Clone using adpatch will only deliver the templates and java code to the Apps Tiers, not the DB Tier, so this step is to collect
the required files and copy them to the DB Tier:-
> perl $AD_TOP/bin/admkappsutil.pl
This will collect the required templates and code,used by AutoConfig on the DB tier, and compress them into the
$APPL_TOP/admin/out/appsutil.zip file.
Copy and extract the appsutil.zip to the RDBMS ORACLE_HOME
> unzip -o appsutil.zip
Run AutoConfig on the DB Tier
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 2/7
The new templates,scripts and java code are delivered by appsutil.zip to the DB Tier.
However, these still need to instantiated into the configuration files and scripts.
To do this, run AutoConfig on the DB Tier.
RapidClone scripts such as adpreclone.pl are also instantiated using the latest templates.
> $ORACLE_HOME/appsutil/bin/adconfig.sh contextfile=$CONTEXT_FILE appspass=apps
Once this step has been completed successfully, all the java .class files under
RDBMS $ORACLE_HOME/appsutil/java/oracle
should be identical to those under
$JAVA_TOP/oracle.
Important: If it is not immediately evident that the appsutil.zip steps above have been followed, after applying a new AutoConfig or Rapid Clone patch,
then do the steps again.
Failure to do this may result in issues that have been resolved in a later version of the software.
Section 2. Preparing Source System
Rapid Clone Commands:
DB Tier :
> perl adpreclone.pl dbTier
Apps Tier(s) :
> perl adpreclone.pl appsTier
The preparation phase collects information about the source system, creates a cloning stage area, and generates templates and drivers. All of these
are to reconfigure the instance on a Target machine.
Preclone will do the following:
Convert Symbolic links
All symbolic links pointing to a static path will be converted into relative paths
Create templates
Any files under the $ORACLE_HOME that contain system specific information, will be replicated and converted into a template. These templates
are placed into the $ORACLE_HOME/appsutil/template directory.
Create driver(s)
A driver file, relating to these new templates is created called instconf.drv. This contains a list of all the templates and their locations, and the
destination configuration files that these templates will create.
This driver file is called instconf.drv and is placed into directory
$ORACLE_HOME/appsutil/driver
For the appsTier, both the 8.0.6 and iAS ORACLE_HOMES will have the appsutil/template and appsutil/driver directories once the preclone
stage is completed
For the dbTier, these directories will already exist, but new templates and the new instconf.drv will be added
For more information on the instconf.drv and potential problems, please refer to Document 1339094.1 - Troubleshooting "instconf.drv"
Failures in 11i Rapid Clone
Create Stage area
A clone stage is created containing the required java code and scripts to reconfigure the instace on the Target machine
Rapid Clone stage area:
dbTier : $ORACLE_HOME/appsutil/clone
appsTier(s) - $COMMON_TOP/clone
The stage area(s) consist of the following directories:-
jre used to run the java code on the Target machine.
bincontains the RapidClone scripts that can be run on the Target machine:-
adclone.pl is the main cloning script
adcfgclone.pl is used to configure the Target system, this calls adclone.pl
adclonectx.pl is used to clone a Source XML file manually
adaddnode.pl is used to add a new node to the Patch History tables
adchkutl.sh checks for existence of require O/S utils, cc, make, ar and ld
jlib contains all the Rapid Clone java code, jdbc libraries etc
context contains templates used for a Target XML file
data(Database Tier only) contains the driver file, and templates used to generate the control file SQL script
adcrdb.zip contains the template and list of datafiles on the Source
addbhomsrc.xml contains information on the datafile mount points of the Source
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 3/7
appl (Applications Tier only) this is used when merging appltops, i.e Multi-node to Single node cloning
Important: The information in the stage area is only updated when running adpreclone.pl.
Whenever an AutoConfig or Rapid Clone patch is applied, and AutoConfig executed, it will not automatically update the stage area. Only
adpreclone.pl will do this. The script adpreclone.pl must be run (after running AutoConfig) to ensure new code, templates and scripts are copied
into the stage area before performing a clone.
If this is not done, then the code used to configure the Target system will not be the code from the latest patches applied. This must be done on
all nodes, db and appsTier(s).
Executing adpreclone.pl will create a log file:-
Rapid Clone:
dbTier : $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_xxxxxx.log
appsTier : $APPL_TOP/admin/$CONTEXT_NAME/log/StageAppsTier_xxxxxx.log
Important: These log files will contain the $Header version of the class file used.
Ensure this is the version delivered via the latest Rapid Clone patch.
If it is not the latest version, ask customer to follow "Section 1: Before starting to clone" above, and rerun adpreclone.pl steps.
Once this adpreclone.pl step has been completed successfully, all the java .class files under the following directories should be identical to those under
$JAVA_TOP/oracle :
RDBMS $ORACLE_HOME/appsutil/java/oracle
RDBMS $ORACLE_HOME/appsutil/clone/jlib/java/oracle
$COMMON_TOP/clone/jlib/java/oracle
If they are not, ensure to follow the pre-req AutoConfig steps, and rerun adpreclone on all DB and Apps Tier nodes.
Highly recommended: It is highly recommended customers rename any existing clone stage area, and allow RapidClone to create a new one. This
should avoid any known issues that are caused by old redundant files from a previous clone.
On the DB Tier:-
Rename RDBMS $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
On the Apps Tier, run "perl $AD_TOP/bin/admkappsutil.pl"
Copy the appsutil.zip to the RDBMS $ORACLE_HOME and unzip
Copy the Context file from RDBMS $ORACLE_HOME/appsutil_old to the new $ORACLE_HOME/appsutil
Run AutoConfig on the DB Tier
Do the "Prepare Source system" steps i.e "perl adpreclone.pl dbTier"
On the Apps Tier:-
Rename $COMMON_TOP/clone to $COMMON_TOP/clone_old
Rename 8.0.6 $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
Rename iAS $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
Do the "Prepare Source system" steps i.e "perl adpreclone.pl appsTier"
Section 3. Copy the files to the Target system
When copying files to the Target, ensure the application tier files copied to the target system are owned by the target APPLMGR user, and that the
database tier files are owned by the ORACLE user.
When copying the datafiles, ensure the DB is shutdown cleanly prior to the copy.
If there is any DB activity while the datafiles are copied, the clone will fail to recreate the control files with error:-
ORA-01194: file <filename> needs more recovery to be consistent
For more potential recovery errors, refer to Appendix A of this document
If a clone must be performed using a hot backup, then follow the alternative cloning steps in Appendix B of Note:230672.1.
Section 4. Configure Target system
Rapid Clone Commands:
DB Tier :
> perl adcfgclone.pl dbTier
Apps Tier:
> perl adcfgclone.pl appsTier
The configuration phase will reconfigure the copied instance after gathering the required information.
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 4/7
adchkutl.sh
The first step performed is the OS utility check (adchkutl.sh).
This ensures make, cc, ld and ar executables are available in the PATH.
CloneContext.class
This will create the Target Context file from the values entered on the screen.
For a full list of the questions, see Note:216664.1
All the questions, values entered, results of the port availability check etc will be logged in a log file located in same location as the
adcfgclone.pl script:-
dbTier : $ORACLE_HOME/appsutil/clone/bin
appsTier(s) : $COMMON_TOP/clone/bin
Log files for this phase:-
dbTier: $TMP/CloneContext_xxxxxx.log
appsTier(s): $TMP/CloneContext_xxxxxx.log
Both Tiers: $TMP/adclonectx.err
Note: With earlier RapidClone code, the logs were generated as follows:-
dbTier: $ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
appsTier(s): $COMMON_TOP/clone/bin/CloneContext_xxxxxx.log
Both Tiers: $TMP/adclonectx.err
If any of the variables in the Target system Context file appear to be incorrect, then check this log file to see how the incorrect value arose.
ApplyDBTier.class/ApplyAppsTier.class
Once the XML is created, or passed as a parameter on the command line, then the configuration phase begins.
This part will instantiate all the templates created during the preparation phase under:
$ORACLE_HOME/appsutil/template.
Templates to instantiate are located in driver file
$ORACLE_HOME/appsutil/driver/instconf.drv
Log files for this phase:-
dbTier: $ORACLE_HOME/appsutil/log/<Target Context Name>/ApplyDBTier_xxxxxx.log
appsTier(s): $APPL_TOP/admin/<Target Context Name>/log/ApplyAppsTier_xxxxxx.log
Relink the ORACLE_HOME(s)
The executables in the ORACLE_HOME(s) will be relinked during the configuration phase.
The commands used, and results of the relinking will displayed in log file :
$ORACLE_HOME/install/make.log
Create the Control files (DB Tier only)
Once the templates are instantiated, and the ORACLE_HOME is relinked, then the control files can be recreated.
The templates used here are located in
$ORACLE_HOME/appsutil/clone/data/stage/adcrdb.zip
The files in this zip file are used to create a SQL file called
adcrdbclone.sql
This above SQL script is called from a shell script
adcrdb.sh.
Both of the above scripts can be found under
$ORACLE_HOME/appsutil/install/$CONTEXT_NAME
If there are any problems creating the control files, then check the log files :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/adcrdb_<SID>.txt
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 5/7
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
If this error is encountered:-
ORA-01194: file <filename> needs more recovery to be consistent
refer to "Section 3 - Copy the files to the Target system" above on why this problem occurs.
AutoConfig
The next phase will run AutoConfig using use the usual templates and drivers associated with AutoConfig on db or appsTier.
The following log files will contain the information on the results of running AutoConfig
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTier_xxxxxx.log
Update the oraInventory (OracleHomeCloner.class)
Once the Target instance is configured, it will update the Global Inventory will the locations of the new ORACLE_HOMES
The global inventory is found by checking the Inventory Pointer file.
This can be found as follows :
AIX and LINUX under : /etc/oraInst.loc
Other UNIX machines : /vat/opt/oracle/oraInst.loc
WINDOWS (Registry): HKEY_LOCAL_MACHINE\Software\Oracle\INST_LOC
This Inventory pointer points to the location of the Global Inventory.
The global inventory will be updated with the locations of the new iAS and RDBMS ORACLE_HOMES
Any problems with this step will be detailed in the log files:-
<path to oraInventory>/logs/OracleHomeCloner_<MMDDHHMM>.log
<path to oraInventory>/logs/appsapps<Year>-<Mon>-<Day>-<Hour>-<Min>-<Sec>-<AM/PM>.log
If there are problems with the Global Inventory refer to Note 295185.1
Appendix A: Common Issues when cloning the Database Tier
How to prevent DB recovery issues when running adcfgclone
These errors can be prevented using the following steps
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: <file>.dbf
ORA-01503: CREATE CONTROLFILE failed
ORA-01159: file is not from same database as previous files - wrong database id
Note: The steps in this section apply to cold backups only. To perform Cloning using HOT Backup, refer the following Document 760772.1 Cloning Oracle
Application 11i /R12 with Rapid Clone - Database (9i/10g/11g) Using Hot Backup on Open Database
To perform Cloning using Cold Backup, make sure these steps are followed to avoid any recovery issues while performing cloning at Target:
Follow the "Prepare the source system database tier for cloning" steps in the cloning notes Note 230672.1(11i) or Note 406982.1(R12) . These are as
follows:
Run autoconfig
Run adpreclone on dbTier
Stop DB and its listener processes
Shutdown the Database with (Normal, Immediate ) Mode
Bring down the application services
Crosscheck with alert log whether DB has been shutdown completely
Run the following OS command to confirm/crosscheck that no DB process running , if any DB processes running , terminate them manually
ps -ef | grep pmon
Copy all the datafiles from Source to Target
On Target System, at mount stage on Database, run the following SQL :
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 6/7
set pagesize 20000
set linesize 180
set pause off
set serveroutput on
set feedback on
set echo on
set numformat 999999999999999
Spool recovery_info.txt
select substr(name, 1, 50), status from v$datafile;
select substr(name,1,40), recover, fuzzy, checkpoint_change# from v$datafile_header;
select GROUP#,substr(member,1,60) from v$logfile;
select * from v$recover_file;
select distinct status from v$backup;
select hxfil FILENUMBER,fhsta STATUS,fhscn SCN,fhrba_Seq SEQUENCE from x$kcvfh;
select distinct (fuzzy) from v$datafile_header;
select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN SCN, FHTNM
TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
spool off
exit
Points to consider:
All Datafiles STATUS should be showing ONLINE , if any datafile is showing RECOVER, then you need to recover at source and redo the
copy again
The column checkpoint_change# of each datafiles different from the checkpoint_change# of all the other datafiles. For a cold backup,
all the checkpoint_change# of the datafiles should be the same
v$recover_file should not show any datafiles need to be recovered
Check each datafile status and make sure that none of datafiles shows "RECOVER" Make sure that the FHSTA column does not show
status "4", if so, status 4 means the datafile is online fuzzy, i.e, the datafile was backed when the database was open.
Run the clone configuration command:
cd <RDBMS ORACLE_HOME>/appsutil/clone/bin
perl adcfgclone.pl dbTier
Troubleshooting DB recovery issues while running adcfgclone
This section shows how to troubleshoot the following error:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '<file>.dbf'
Run the following to verify the backup
In target system, mount the database and run the following SQL :
set pagesize 20000
set linesize 180
set pause off
set serveroutput on
set feedback on
set echo on
set numformat 999999999999999
Spool recovery_info.txt
select substr(name, 1, 50), status from v$datafile;
select substr(name,1,40), recover, fuzzy, checkpoint_change# from v$datafile_header;
select GROUP#,substr(member,1,60) from v$logfile;
select * from v$recover_file;
select distinct status from v$backup;
select hxfil FILENUMBER,fhsta STATUS,fhscn SCN,fhrba_Seq SEQUENCE from x$kcvfh;
select distinct (fuzzy) from v$datafile_header;
select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN SCN, FHTNM
TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
spool off
exit
For a cold backup, all the checkpoint_change# of the datafiles should be the same.
If there are any differences in checkpoint_change# on any of the datafiles, the datafile needs to be recovered at source, the database
shutdown and the files the copied again to the Target.
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 7/7
If the checkpoint_change# of each datafile is the same, then try the following option:
SQL> startup mount;
SQL> recover database using backup controlfile until cancel;
-- type CANCEL
SQL> alter database open resetlogs;
If this is fails, then files will need to be recopied from the source.
If the select from X$KCVFH shows status as "4" for any of the datafiles, this indicates the datafile is "online fuzzy", i.e, the datafile was
backed up when the database was open.
For example:
SQL> select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN
2 SCN, FHTNM
3 TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
FILE_NUM FILE_NAME TYPE VALIDITY
SCN TABLESPACE_NAME STATUS SEQUENCE
---------------- ---------------------------------------- ---------------- -------------
---
---------------- ------------------------------ ---------------- ----------------
1 <DATA_TOP>/system01.dbf
3 0
2371803541 SYSTEM 4 124273
2 <DATA_TOP>/system02.dbf
3 0
2371803541 SYSTEM 4 124273
3 <DATA_TOP>/system03.dbf
3 0
2371803541 SYSTEM 4 124273
4 <DATA_TOP>/system04.dbf
3 0
2371803541 SYSTEM 4 124273
In this case the datafiles need to be recopied from the source again, making sure the database was shutdown cleanly prior to the
copy.
REFERENCES
NOTE:1339094.1 - Troubleshooting "instconf.drv" Failures in 11i Rapid Clone
NOTE:216664.1 - FAQ: Cloning Oracle Applications Release 11i
NOTE:230672.1 - Cloning Oracle Applications Release 11i with Rapid Clone
NOTE:295185.1 - How to Recreate the Global oraInventory
NOTE:364565.1 - Troubleshooting RapidClone issues with Oracle Applications 11i
NOTE:406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone
NOTE:760772.1 - Cloning Oracle Application 11i /R12 with Rapid Clone - Database (9i/10g/11g) Using Hot Backup on Open Database
Modified: 23-Mar-2013 Type: HOWTO
In this Document
Goal
Fix
Section 1. Before starting to clone
Section 2. Preparing Source System
Section 3. Copy the files to the Target system
Section 4. Configure Target system
Appendix A: Common Issues when cloning the Database Tier
How to prevent DB recovery issues when running adcfgclone
Troubleshooting DB recovery issues while running adcfgclone
References
APPLIES TO:
Oracle Applications Manager - Version 11.5.2 to 11.5.10.2 [Release 11.5 to 11.5.10]
Information in this document applies to any platform.
Checked for relevance on 23-MAR-2013
GOAL
This Note is NOT a replacement for Note:230672.1.
Note:230672.1 must be followed when performing a clone of Oracle Applications 11i.
This troubleshooting note is intended as a guide to help determine the cause of some problems that may arise during a Rapid Clone.
The idea is to provide a detailed explanation into what each step is trying to achieve making it easier to identify which part of the clone has failed and why.
Locations of log files are listed, which should be uploaded when reporting a cloning issue to Oracle Support.
FIX
Section 1. Before starting to clone
The list of pre-requisite patches can be found on Note:230672.1
The first thing to do is check is that all pre-requisite patches have been applied.
Once this is done, ensure the latest code and templates is being used by applying latest Rapid Clone patch.
Check Note 230672.1 for the latest Rapid Clone patch.
Important: There are several steps to applying a rapid clone patch. Checking the recently applied patch list, or querying the AD_BUGS table, is not sufficient in
verifying that the latest cloning code exists.
Important: When applying a Rapid Clone patch using adpatch, ensure all of the following steps are performed:-
Apply the RapidClone patch using adpatch to Apps Tier nodes.
Run AutoConfig on all Apps Tier nodes
This will ensure the new templates supplied by the patch are instantiated.
Rapid Clone scripts, such as adpreclone.pl, are also instantiated using the latest templates
> $COMMON_TOP/admin/scripts/$CONTEXT_NAME/adautocfg.sh appspass=<appspwd>
Copy the new templates and code from the patch to the DB Tier
Applying the Rapid Clone using adpatch will only deliver the templates and java code to the Apps Tiers, not the DB Tier, so this step is to collect
the required files and copy them to the DB Tier:-
> perl $AD_TOP/bin/admkappsutil.pl
This will collect the required templates and code,used by AutoConfig on the DB tier, and compress them into the
$APPL_TOP/admin/out/appsutil.zip file.
Copy and extract the appsutil.zip to the RDBMS ORACLE_HOME
> unzip -o appsutil.zip
Run AutoConfig on the DB Tier
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 2/7
The new templates,scripts and java code are delivered by appsutil.zip to the DB Tier.
However, these still need to instantiated into the configuration files and scripts.
To do this, run AutoConfig on the DB Tier.
RapidClone scripts such as adpreclone.pl are also instantiated using the latest templates.
> $ORACLE_HOME/appsutil/bin/adconfig.sh contextfile=$CONTEXT_FILE appspass=apps
Once this step has been completed successfully, all the java .class files under
RDBMS $ORACLE_HOME/appsutil/java/oracle
should be identical to those under
$JAVA_TOP/oracle.
Important: If it is not immediately evident that the appsutil.zip steps above have been followed, after applying a new AutoConfig or Rapid Clone patch,
then do the steps again.
Failure to do this may result in issues that have been resolved in a later version of the software.
Section 2. Preparing Source System
Rapid Clone Commands:
DB Tier :
> perl adpreclone.pl dbTier
Apps Tier(s) :
> perl adpreclone.pl appsTier
The preparation phase collects information about the source system, creates a cloning stage area, and generates templates and drivers. All of these
are to reconfigure the instance on a Target machine.
Preclone will do the following:
Convert Symbolic links
All symbolic links pointing to a static path will be converted into relative paths
Create templates
Any files under the $ORACLE_HOME that contain system specific information, will be replicated and converted into a template. These templates
are placed into the $ORACLE_HOME/appsutil/template directory.
Create driver(s)
A driver file, relating to these new templates is created called instconf.drv. This contains a list of all the templates and their locations, and the
destination configuration files that these templates will create.
This driver file is called instconf.drv and is placed into directory
$ORACLE_HOME/appsutil/driver
For the appsTier, both the 8.0.6 and iAS ORACLE_HOMES will have the appsutil/template and appsutil/driver directories once the preclone
stage is completed
For the dbTier, these directories will already exist, but new templates and the new instconf.drv will be added
For more information on the instconf.drv and potential problems, please refer to Document 1339094.1 - Troubleshooting "instconf.drv"
Failures in 11i Rapid Clone
Create Stage area
A clone stage is created containing the required java code and scripts to reconfigure the instace on the Target machine
Rapid Clone stage area:
dbTier : $ORACLE_HOME/appsutil/clone
appsTier(s) - $COMMON_TOP/clone
The stage area(s) consist of the following directories:-
jre used to run the java code on the Target machine.
bincontains the RapidClone scripts that can be run on the Target machine:-
adclone.pl is the main cloning script
adcfgclone.pl is used to configure the Target system, this calls adclone.pl
adclonectx.pl is used to clone a Source XML file manually
adaddnode.pl is used to add a new node to the Patch History tables
adchkutl.sh checks for existence of require O/S utils, cc, make, ar and ld
jlib contains all the Rapid Clone java code, jdbc libraries etc
context contains templates used for a Target XML file
data(Database Tier only) contains the driver file, and templates used to generate the control file SQL script
adcrdb.zip contains the template and list of datafiles on the Source
addbhomsrc.xml contains information on the datafile mount points of the Source
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 3/7
appl (Applications Tier only) this is used when merging appltops, i.e Multi-node to Single node cloning
Important: The information in the stage area is only updated when running adpreclone.pl.
Whenever an AutoConfig or Rapid Clone patch is applied, and AutoConfig executed, it will not automatically update the stage area. Only
adpreclone.pl will do this. The script adpreclone.pl must be run (after running AutoConfig) to ensure new code, templates and scripts are copied
into the stage area before performing a clone.
If this is not done, then the code used to configure the Target system will not be the code from the latest patches applied. This must be done on
all nodes, db and appsTier(s).
Executing adpreclone.pl will create a log file:-
Rapid Clone:
dbTier : $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_xxxxxx.log
appsTier : $APPL_TOP/admin/$CONTEXT_NAME/log/StageAppsTier_xxxxxx.log
Important: These log files will contain the $Header version of the class file used.
Ensure this is the version delivered via the latest Rapid Clone patch.
If it is not the latest version, ask customer to follow "Section 1: Before starting to clone" above, and rerun adpreclone.pl steps.
Once this adpreclone.pl step has been completed successfully, all the java .class files under the following directories should be identical to those under
$JAVA_TOP/oracle :
RDBMS $ORACLE_HOME/appsutil/java/oracle
RDBMS $ORACLE_HOME/appsutil/clone/jlib/java/oracle
$COMMON_TOP/clone/jlib/java/oracle
If they are not, ensure to follow the pre-req AutoConfig steps, and rerun adpreclone on all DB and Apps Tier nodes.
Highly recommended: It is highly recommended customers rename any existing clone stage area, and allow RapidClone to create a new one. This
should avoid any known issues that are caused by old redundant files from a previous clone.
On the DB Tier:-
Rename RDBMS $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
On the Apps Tier, run "perl $AD_TOP/bin/admkappsutil.pl"
Copy the appsutil.zip to the RDBMS $ORACLE_HOME and unzip
Copy the Context file from RDBMS $ORACLE_HOME/appsutil_old to the new $ORACLE_HOME/appsutil
Run AutoConfig on the DB Tier
Do the "Prepare Source system" steps i.e "perl adpreclone.pl dbTier"
On the Apps Tier:-
Rename $COMMON_TOP/clone to $COMMON_TOP/clone_old
Rename 8.0.6 $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
Rename iAS $ORACLE_HOME/appsutil to $ORACLE_HOME/appsutil_old
Do the "Prepare Source system" steps i.e "perl adpreclone.pl appsTier"
Section 3. Copy the files to the Target system
When copying files to the Target, ensure the application tier files copied to the target system are owned by the target APPLMGR user, and that the
database tier files are owned by the ORACLE user.
When copying the datafiles, ensure the DB is shutdown cleanly prior to the copy.
If there is any DB activity while the datafiles are copied, the clone will fail to recreate the control files with error:-
ORA-01194: file <filename> needs more recovery to be consistent
For more potential recovery errors, refer to Appendix A of this document
If a clone must be performed using a hot backup, then follow the alternative cloning steps in Appendix B of Note:230672.1.
Section 4. Configure Target system
Rapid Clone Commands:
DB Tier :
> perl adcfgclone.pl dbTier
Apps Tier:
> perl adcfgclone.pl appsTier
The configuration phase will reconfigure the copied instance after gathering the required information.
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 4/7
adchkutl.sh
The first step performed is the OS utility check (adchkutl.sh).
This ensures make, cc, ld and ar executables are available in the PATH.
CloneContext.class
This will create the Target Context file from the values entered on the screen.
For a full list of the questions, see Note:216664.1
All the questions, values entered, results of the port availability check etc will be logged in a log file located in same location as the
adcfgclone.pl script:-
dbTier : $ORACLE_HOME/appsutil/clone/bin
appsTier(s) : $COMMON_TOP/clone/bin
Log files for this phase:-
dbTier: $TMP/CloneContext_xxxxxx.log
appsTier(s): $TMP/CloneContext_xxxxxx.log
Both Tiers: $TMP/adclonectx.err
Note: With earlier RapidClone code, the logs were generated as follows:-
dbTier: $ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
appsTier(s): $COMMON_TOP/clone/bin/CloneContext_xxxxxx.log
Both Tiers: $TMP/adclonectx.err
If any of the variables in the Target system Context file appear to be incorrect, then check this log file to see how the incorrect value arose.
ApplyDBTier.class/ApplyAppsTier.class
Once the XML is created, or passed as a parameter on the command line, then the configuration phase begins.
This part will instantiate all the templates created during the preparation phase under:
$ORACLE_HOME/appsutil/template.
Templates to instantiate are located in driver file
$ORACLE_HOME/appsutil/driver/instconf.drv
Log files for this phase:-
dbTier: $ORACLE_HOME/appsutil/log/<Target Context Name>/ApplyDBTier_xxxxxx.log
appsTier(s): $APPL_TOP/admin/<Target Context Name>/log/ApplyAppsTier_xxxxxx.log
Relink the ORACLE_HOME(s)
The executables in the ORACLE_HOME(s) will be relinked during the configuration phase.
The commands used, and results of the relinking will displayed in log file :
$ORACLE_HOME/install/make.log
Create the Control files (DB Tier only)
Once the templates are instantiated, and the ORACLE_HOME is relinked, then the control files can be recreated.
The templates used here are located in
$ORACLE_HOME/appsutil/clone/data/stage/adcrdb.zip
The files in this zip file are used to create a SQL file called
adcrdbclone.sql
This above SQL script is called from a shell script
adcrdb.sh.
Both of the above scripts can be found under
$ORACLE_HOME/appsutil/install/$CONTEXT_NAME
If there are any problems creating the control files, then check the log files :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/adcrdb_<SID>.txt
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 5/7
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
If this error is encountered:-
ORA-01194: file <filename> needs more recovery to be consistent
refer to "Section 3 - Copy the files to the Target system" above on why this problem occurs.
AutoConfig
The next phase will run AutoConfig using use the usual templates and drivers associated with AutoConfig on db or appsTier.
The following log files will contain the information on the results of running AutoConfig
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTier_xxxxxx.log
Update the oraInventory (OracleHomeCloner.class)
Once the Target instance is configured, it will update the Global Inventory will the locations of the new ORACLE_HOMES
The global inventory is found by checking the Inventory Pointer file.
This can be found as follows :
AIX and LINUX under : /etc/oraInst.loc
Other UNIX machines : /vat/opt/oracle/oraInst.loc
WINDOWS (Registry): HKEY_LOCAL_MACHINE\Software\Oracle\INST_LOC
This Inventory pointer points to the location of the Global Inventory.
The global inventory will be updated with the locations of the new iAS and RDBMS ORACLE_HOMES
Any problems with this step will be detailed in the log files:-
<path to oraInventory>/logs/OracleHomeCloner_<MMDDHHMM>.log
<path to oraInventory>/logs/appsapps<Year>-<Mon>-<Day>-<Hour>-<Min>-<Sec>-<AM/PM>.log
If there are problems with the Global Inventory refer to Note 295185.1
Appendix A: Common Issues when cloning the Database Tier
How to prevent DB recovery issues when running adcfgclone
These errors can be prevented using the following steps
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: <file>.dbf
ORA-01503: CREATE CONTROLFILE failed
ORA-01159: file is not from same database as previous files - wrong database id
Note: The steps in this section apply to cold backups only. To perform Cloning using HOT Backup, refer the following Document 760772.1 Cloning Oracle
Application 11i /R12 with Rapid Clone - Database (9i/10g/11g) Using Hot Backup on Open Database
To perform Cloning using Cold Backup, make sure these steps are followed to avoid any recovery issues while performing cloning at Target:
Follow the "Prepare the source system database tier for cloning" steps in the cloning notes Note 230672.1(11i) or Note 406982.1(R12) . These are as
follows:
Run autoconfig
Run adpreclone on dbTier
Stop DB and its listener processes
Shutdown the Database with (Normal, Immediate ) Mode
Bring down the application services
Crosscheck with alert log whether DB has been shutdown completely
Run the following OS command to confirm/crosscheck that no DB process running , if any DB processes running , terminate them manually
ps -ef | grep pmon
Copy all the datafiles from Source to Target
On Target System, at mount stage on Database, run the following SQL :
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 6/7
set pagesize 20000
set linesize 180
set pause off
set serveroutput on
set feedback on
set echo on
set numformat 999999999999999
Spool recovery_info.txt
select substr(name, 1, 50), status from v$datafile;
select substr(name,1,40), recover, fuzzy, checkpoint_change# from v$datafile_header;
select GROUP#,substr(member,1,60) from v$logfile;
select * from v$recover_file;
select distinct status from v$backup;
select hxfil FILENUMBER,fhsta STATUS,fhscn SCN,fhrba_Seq SEQUENCE from x$kcvfh;
select distinct (fuzzy) from v$datafile_header;
select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN SCN, FHTNM
TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
spool off
exit
Points to consider:
All Datafiles STATUS should be showing ONLINE , if any datafile is showing RECOVER, then you need to recover at source and redo the
copy again
The column checkpoint_change# of each datafiles different from the checkpoint_change# of all the other datafiles. For a cold backup,
all the checkpoint_change# of the datafiles should be the same
v$recover_file should not show any datafiles need to be recovered
Check each datafile status and make sure that none of datafiles shows "RECOVER" Make sure that the FHSTA column does not show
status "4", if so, status 4 means the datafile is online fuzzy, i.e, the datafile was backed when the database was open.
Run the clone configuration command:
cd <RDBMS ORACLE_HOME>/appsutil/clone/bin
perl adcfgclone.pl dbTier
Troubleshooting DB recovery issues while running adcfgclone
This section shows how to troubleshoot the following error:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '<file>.dbf'
Run the following to verify the backup
In target system, mount the database and run the following SQL :
set pagesize 20000
set linesize 180
set pause off
set serveroutput on
set feedback on
set echo on
set numformat 999999999999999
Spool recovery_info.txt
select substr(name, 1, 50), status from v$datafile;
select substr(name,1,40), recover, fuzzy, checkpoint_change# from v$datafile_header;
select GROUP#,substr(member,1,60) from v$logfile;
select * from v$recover_file;
select distinct status from v$backup;
select hxfil FILENUMBER,fhsta STATUS,fhscn SCN,fhrba_Seq SEQUENCE from x$kcvfh;
select distinct (fuzzy) from v$datafile_header;
select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN SCN, FHTNM
TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
spool off
exit
For a cold backup, all the checkpoint_change# of the datafiles should be the same.
If there are any differences in checkpoint_change# on any of the datafiles, the datafile needs to be recovered at source, the database
shutdown and the files the copied again to the Target.
2/7/2014 Document Display
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=4vbvzgz0b_120 7/7
If the checkpoint_change# of each datafile is the same, then try the following option:
SQL> startup mount;
SQL> recover database using backup controlfile until cancel;
-- type CANCEL
SQL> alter database open resetlogs;
If this is fails, then files will need to be recopied from the source.
If the select from X$KCVFH shows status as "4" for any of the datafiles, this indicates the datafile is "online fuzzy", i.e, the datafile was
backed up when the database was open.
For example:
SQL> select HXFIL File_num,substr(HXFNM,1,40) File_name,FHTYP Type,HXERR Validity, FHSCN
2 SCN, FHTNM
3 TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence from X$KCVFH;
FILE_NUM FILE_NAME TYPE VALIDITY
SCN TABLESPACE_NAME STATUS SEQUENCE
---------------- ---------------------------------------- ---------------- -------------
---
---------------- ------------------------------ ---------------- ----------------
1 <DATA_TOP>/system01.dbf
3 0
2371803541 SYSTEM 4 124273
2 <DATA_TOP>/system02.dbf
3 0
2371803541 SYSTEM 4 124273
3 <DATA_TOP>/system03.dbf
3 0
2371803541 SYSTEM 4 124273
4 <DATA_TOP>/system04.dbf
3 0
2371803541 SYSTEM 4 124273
In this case the datafiles need to be recopied from the source again, making sure the database was shutdown cleanly prior to the
copy.
REFERENCES
NOTE:1339094.1 - Troubleshooting "instconf.drv" Failures in 11i Rapid Clone
NOTE:216664.1 - FAQ: Cloning Oracle Applications Release 11i
NOTE:230672.1 - Cloning Oracle Applications Release 11i with Rapid Clone
NOTE:295185.1 - How to Recreate the Global oraInventory
NOTE:364565.1 - Troubleshooting RapidClone issues with Oracle Applications 11i
NOTE:406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone
NOTE:760772.1 - Cloning Oracle Application 11i /R12 with Rapid Clone - Database (9i/10g/11g) Using Hot Backup on Open Database
