Migration: Oracle Enterprise Manager
Cloud Control 12c R5
Moving Oracle Management Repository
(OMR) to New Server (11.2.0.4.6 - PSU)
Today's
demonstration is for data center migration from one location to other location,
where we have to plan migrate our existing setup to new environment. In our environment we have Oracle Enterprise
Manger Cloud Control - Release 5 (OEM12c
- R5) having management repository 11.2.0.4.6 (PSU) applied.
In first phase we are planning to all
management repository to new environment.
In order to achieve this task we are following one approach in list of
below.
1.
RMAN Backup restoration and
recovery
2.
Export and Import
3.
Data Guard with broker
4.
Cold Backup
In
this demonstration I am using approach 1 which is RMAN Backup restoration and
recovery.
In our
next post I will do this with the help of data guard which is coming soon.
In Second phase we will migrate our existing
management server to new environment in next post.
We are
considering that we have taken Level 0 RMAN backup and redo archive log till
level 0 backup.
Source Repository Server : rac126.ora.com
Target Repository Server : rac127.ora.com
Oracle Management Server : oem250.ora.com
Process will be looking for the
"emkey". This key needs to be placed inside the database so it can be
referenced during the Migration. If not available, then we will face issue
during process.
After that we will take RMAN
level 0 backup from existing repository server.
[oracle@oem250 config]$ emctl
status emkey
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Enterprise Manager Root (SYSMAN) Password :
The EMKey is
configured properly.
[oracle@oem250 config]$ emctl
config emkey -copy_to_repos
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter Enterprise
Manager Root (SYSMAN) Password :
The EMKey has
been copied to the Management Repository. This operation will cause the EMKey
to become unsecure.
After the
required operation has been completed, secure the EMKey by running "emctl
config emkey -remove_from_repos".
Now Stop OMS and management
agent on OMS Server. Here do not use
"all" option while stopping OMS.
[oracle@oem250 bin]$ emctl stop
oms
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Stopping
WebTier...
WebTier
Successfully Stopped
Stopping
Oracle Management Server...
Oracle
Management Server Successfully Stopped
Oracle
Management Server is Down
[oracle@oem250 bin]$ emctl stop
agent
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2014 Oracle Corporation. All
rights reserved.
Stopping agent
..... stopped.
[oracle@oem250
bin]$
After shutdown the OMS and management
Agent we will start database backup restoration on Target Repository Server
Create necessary directory on target database "rac127"
mkdir -p
/u01/app/oracle/admin/EMREP12C/adump
mkdir -p /u01/app/oracle/fast_recovery_area/EMREP12C/
mkdir -p
/u01/app/oracle/oradata/EMREP12C/
Go the backup location where Level 0 backup and
archivelog. Set the environment for new database and put database in nomount
and restore spfile , and then after restore controlfile, catalog backup and
start restoration and recovery and open database in resetlogs.
RMAN> startup force nomount
Oracle instance
started
Total System
Global Area 2137886720 bytes
Fixed
Size 2254952 bytes
Variable
Size 1493174168 bytes
Database
Buffers 637534208 bytes
Redo
Buffers 4923392 bytes
RMAN>
restore controlfile from '/Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00'
;
Starting
restore at 26-SEP-15
using channel
ORA_DISK_1
channel
ORA_DISK_1: restoring control file
channel
ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file
name=/u01/app/oracle/oradata/EMREP12C/control01.ctl
output file name=/u01/app/oracle/fast_recovery_area/EMREP12C/control02.ctl
Finished
restore at 26-SEP-15
RMAN> alter database mount ;
database
mounted
released
channel: ORA_DISK_1
RMAN> catalog start with '/Shared_NAS/BACKUP/EMREP12c/'
;
Starting
implicit crosscheck backup at 26-SEP-15
allocated
channel: ORA_DISK_1
channel
ORA_DISK_1: SID=21 device type=DISK
Crosschecked 3
objects
Finished
implicit crosscheck backup at 26-SEP-15
Starting
implicit crosscheck copy at 26-SEP-15
using channel
ORA_DISK_1
Crosschecked 2
objects
Finished
implicit crosscheck copy at 26-SEP-15
searching for
all files in the recovery area
cataloging
files...
no files
cataloged
searching for
all files that match the pattern /Shared_NAS/BACKUP/EMREP12c/
List of Files
Unknown to the Database
=====================================
File Name: /Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_195_byyqz2vh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_197_byyr0d1s_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_196_byyr0cqv_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_194_byx5q8wd_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_193_byvgxqsh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_192_bysbor59_.arc
Do you really
want to catalog the above files (enter YES or NO)? yes
cataloging
files...
cataloging
done
List of
Cataloged Files
=======================
File Name: /Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_195_byyqz2vh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_197_byyr0d1s_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_196_byyr0cqv_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_194_byx5q8wd_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_193_byvgxqsh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_192_bysbor59_.arc
RMAN> restore database ;
Starting
restore at 26-SEP-15
using channel
ORA_DISK_1
channel
ORA_DISK_1: starting datafile backup set restore
channel
ORA_DISK_1: specifying datafile(s) to restore from backup set
channel
ORA_DISK_1: restoring datafile 00001 to
/u01/app/oracle/oradata/EMREP12C/system01.dbf
channel
ORA_DISK_1: restoring datafile 00002 to
/u01/app/oracle/oradata/EMREP12C/sysaux01.dbf
channel
ORA_DISK_1: restoring datafile 00003 to
/u01/app/oracle/oradata/EMREP12C/undotbs01.dbf
channel
ORA_DISK_1: restoring datafile 00004 to
/u01/app/oracle/oradata/EMREP12C/users01.dbf
channel
ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/EMREP12C/example01.dbf
channel
ORA_DISK_1: restoring datafile 00006 to
/u01/app/oracle/oradata/EMREP12C/mgmt_depot.dbf
channel
ORA_DISK_1: restoring datafile 00007 to
/u01/app/oracle/oradata/EMREP12C/mgmt.dbf
channel
ORA_DISK_1: restoring datafile 00008 to
/u01/app/oracle/oradata/EMREP12C/mgmt_ad4j.dbf
channel
ORA_DISK_1: reading from backup piece /Shared_NAS/BACKUP/EMREP12c/EMREP12c_0mqgmjnq_1_1
channel
ORA_DISK_1: piece handle=/Shared_NAS/BACKUP/EMREP12c/EMREP12c_0mqgmjnq_1_1
tag=TAG20150909T034602
channel
ORA_DISK_1: restored backup piece 1
channel
ORA_DISK_1: restore complete, elapsed time: 00:03:35
Finished
restore at 26-SEP-15
RMAN> recover database ;
Starting
recover at 26-SEP-15
using channel
ORA_DISK_1
starting media
recovery
archived log
for thread 1 with sequence 198 is already on disk as file /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
archived log
file name=/Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
thread=1 sequence=198
unable to find
archived log
archived log
thread=1 sequence=199
RMAN-00571:
===========================================================
RMAN-00569:
=============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571:
===========================================================
RMAN-03002:
failure of recover command at 09/26/2015 23:52:11
RMAN-06054:
media recovery requesting unknown archived log for thread 1 with sequence 199
and starting SCN of 8855553
RMAN> alter database open resetlogs
;
database
opened
[oracle@rac127 admin]$ sqlplus / as sysdba
SQL*Plus:
Release 11.2.0.4.0 Production on Sat Sep 26 23:58:54 2015
Copyright (c)
1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle
Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the
Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select name, open_mode
from v$database ;
NAME OPEN_MODE
---------
--------------------
EMREP12C READ WRITE
SQL> select flashback_on from
v$database ;
FLASHBACK_ON
------------------
NO
Reconfigure Existing OMS For New Repository Database
Now Start
the OMS and agent on the existing OMS server. Startup will fail as
expected, as I have not yet reconfigured the repository on newly migrated
repository server.
[oracle@oem250 bin]$ . oraenv
ORACLE_SID = [agent] ? oms
The Oracle
base for ORACLE_HOME=/u01/app/oracle/Middleware12105/oms is
/u01/app/oracle/Agent12c/core/12.1.0.5.0
[oracle@oem250 bin]$ emctl start
oms
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Starting
Oracle Management Server...
Starting
WebTier...
WebTier
Successfully Started
Oracle
Management Server is not functioning because of the following reason:
Failed to connect to repository
database. OMS will be automatically restarted once it identifies that database
and listener are up.
Check EM Server log file for
details:
/u01/app/oracle/gc_inst1/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
Now try to connect newly
migrated repository on existing OMS Server using eazyconnect.
[oracle@oem250 bin]$ sqlplus
system/oracle@rac127:1521/emrep12c
SQL*Plus:
Release 11.1.0.7.0 - Production on Wed Sep 9 12:58:15 2015
Copyright (c)
1982, 2008, Oracle. All rights reserved.
Connected to:
Oracle
Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the
Partitioning, OLAP, Data Mining and Real Application Testing options
Successfully connected to Migrated
OMR.
Now it's high time to
reconfigure to get everything back to normal.
Also it will prompt for password, you will provide sysman password for
this.
Get the TNS connection string
from newly migrated repository server.
[oracle@oem250 bin]$ tnsping
rac127:1521/emrep12c
TNS Ping
Utility for Linux: Version 11.1.0.7.0 - Production on 09-SEP-2015 13:20:45
Copyright (c)
1997, 2008, Oracle. All rights reserved.
Used parameter
files:
Used HOSTNAME
adapter to resolve the alias
Attempting to
contact
(DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=emrep12c))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.127)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=
192.168.1.127)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.127)(PORT=1521)))
OK (0 msec)
So my new connection string as
below :
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac127.ora.com)(PORT = 1521)) (CONNECT_DATA
= (SERVER = DEDICATED) (SERVICE_NAME = EMREP12C)))
Now set the OMS environment
variable and configure new connection identifier.
[oracle@oem250 config]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c
Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Starting
Oracle Management Server...
Starting
WebTier...
WebTier
Successfully Started
Oracle
Management Server is not functioning because of the following reason:
Failed to
connect to repository database. OMS will be automatically restarted once it
identifies that database and listener are up.
Check EM
Server log file for details:
/u01/app/oracle/gc_inst1/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
Now migrating connection
descriptor from old OMR to Newly Restore Database.
[oracle@oem250 config]$ emctl
config oms -store_repos_details -repos_conndesc
"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac127.ora.com)(
PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))"
-repos_user sysman
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Repository User's Password :
Successfully
updated datasources and stored repository details in Credential Store.
If there are
multiple OMSs in this environment, run this store_repos_details command on all
of them.
And finally,
restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully migrated to new
OMR. Now stop OMS and then after
restart.
[oracle@oem250 config]$ emctl stop
oms
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Stopping
WebTier...
WebTier
Successfully Stopped
Stopping
Oracle Management Server...
Oracle
Management Server Successfully Stopped
Oracle
Management Server is Down
Starting OMS
[oracle@oem250 config]$ emctl
start oms
Oracle Enterprise Manager Cloud
Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Starting
Oracle Management Server...
Starting
WebTier...
WebTier
Successfully Started
Oracle
Management Server Successfully Started
Oracle
Management Server is Up
Checking health check for OMS
[oracle@oem250 config]$ emctl status oms
-details
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Enterprise Manager Root (SYSMAN) Password :
Console Server
Host : oem250.ora.com
HTTP Console
Port : 7788
HTTPS Console
Port : 7802
HTTP Upload
Port : 4889
HTTPS Upload
Port : 4903
EM Instance
Home :
/u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log
Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not
configured with SLB or virtual hostname
Agent Upload
is locked.
OMS Console is
locked.
Active CA ID:
1
Console URL:
https://oem250.ora.com:7802/em
Upload URL:
https://oem250.ora.com:4903/empbs/upload
WLS Domain
Information
Domain
Name : GCDomain
Admin Server
Host : oem250.ora.com
Admin Server
HTTPS Port: 7102
Admin Server
is RUNNING
Oracle
Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host:
oem250.ora.com
WebTier is Up
Oracle Management Server is Up
BI Publisher
is not configured to run on this host.
Check
the newly migrated repository details.
[oracle@oem250 config]$ emctl
config oms -list_repos_details
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Repository
Connect Descriptor :
(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac127.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))
Repository
User : sysman
During migration if anything
goes wrong then we have to use below rollback plan to in order to go back to
existing position.
ROLLBACK PLAN
In case of any issue you have
to stop your newly created repository and login to old repository server and
rollback to existing one.
Start you old Repository
Database , start listener and cluster service is any have!!!
[oracle@oem250 config]$ emctl
config oms -store_repos_details -repos_conndesc
"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac126.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))"
-repos_user sysman
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Repository User's Password :
Successfully
updated datasources and stored repository details in Credential Store.
If there are
multiple OMSs in this environment, run this store_repos_details command on all
of them.
And finally,
restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Stop management server on all
nodes if we have multi node OMS
[oracle@oem250 config]$ emctl stop oms -all
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Stopping
WebTier...
WebTier
Successfully Stopped
Stopping
Oracle Management Server...
Oracle
Management Server Successfully Stopped
AdminServer
Successfully Stopped
Oracle
Management Server is Down
[oracle@oem250
config]$
Start management server on all
nodes
[oracle@oem250 config]$ emctl start oms
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Starting
Oracle Management Server...
Starting
WebTier...
WebTier
Successfully Started
Oracle
Management Server Successfully Started
Oracle
Management Server is Up
Checking health check for OMS
[oracle@oem250 config]$ emctl status oms -details
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Enterprise Manager Root (SYSMAN) Password :
Console Server
Host : oem250.ora.com
HTTP Console
Port : 7788
HTTPS Console
Port : 7802
HTTP Upload
Port : 4889
HTTPS Upload
Port : 4903
EM Instance
Home :
/u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log
Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not
configured with SLB or virtual hostname
Agent Upload
is locked.
OMS Console is
locked.
Active CA ID:
1
Console URL:
https://oem250.ora.com:7802/em
Upload URL:
https://oem250.ora.com:4903/empbs/upload
WLS Domain
Information
Domain
Name : GCDomain
Admin Server
Host : oem250.ora.com
Admin Server
HTTPS Port: 7102
Admin Server
is RUNNING
Oracle
Management Server Information
Managed Server
Instance Name: EMGC_OMS1
Oracle
Management Server Instance Host: oem250.ora.com
WebTier is Up
Oracle
Management Server is Up
Checking repository
configuration details
[oracle@oem250 config]$ emctl config oms -list_repos_details
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Repository
Connect Descriptor :
(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac126.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))
Repository
User : sysman
Also checking the status of
EMKEY.
[oracle@oem250 config]$ emctl status emkey
Oracle
Enterprise Manager Cloud Control 12c Release 5
Copyright (c)
1996, 2015 Oracle Corporation. All
rights reserved.
Enter
Enterprise Manager Root (SYSMAN) Password :
The EMKey is
configured properly.
[oracle@oem250
config]$
Next post will be for migrating
OMS server to new environment.
Hope this will help...