Sunday, September 27, 2015

OEM12cR5 : Moving Oracle Management Repository (OMR) to New Server


Migration: Oracle Enterprise Manager Cloud Control 12c R5
Moving Oracle Management Repository (OMR) to New Server (11.2.0.4.6 - PSU)


Today's demonstration is for data center migration from one location to other location, where we have to plan migrate our existing setup to new environment.  In our environment we have Oracle Enterprise Manger Cloud Control - Release 5  (OEM12c - R5) having management repository 11.2.0.4.6 (PSU) applied. 

In first phase we are planning to all management repository to new environment.  In order to achieve this task we are following one approach in list of below.

1.   RMAN Backup restoration and recovery
2.   Export and Import
3.   Data Guard with broker
4.   Cold Backup


In this demonstration I am using approach 1 which is RMAN Backup restoration and recovery.

In our next post I will do this with the help of data guard which is coming soon.

In Second phase we will migrate our existing management server to new environment in next post.

We are considering that we have taken Level 0 RMAN backup and redo archive log till level 0 backup.

Source Repository Server     : rac126.ora.com
Target Repository Server     : rac127.ora.com

Oracle Management Server  : oem250.ora.com

Process will be looking for the "emkey". This key needs to be placed inside the database so it can be referenced during the Migration. If not available, then we will face issue during process.

After that we will take RMAN level 0 backup from existing repository server.


[oracle@oem250 config]$ emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.


[oracle@oem250 config]$ emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

Now Stop OMS and management agent on OMS Server.  Here do not use "all" option while stopping OMS.

[oracle@oem250 bin]$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down


[oracle@oem250 bin]$ emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
[oracle@oem250 bin]$


After shutdown the OMS and management Agent we will start database backup restoration on Target Repository Server

Create necessary directory on target database "rac127"


mkdir -p /u01/app/oracle/admin/EMREP12C/adump
mkdir -p /u01/app/oracle/fast_recovery_area/EMREP12C/
mkdir -p /u01/app/oracle/oradata/EMREP12C/

Go the backup location where Level 0 backup and archivelog. Set the environment for new database and put database in nomount and restore spfile , and then after restore controlfile, catalog backup and start restoration and recovery and open database in resetlogs. 

RMAN>  startup force nomount

Oracle instance started

Total System Global Area    2137886720 bytes

Fixed Size                     2254952 bytes
Variable Size               1493174168 bytes
Database Buffers             637534208 bytes
Redo Buffers                   4923392 bytes


RMAN> restore controlfile from '/Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00' ;

Starting restore at 26-SEP-15
using channel ORA_DISK_1

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/u01/app/oracle/oradata/EMREP12C/control01.ctl
output file name=/u01/app/oracle/fast_recovery_area/EMREP12C/control02.ctl
Finished restore at 26-SEP-15

RMAN> alter database mount ;

database mounted
released channel: ORA_DISK_1

RMAN> catalog start with '/Shared_NAS/BACKUP/EMREP12c/' ;

Starting implicit crosscheck backup at 26-SEP-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=21 device type=DISK
Crosschecked 3 objects
Finished implicit crosscheck backup at 26-SEP-15

Starting implicit crosscheck copy at 26-SEP-15
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 26-SEP-15

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern /Shared_NAS/BACKUP/EMREP12c/

List of Files Unknown to the Database
=====================================
File Name: /Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_195_byyqz2vh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_197_byyr0d1s_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_196_byyr0cqv_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_194_byx5q8wd_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_193_byvgxqsh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_192_bysbor59_.arc

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /Shared_NAS/BACKUP/EMREP12c/control_c-3728288860-20150909-00
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_195_byyqz2vh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_197_byyr0d1s_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_196_byyr0cqv_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_194_byx5q8wd_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_193_byvgxqsh_.arc
File Name: /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_192_bysbor59_.arc



RMAN> restore database  ;

Starting restore at 26-SEP-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/EMREP12C/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /u01/app/oracle/oradata/EMREP12C/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/EMREP12C/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/EMREP12C/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/EMREP12C/example01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/EMREP12C/mgmt_depot.dbf
channel ORA_DISK_1: restoring datafile 00007 to /u01/app/oracle/oradata/EMREP12C/mgmt.dbf
channel ORA_DISK_1: restoring datafile 00008 to /u01/app/oracle/oradata/EMREP12C/mgmt_ad4j.dbf
channel ORA_DISK_1: reading from backup piece /Shared_NAS/BACKUP/EMREP12c/EMREP12c_0mqgmjnq_1_1
channel ORA_DISK_1: piece handle=/Shared_NAS/BACKUP/EMREP12c/EMREP12c_0mqgmjnq_1_1 tag=TAG20150909T034602
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:03:35
Finished restore at 26-SEP-15

RMAN> recover database  ;

Starting recover at 26-SEP-15
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 198 is already on disk as file /Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc
archived log file name=/Shared_NAS/BACKUP/EMREP12c/ARCHIVELOG/o1_mf_1_198_byys39v2_.arc thread=1 sequence=198
unable to find archived log
archived log thread=1 sequence=199
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 09/26/2015 23:52:11
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 199 and starting SCN of 8855553



RMAN> alter database open resetlogs ;

database opened


 [oracle@rac127 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Sep 26 23:58:54 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select name, open_mode from v$database ;

NAME      OPEN_MODE
--------- --------------------
EMREP12C  READ WRITE

SQL> select flashback_on from v$database ;

FLASHBACK_ON
------------------
NO


Reconfigure Existing OMS For New Repository Database

Now Start the OMS and agent on the existing OMS server.  Startup will fail as expected, as I have not yet reconfigured the repository on newly migrated repository server.

[oracle@oem250 bin]$ . oraenv
ORACLE_SID = [agent] ? oms
The Oracle base for ORACLE_HOME=/u01/app/oracle/Middleware12105/oms is /u01/app/oracle/Agent12c/core/12.1.0.5.0
[oracle@oem250 bin]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /u01/app/oracle/gc_inst1/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out


Now try to connect newly migrated repository on existing OMS Server using eazyconnect.



[oracle@oem250 bin]$ sqlplus system/oracle@rac127:1521/emrep12c

SQL*Plus: Release 11.1.0.7.0 - Production on Wed Sep 9 12:58:15 2015

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Successfully connected to Migrated OMR.


Now it's high time to reconfigure to get everything back to normal.  Also it will prompt for password, you will provide sysman password for this.

Get the TNS connection string from newly migrated repository server.


[oracle@oem250 bin]$ tnsping rac127:1521/emrep12c

TNS Ping Utility for Linux: Version 11.1.0.7.0 - Production on 09-SEP-2015 13:20:45

Copyright (c) 1997, 2008, Oracle.  All rights reserved.

Used parameter files:

Used HOSTNAME adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=emrep12c))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.127)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=
192.168.1.127)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.127)(PORT=1521)))
OK (0 msec)


So my new connection string as below :

(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac127.ora.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = EMREP12C)))

Now set the OMS environment variable and configure new connection identifier.


 [oracle@oem250 config]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /u01/app/oracle/gc_inst1/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

Now migrating connection descriptor from old OMR to Newly Restore Database.


[oracle@oem250 config]$ emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac127.ora.com)(                    PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))" -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Repository User's Password :
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.

Successfully migrated to new OMR.  Now stop OMS and then after restart.


[oracle@oem250 config]$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

Starting OMS

[oracle@oem250 config]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Checking health check for OMS

 [oracle@oem250 config]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem250.ora.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oem250.ora.com:7802/em
Upload URL: https://oem250.ora.com:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oem250.ora.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oem250.ora.com
WebTier is Up
Oracle Management Server is Up

BI Publisher is not configured to run on this host.

Check the newly migrated repository details.

[oracle@oem250 config]$ emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Repository Connect Descriptor : (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac127.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))
Repository User : sysman



During migration if anything goes wrong then we have to use below rollback plan to in order to go back to existing position.


ROLLBACK PLAN

In case of any issue you have to stop your newly created repository and login to old repository server and rollback to existing one.

Start you old Repository Database , start listener and cluster service is any have!!!


[oracle@oem250 config]$ emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac126.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))" -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Repository User's Password :
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.

Stop management server on all nodes if we have multi node OMS

[oracle@oem250 config]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
[oracle@oem250 config]$

Start management server on all nodes

[oracle@oem250 config]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Checking health check for OMS

[oracle@oem250 config]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem250.ora.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/gc_inst1/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst1/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oem250.ora.com:7802/em
Upload URL: https://oem250.ora.com:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oem250.ora.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oem250.ora.com
WebTier is Up
Oracle Management Server is Up

Checking repository configuration details

[oracle@oem250 config]$ emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Repository Connect Descriptor : (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac126.ora.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP12C)))
Repository User : sysman

Also checking the status of EMKEY.

[oracle@oem250 config]$ emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
[oracle@oem250 config]$


Next post will be for migrating OMS server to new environment.




Hope this will help...