Monday, December 28, 2015

OEM13c Upgrade




Enterprise Manager Cloud Control 13c

Upgrade - OEM13c 





EM Upgrade - runInstaller (OUI) (More Downtime)

Step 0: Mandatory: Upgrade Repository DB from 11g to 12.1.0.2 version
Step 1: Bring down your 12.1.0.3 /12.1.0.4/12.1.0.5 OMS and invoke EM 13.1em_linux64.bin
Step 2: Upgrade your OMS and repository (Use1- system option in Installer) – Copying the bits and creating new homes – Start the configuration assistants Step 3: You will get Gold Agent Image console in the EM 13.1 console
Step 4: Use Gold Agent Image console to upgrade your 12.1.0.3 /12.1.0.4 /12.1.0.5 agents to 13.1


EM Upgrade - Software only Install (Less Downtime)

Step 0: Mandatory: Upgrade Repository DB from 11g to 12.1.0.2 version
Step 1: Invoke EM 13.1 installer and choose option: Install software only
          – Copying the bits and creating new homes Step 2: Bring down your 12.1.0.3 /12.1.0.4/12.1.0.5 OMS.
          - Start the configuration and Upgrade your OMS and repository (Use1-system option in Installer) Run $/sysman/install/ConfigureGC.sh
Step 3: You will get Gold Agent Image console in the EM 13.1 console
Step 4: Use Gold Agent Image console to upgrade your 12.1.0.3 /12.1.0.4 /12.1.0.5 agents to 13.1

If we are planning for Multi OMS Upgrade then follow below plan.

Additional OMS Upgrade 12.1.0.3/12.1.0.4/12.1.0.5  => 13.1

• High level steps:

Step 1: Once Primary OMS is upgraded, stop all the additional OMS instances by executing $OMS_HOME/bin/emctl stop oms -all
Step 2: On every other host where an additional OMS is running, invoke the EM  installation wizard and on the Installation Type screen select >> Upgrade an Existing Enterprise Manager System >> One System Upgrade >> Select the additional OMS home you want to upgrade

• Note: Parallel Additional OMS upgrade is not supported. Upgrade Additional OMS instances one after the other



Creating directory for New Middleware home and Agent home for 13c OMS up-gradation.

[oracle@oem252 oracle]$ cd /u01/app/oracle
[oracle@oem252 oracle]$ mkdir Middleware13c
[oracle@oem252 oracle]$ mkdir Agent13c

Installing glib* required library.

[root@oem252 ~]#  yum install -y glibc-common glibc-devel glibc-profile glibc-headers  glibc-utils glibc
Loaded plugins: refresh-packagekit, security
OL64                                        | 3.7 kB     00:00 ...
Setting up Install Process
Package glibc-common-2.12-1.132.el6.x86_64 already installed and latest version
Package glibc-devel-2.12-1.132.el6.x86_64 already installed and latest version
No package glibc-profile available.
Package glibc-headers-2.12-1.132.el6.x86_64 already installed and latest version
Package glibc-2.12-1.132.el6.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package glibc-utils.x86_64 0:2.12-1.132.el6 will be installed
--> Processing Dependency: libgd.so.2()(64bit) for package: glibc-utils-2.12-1.132.el6.x86_64
--> Running transaction check
---> Package gd.x86_64 0:2.0.35-11.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================
 Package                            Arch    Version         Repository Size
=================================================================================
Installing:
 glibc-utils                        x86_64  2.12-1.132.el6   OL64     163 k
Installing for dependencies:
 gd                                 x86_64  2.0.35-11.el6    OL64     141 k

Transaction Summary
=================================================================================
Install       2 Package(s)

Total download size: 304 k
Installed size: 581 k
Downloading Packages:
---------------------------------------------------------------------------------
Total                                     6.6 MB/s | 304 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : gd-2.0.35-1.el6.x86_64                                                                                                                  1/2
  Installing : glibc-utils-2.12-1.132.el6.x86_64                                                                                                      2/2
  Verifying  : glibc-utils-2.12-1.132.el6.x86_64                                                                                                      1/2
  Verifying  : gd-2.0.35-11.el6.x86_64                                                                                                                2/2

Installed:
  glibc-utils.x86_64 0:2.12-1.132.el6

Dependency Installed:
  gd.x86_64 0:2.0.35-11.el6

Complete!

Validate the library.

[root@oem252 ~]# rpm -qa | grep glibc
glibc-common-2.12-1.132.el6.x86_64
glibc-devel-2.12-1.132.el6.x86_64
glibc-2.12-1.132.el6.x86_64
glibc-2.12-1.132.el6.i686
glibc-utils-2.12-1.132.el6.x86_64
glibc-headers-2.12-1.132.el6.x86_64
glibc-devel-2.12-1.132.el6.i686


[root@oem252 ~]# echo 11000 65000 > /proc/sys/net/ipv4/ip_local_port_range
[root@oem252 ~]# more /proc/sys/net/ipv4/ip_local_port_range
11000   65000

Add below lines in /etc/sysctl.conf

net.ipv4.ip_local_port_range = 11000 65000

[root@oem252 ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 11000 65000


Validate installer group

[root@oem252 ~]# more /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall

Required mandatory parameter changes on repository database and ensure that repository is having latest patch set like (Character set AL32UTF8, OPTIMIZER_ADAPTIVE_FEATURES=False,Compatible).


[oracle@oem252 tmp]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Dec 23 18:37:22 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> show parameter optimizer_adaptive_features

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features          boolean     TRUE

SQL> alter system set optimizer_adaptive_features=false scope=both sid='*';

System altered.

SQL> select patch_id, patch_uid, version, status, description
 from dba_registry_sqlpatch
 where bundle_series = 'PSU' ;

  PATCH_ID  PATCH_UID VERSION   STATUS   DESCRIPTION
---------- ---------- --------- -------- ----------------------------------------
  20299023   18703022 12.1.0.2  SUCCESS  Database Patch Set Update : 12.1.0.2.3 (20299023)


  PATCH_ID  PATCH_UID DESCRIPTION                                          APPLIED_DATE              
---------- ---------- ---------------------------------------------------- ---------------------------
  20299023   18703022 Database Patch Set Update : 12.1.0.2.3 (20299023)    2015-12-23T10:37:40+05:30 
  19769480   18350083 Database Patch Set Update : 12.1.0.2.2 (19769480)    2015-12-23T10:37:10+05:30 


SQL> show parameter session_

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
session_cached_cursors               integer     500
session_max_open_files               integer     10
sessions                             integer     922


Bounce the database to take impact the change system parameter value



Copy the 13c Software in staging location with appropriate permission


[oracle@oem252 admin]$ cd /SOFT/OEM13c/

[oracle@oem252 OEM13c]$ ls -ltrah
total 6.4G
-rwxr-xr-x. 1 oracle oinstall 317M Dec 21 09:15 em13100_linux64-5.zip
-rwxr-xr-x. 1 oracle oinstall 803M Dec 21 10:12 em13100_linux64.bin
-rwxr-xr-x. 1 oracle oinstall 1.3G Dec 21 11:20 em13100_linux64-2.zip
-rwxr-xr-x. 1 oracle oinstall 2.0G Dec 21 12:09 em13100_linux64-4.zip
-rwxr-xr-x. 1 oracle oinstall 2.0G Dec 21 12:10 em13100_linux64-3.zip




Here I am going to Upgrade the OEM13c in 2 phase.

·         Phase 1 will cover binary installation only where I choose Software Only

·         Phase 2 will cover OMS Upgrade


Phase 1


Before upgrading management server its mandatory to move emkey to repository else we are facing error in middle of installation.  It’s recommended to move emkey to repository where we have to compromise with security where it’s going to unsecure.  We can secure after upgrade.  In this demonstration I will show and fix the error in middle of installation. For more details Click Here.


Set the environment for OMS
  

[oracle@oem252 ~]$ emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey  is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos".


In this demonstration I have already copied emkey into repository.

Now its time to install the OMS13c Binary (only software),  configuration part will be taken care in phase 2 in below blog.

Before running the OMS13c installer you don't  need to unzip any file here, just ensure that files is having required permission.


Click Next 


Choose Skip and Click Next 


Ensure that prerequisite executed successfully. If any error, fix it and then Click Next 


To reduce downtime, I choose Software only here,  Configuration part will take care in Phase 2.  Click Next 


Use the new Middleware and Agent home for OEM13c installation.  Click Next 


Review and Install Binary



 Run allroot.sh and click OK 




Copy the below configuration script, this script will use in Phase 2, during OMS upgradation.



  
Successfully installed the software binaries.

To configure the installation, run the following command from the OMS home:

/u01/app/oracle/Middleware13c/sysman/install/ConfigureGC.sh


Phase 2

In this phase I am going to Upgrade existing 12cR5 OMS.  To continue phase 2 run the below script in your VNC Terminal/X-Manager. 

/u01/app/oracle/Middleware13c/sysman/install/ConfigureGC.sh



Choose Upgrade existing EM System and pick One-System Upgrade and select existing Middleware home.  Then Click Next


Before doing upgrade, ensure that you have valid repository backup. Also provide the Existing 12c Repository (which is having latest Patch applied) connection credentials.   Click Next 
 

Click Next 



Here in middle of installation, i found some deployed plug-ins has obsolete.  In order to continue the upgrade wizard, I have to un-deploy obsolete plug-ins.  

Here we are having Exadata Elastic Cloud Infrastructure obsolete plugins 





I followed below procedure to un-deploy plugins as below in existing 12cR5 OMS.

Click Setup > Extinsibility > Self Update




In Self Update > select Plugins > Choose your obsolete plugins >  and Click on Action and select Open



Click Remove.  This will open a new window



Select your Exadata Plugins and click  Undeploy from Management Server and Agent 



Provide sys password for Repository database and validate the status. 








Finally your obsolete plugins has removed. Now Click OK and continue the Upgrade wizard




Also ensure that the compatible parameter must be equivalent to RDBMS version and click OK





Review the details and follow the wizard



Select the additional plugins as per your requirement and Click Nex



Installer will ask for OMS server complete shutdown, First shutdown the OMS and then Click Nex


Set the Weblogic password and Click Nex


Un-check the shared location for Oracle BI Publisher as I don't have that, then Click Nex


Review the Recommended port and Click Nex




Review and Click Upgrade






Finally I have successfully upgraded the 12cR5 OMS to 13cR1.  Save the below details and Click close the installation.   

Now its high time to verify the installation and Upgrade Management Agent.

Login to 13c OEM Console with your existing credentials.





Feeling great to see new 13c Enterprise Manager Cloud Control.






UPGRADING AGENT

In 13c EM Console find the target which you want to upgrade.  

Upgrading agent is very similar to 12c Upgrading Agent.  

Click System Icon > Manage Cloud Console > Upgrade Agent

You can add your selecting target where you want to upgrade the agent and submit the job.







For any troubleshooting issue, go to concern logfile directory.

Log File Location

Installation Logs - oraInventory/logs/*.* - /cfgtoollogs/oui/*.* 

Eg. ORACLE_HOME=/u01/app/oracle/Middleware13c

  • Configuration Logs - /cfgtoollogs/cfgfw/*.*
  • Specific Configuration tool logs 
    • OMS configuration tool logs: /cfgtoollogs/omsca/*.* 
    • PluginCA configuration tool logs: /cfgtoollogs/pluginca 
    • Repository configuration tool logs: / sysman/log/schemamanager
  
For other information, Refer to 13c Oracle Documentation.

Hope this will help....  :)