Friday, October 2, 2015

OEM-12c: Agent Upload issue EMD upload error:full upload has failed:uploadXMLFiles skipped:: OMS version not checked yet


Enterprise Manager : Agent Upload issue
EMD upload error:full upload has failed:
uploadXMLFiles skipped:: OMS version not checked yet


In today demonstration, I am showing how to fix Agent Upload Issue. 

In this scenario I had migrate OMS12cR5 from our existing server to newly setup environment as its business need.

I have successfully migrated OMS Repository (11.2.0.4 (PSU), OMS 12cR5.  After migration some of my server having agent upload issue.  I will try to fix using standard method (stop/start/delete upload log/Start but couldn't got success.

Because of Agent is still try to access my old Management server (OEM250).  So applying below approach has solved my problem.


Problem

12c agent fails to upload after reinstallation with following errors:

EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version n                                                                             ot checked yet. If this issue persists check trace files for ping to OMS related                                                                              errors. (OMS_DOWN)

Heartbeat Status: OMS is unreachable



Applied standard method but doesn't get success.  Here is the below

dns100:> emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/Agent12cR5/agent_inst
Agent Log Directory    : /u01/app/oracle/Agent12cR5/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/Agent12cR5/core/12.1.0.5.0
Agent Process ID       : 3202
Parent Process ID      : 3125
Agent URL              : https://dns100.ora.com:3872/emd/main/
Local Agent URL in NAT : https://dns100.ora.com:3872/emd/main/
Repository URL         : https://oem251.ora.com:4902/empbs/upload
Started at             : 2015-10-03 05:39:37
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.209.1.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 87
Size of XML files pending upload(MB)         : 0.28
Available disk space on upload filesystem    : 87.64%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2015-10-03 05:40:49
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2015-10-03 05:41:20

---------------------------------------------------------------
Agent is Running and Ready

Stopping Agent

dns100:> emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.

Deleting pending upload logs


dns100:> cd /u01/app/oracle/Agent12cR5/agent_inst/sysman/emd/upload
dns100:> ls
pending  upload
dns100:> rm -rf *
dns100:> cd ../state/
dns100:> ls
configLogs      inbox                  recvlet_state         storage
configstate     mcePersist             severityTraceRecords
fetchlet_state  oracleHomeFirstConfig  statemgmt
dns100:> rm -rf *

Starting Agent


dns100:> emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting agent ........... started.

Checking the status of Agent,  still showing "OMS IS UNREACHABLE"


dns100:> emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/Agent12cR5/agent_inst
Agent Log Directory    : /u01/app/oracle/Agent12cR5/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/Agent12cR5/core/12.1.0.5.0
Agent Process ID       : 4252
Parent Process ID      : 4209
Agent URL              : https://dns100.ora.com:3872/emd/main/
Local Agent URL in NAT : https://dns100.ora.com:3872/emd/main/
Repository URL         : https://oem251.ora.com:4902/empbs/upload
Started at             : 2015-10-03 05:44:59
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.209.1.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 41
Size of XML files pending upload(MB)         : 0.23
Available disk space on upload filesystem    : 87.64%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2015-10-03 05:45:16
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2015-10-03 05:45:46

---------------------------------------------------------------
Agent is Running and Ready

And when uploading the getting below error


dns100:> emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version n                                                                             ot checked yet. If this issue persists check trace files for ping to OMS related                                                                              errors. (OMS_DOWN)


Cause

The agent has been uploading to same OMS earlier and has been reinstalled on the same host and port. OMS blocks the agent after reinstallation and will not allow for any further uploads until the information available at repository and agent  are in synchronization.



Solution

After reinstallation need to add agent and host target and then need to perform sync operation for the upload to be successful.

Perform the following steps to resolve the issue:

Before performing below ensure that Agent must up.  Issue following command to add agent and host targets:


dns100:> emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.

This would list the agent target in Setup > Add Target > Auto Discovery Results 

Now Securing the agent.  By doing this would show the host and agent in pending status under Targets > All targets

dns100:> emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
Agent successfully restarted...   Done.
Securing agent...   Successful.


Following steps to show the status of host and agent as up

  • Stopping Agent
  • Clearstate Agent
  • Starting Agent



Stopping Agent

dns100:> emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.

Clearing the Agent State


dns100:> emctl clearstate agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
EMD clearstate completed successfully

Starting Agent

dns100:> emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting agent ........... started.

Checking status of Agent


dns100:> emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 12.1.0.5.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/Agent12cR5/agent_inst
Agent Log Directory    : /u01/app/oracle/Agent12cR5/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/Agent12cR5/core/12.1.0.5.0
Agent Process ID       : 6608
Parent Process ID      : 6565
Agent URL              : https://dns100.ora.com:3872/emd/main/
Local Agent URL in NAT : https://dns100.ora.com:3872/emd/main/
Repository URL         : https://oem251.ora.com:4903/empbs/upload
Started at             : 2015-10-03 05:52:05
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.209.1.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2015-10-03 05:52:14
Last attempted upload                        : 2015-10-03 05:52:14
Total Megabytes of XML files uploaded so far : 0.01
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 87.64%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-10-03 05:52:10
Last successful heartbeat to OMS             : 2015-10-03 05:52:10
Next scheduled heartbeat to OMS              : 2015-10-03 05:53:10

---------------------------------------------------------------
Agent is Running and Ready

Uploading Agent


dns100:> emctl upload
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Checking status of Agent


dns100:> emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 12.1.0.5.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/Agent12cR5/agent_inst
Agent Log Directory    : /u01/app/oracle/Agent12cR5/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/Agent12cR5/core/12.1.0.5.0
Agent Process ID       : 6608
Parent Process ID      : 6565
Agent URL              : https://dns100.ora.com:3872/emd/main/
Local Agent URL in NAT : https://dns100.ora.com:3872/emd/main/
Repository URL         : https://oem251.ora.com:4903/empbs/upload
Started at             : 2015-10-03 05:52:05
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.209.1.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2015-10-03 05:52:40
Last attempted upload                        : 2015-10-03 05:52:40
Total Megabytes of XML files uploaded so far : 0.02
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 87.64%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-10-03 05:52:10
Last successful heartbeat to OMS             : 2015-10-03 05:52:10
Next scheduled heartbeat to OMS              : 2015-10-03 05:53:10

---------------------------------------------------------------
Agent is Running and Ready



In My case everything running smooth now.  But in some case I observer issue with synchronization, if you are facing then follow below support doc (1376968.1).

EM 12c: How to Perform Enterprise Manager 12c Cloud Control Agent Resynchronization (Doc ID 1376968.1)





Hope this help ....

1 comment: