Wednesday, August 12, 2015

Securing agent... Failed - OEM 12cR4


Oracle Enterprise Manager Cloud Control 12c Release 4
Securing agent...   Failed.


Today I was installing 12cR4 Management Agent on 12c four node RAC using deployment script (agent pull).  Everything was running fine but unfortunately at the end I got agent deployment failed on all nodes.  While long troubleshooting I found the problem was not in the deployment.  It’s there in time synchronization between Oracle Management Server and RAC nodes. After fixing it got resolved.  I took help from (Doc ID 1388851.1)

 
Problem :

 EM 12c: Agent Secure Fails with Error: "javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated"

 
[oracle@rac122 bin]$ tail -100f /u01/app/oracle/Agent12c/agent_inst/sysman/log/secure.log
[02-07-2015 01:18:07] USERINFO ::Securing agent...   Started.
2015-07-02 01:18:12,633 [main] INFO  agent.SecureAgentCmd main.235 - emdWalletSrcUrl: https://oem252.ora.com:4900/em/wallets/emd
2015-07-02 01:18:12,649 [main] INFO  agent.SecureAgentCmd secureAgent.281 - Querying HTTPS Upload Port of the OMS
2015-07-02 01:18:12,650 [main] INFO  agent.SecureAgentCmd openPage.803 - Opening: https://oem252.ora.com:4900/empbs/genwallet
2015-07-02 01:18:22,439 [main] INFO  agent.SecureAgentCmd openPage.830 - Response Status Code: 200
2015-07-02 01:18:22,440 [main] INFO  agent.SecureAgentCmd openPage.838 - RESPONSE_STATUS header: OK
2015-07-02 01:18:22,440 [main] INFO  agent.SecureAgentCmd openPage.839 - Response body: 4900
2015-07-02 01:18:22,440 [main] INFO  agent.SecureAgentCmd getOMSSecurePort.785 - OMS Secure Port:  4900
2015-07-02 01:18:22,440 [main] INFO  agent.SecureAgentCmd secureAgent.294 - Requesting an Oracle Wallet and Agent Key from the OMS...
2015-07-02 01:18:22,442 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.365 - Performing INIT
2015-07-02 01:18:22,442 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.366 - Agent Host: rac121.ora.com
2015-07-02 01:18:22,442 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.367 - Agent Port: 3872
2015-07-02 01:18:22,442 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.368 - Agent Version: 12.1.0.4.0
2015-07-02 01:18:22,443 [main] INFO  agent.SecureAgentCmd openPage.803 - Opening: https://oem252.ora.com:4900/empbs/genwallet
2015-07-02 01:18:23,899 [main] INFO  agent.SecureAgentCmd openPage.830 - Response Status Code: 200
2015-07-02 01:18:23,900 [main] INFO  agent.SecureAgentCmd openPage.838 - RESPONSE_STATUS header: OK
2015-07-02 01:18:23,900 [main] INFO  agent.SecureAgentCmd openPage.839 - Response body: -7933286324006274590
2015-07-02 01:18:23,900 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.393 - INIT operation succeded
2015-07-02 01:18:23,901 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.401 - Performing AUTH
2015-07-02 01:18:23,901 [main] INFO  agent.SecureAgentCmd openPage.803 - Opening: https://oem252.ora.com:4900/empbs/genwallet
2015-07-02 01:18:25,525 [main] INFO  agent.SecureAgentCmd openPage.830 - Response Status Code: 200
2015-07-02 01:18:25,526 [main] INFO  agent.SecureAgentCmd openPage.838 - RESPONSE_STATUS header: OK
2015-07-02 01:18:25,526 [main] INFO  agent.SecureAgentCmd openPage.839 - Response body: AUTHOK
2015-07-02 01:18:25,526 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.422 - AUTH operation succeded
2015-07-02 01:18:25,526 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.431 - Downloading from https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/ewall
et.p12 to /u01/app/oracle/Agent12c/agent_inst/sysman/config/server/ewallet.p12
2015-07-02 01:18:25,527 [main] INFO  agent.SecureAgentCmd downloadFile.1014 - Download URL source: https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/ewalle
t.p12
2015-07-02 01:18:25,527 [main] INFO  agent.SecureAgentCmd downloadFile.1015 - Download destination: /u01/app/oracle/Agent12c/agent_inst/sysman/config/server/ewallet.p12
2015-07-02 01:18:25,531 [main] INFO  agent.SecureAgentCmd downloadFile.1043 - download pathAndQuery /empbs/wallets/agent.rac121.ora.com/ewallet.p12
2015-07-02 01:18:38,860 [main] INFO  agent.SecureAgentCmd downloadFile.1046 - Response Status Code: 200
2015-07-02 01:18:38,862 [main] INFO  agent.SecureAgentCmd storeAgentKey.933 - Writing agent key in repoconn.ora
2015-07-02 01:18:38,867 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.440 - Downloading from https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/b64Lo
calCertificate.txt to /u01/app/oracle/Agent12c/agent_inst/sysman/config/b64LocalCertificate.txt
2015-07-02 01:18:38,868 [main] INFO  agent.SecureAgentCmd downloadFile.1014 - Download URL source: https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/b64Loc
alCertificate.txt
2015-07-02 01:18:38,868 [main] INFO  agent.SecureAgentCmd downloadFile.1015 - Download destination: /u01/app/oracle/Agent12c/agent_inst/sysman/config/b64LocalCertificat
e.txt
2015-07-02 01:18:38,869 [main] INFO  agent.SecureAgentCmd downloadFile.1043 - download pathAndQuery /empbs/wallets/agent.rac121.ora.com/b64LocalCertificate.txt
2015-07-02 01:18:39,303 [main] INFO  agent.SecureAgentCmd downloadFile.1046 - Response Status Code: 200
2015-07-02 01:18:39,305 [main] INFO  agent.SecureAgentCmd secureAgentWithOMS.445 - Downloading from https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/b64In
ternetCertificate.txt to /u01/app/oracle/Agent12c/agent_inst/sysman/config/b64InternetCertificate.txt
2015-07-02 01:18:39,305 [main] INFO  agent.SecureAgentCmd downloadFile.1014 - Download URL source: https://oem252.ora.com:4900/empbs/wallets/agent.rac121.ora.com/b64Int
ernetCertificate.txt
2015-07-02 01:18:39,305 [main] INFO  agent.SecureAgentCmd downloadFile.1015 - Download destination: /u01/app/oracle/Agent12c/agent_inst/sysman/config/b64InternetCertifi
cate.txt
2015-07-02 01:18:39,305 [main] INFO  agent.SecureAgentCmd downloadFile.1043 - download pathAndQuery /empbs/wallets/agent.rac121.ora.com/b64InternetCertificate.txt
2015-07-02 01:18:39,951 [main] INFO  agent.SecureAgentCmd downloadFile.1046 - Response Status Code: 200
2015-07-02 01:18:39,963 [main] INFO  agent.SecureAgentCmd secureAgent.297 - Finding CN in OMS cert to form UploadURL
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd getOmsHostName.572 - DN in oms cert is: CN=oem252.ora.com
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd getOmsHostName.574 - CN in oms cert is: oem252.ora.com
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd getOmsHostName.576 - CN in oms cert is: oem252.ora.com
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd secureAgent.302 - Hostname to be used to form UploadURL: oem252.ora.com
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd secureAgent.307 - Computed Upload url :https://oem252.ora.com:4900/empbs/upload
2015-07-02 01:18:39,975 [main] INFO  agent.SecureAgentCmd secureAgent.308 - Checking if HTTPS Upload URL is accessible from the agent...
2015-07-02 01:18:39,976 [main] INFO  agent.SecureAgentCmd secureAgent.309 - Accessing: https://oem252.ora.com:4900/empbs/upload
2015-07-02 01:18:40,080 [main] ERROR agent.SecureAgentCmd main.257 - Failed to secure the Agent:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificateChain(SSLSessionImpl.java:401)
        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.checkUpload(SecureAgentCmd.java:495)
        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.secureAgent(SecureAgentCmd.java:311)
        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.main(SecureAgentCmd.java:244)
[02-07-2015 01:18:40] USERINFO ::Securing agent...   Failed.

 

Cause

 Time synchronization between Oracle Management Server and RAC nodes

 Solution

Fixed time on OMS and RAC nodes and retry on all nodes.

 

[oracle@rac122 bin]$ cd /u01/app/oracle/Agent12c/agent_inst/bin
[oracle@rac122 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent is Not Running
[oracle@rac122 bin]$ ./emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
Securing agent...   Successful.
[oracle@rac122 bin]$
[oracle@rac122 bin]$
[oracle@rac122 bin]$
[oracle@rac122 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting agent .............. started.
[oracle@rac122 bin]$
[oracle@rac122 bin]$
[oracle@rac122 bin]$ ./emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
[oracle@rac122 bin]$
[oracle@rac122 bin]$
[oracle@rac122 bin]$
[oracle@rac122 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/Agent12c/agent_inst
Agent Log Directory    : /u01/app/oracle/Agent12c/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/Agent12c/core/12.1.0.4.0
Agent Process ID       : 19676
Parent Process ID      : 19613
Agent URL              : https://rac122.ora.com:3872/emd/main/
Local Agent URL in NAT : https://rac122.ora.com:3872/emd/main/
Repository URL         : https://oem252.ora.com:4900/empbs/upload
Started at             : 2015-08-11 19:17:50
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.17.1.el6uek.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : 2015-08-11 19:18:33
Last attempted upload                        : 2015-08-11 19:18:40
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 40
Size of XML files pending upload(MB)         : 0.03
Available disk space on upload filesystem    : 20.07%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-08-11 19:17:56
Last successful heartbeat to OMS             : 2015-08-11 19:17:56
Next scheduled heartbeat to OMS              : 2015-08-11 19:18:56
 
---------------------------------------------------------------
Agent is Running and Ready
 

 
Posted by at
Labels: , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)