OEM 12C - Authentication failed. If problem persists, contact your system administrator.

Login to the 12c EM Console as any EM user is failing with the error:

Authentication failed. If problem persists, contact your system administrator.

Login to the 12c EM Console as any EM user is failing with the error:

Authentication failed. If problem persists, contact your system administrator.

-  The sysman account in the repository database is open and not expired.

-  Below command with the same sysman password as entered in the Console UI, works successfully:

cd $OMS_HOME/bin

emctl status oms –details

-  Error logged in the <gc_inst>/em/EMGC_OMS1/sysman/log/emoms.trc file:

[[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR audit.AuditLogHandler auditLog.381 - Could not log the audit record java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 873

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 402

ORA-06512: at line 1

OR

[[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN auth.EMRepLoginFilter doFilter.450 - InvalidEMUserException caught in EMRepLoginFilter: Failed to login using repository authentication for user: SYSMAN

oracle.sysman.emSDK.sec.auth.InvalidEMUserException: Failed to login using repository authentication for user: SYSMAN

at oracle.sysman.emSDK.sec.auth.EMLoginService._performLogin(EMLoginService.java:1289)

...........

 [[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR audit.AuditLogHandler auditLog.378 - Could not log the audit record java.sql.SQLException: ORA-04031: unable to allocate 4160 bytes of shared memory ("shared pool","unknown object","sga heap(4,0)","modification ")

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 492

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 406

ORA-06512: at line 1

OR

ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR audit.AuditLogHandler auditLog.381 - Could not log the audit record java.sql.SQLException: ORA-01658: unable to create INITIAL extent for segment in tablespace MGMT_ECM_DEPOT_TS

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 873

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 402

ORA-06512: at line 1

OR

 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG auth.EMLoginService _performLogin.825 - Error

java.sql.SQLException: ORA-04063: package body "SYSMAN.MGMT_AUDIT_ADMIN" has errors

ORA-06508: PL/SQL: could not find program unit being called: "SYSMAN.MGMT_AUDIT_ADMIN"

ORA-06512: at "SYSMAN.MGMT_AUDIT", line 143

ORA-06512: at "SYSMAN.EM_USER_MODEL_UI", line 1521

ORA-04063: package body "SYSMAN.MGMT_AUDIT_ADMIN" has errors

ORA-06508: PL/SQL: could not find program unit being called: "SYSMAN.MGMT_AUDIT_ADMIN"

ORA-06512: at line 1

-  The error occurs for all EM Administrators, there is no external authentication like AD / SSO configured.

CAUSE

The console login creates an audit entry in the EM related audit table. If this audit entry is not successful for some reason, then the console login will fail.

Issue was investigated in the below bugs:

Bug 14285322: GC12:VALUE FOR JOB_QUEUE_PROCESSES SHOULD NOT BE ZERO AFTER UPGRADE

BUG 16205038 : [EMCC] DBMS JOBS ARE NOT EXECUTED REGULARLY CAUSING AUTHENTICATION FAILURE

It was identified that a recent upgrade / maintenance activity (for example: patching or plugin upgrade) was performed on the setup, which had set the job_queue_processes initialization parameter to 0 in the EM repository database. This had prevent the execution of the DBMS_JOBS responsible for maintenance of the repository. Console login needs to create an entry into the Audit tables but the necessary partitions had not been created for this table.

SOLUTION

If an upgrade / maintenance activity (for example: patching or plugin upgrade) was performed recently, the job_queue_processes initialization parameter could be still set to 0 and not reset to the original value. This would prevent the DBMS Scheduler jobs related to the repository from getting executed correctly due to which the new partitions for the Audit tables entries are not created.

1.    Login to the database as a SYSDBA user and set the job_queue_processes initialization parameter to a value of 20:

SQL> ALTER SYSTEM SET job_queue_processes=20 SCOPE=BOTH SID='*';

2.    If the job_queue_processes value is already set, verify that this set to 20 or more.

Verify that the DBMS scheduler jobs of the repository are running fine using the steps in:
Note 1470166.1 : 12c DBMS Jobs in Invalid Schedule Status in the Console Page

3.    Execute the following queries in repository DB as SYSMAN user:

SQL> update mgmt_audit_master set prepopulate_days=5 where prepopulate_days is null;

SQL> exec mgmt_audit_admin.add_audit_partition;

4.    Restart the OMS:

[oracle@oem250 encrypt_pass]$ . oraenv

ORACLE_SID = [oms] ?

The Oracle base for ORACLE_HOME=/u01/app/oracle/Middleware/oms is /u01/app/oracle/Agent12c/core/12.1.0.4.0

[oracle@oem250 encrypt_pass]$ cd /u01/app/oracle/Middleware/oms/bin/

[oracle@oem250 bin]$ ./emctl stop oms -all

Oracle Enterprise Manager Cloud Control 12c Release 4

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Stopping WebTier...

WebTier Successfully Stopped

Stopping Oracle Management Server...

Oracle Management Server Successfully Stopped

AdminServer Successfully Stopped

Oracle Management Server is Down

[oracle@oem250 bin]$ ./emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 4

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Starting Oracle Management Server...

Starting WebTier...

WebTier Successfully Started

Oracle Management Server Successfully Started

Oracle Management Server is Up

[oracle@oem250 bin]$ emcli login -username=sysman -password=amit1234

Login successful

[oracle@oem250 bin]$ emcli sync

Synchronized successfully

[oracle@oem250 bin]$ emcli list_plugins_on_server

OMS name is oem250.ora.com:4889_Management_Service

Plug-in Name                Plugin-id         Version [revision]

Oracle Cloud Framework         oracle.sysman.cfw  12.1.0.1.0

Oracle Database                oracle.sysman.db   12.1.0.6.0

Oracle Fusion Middleware       oracle.sysman.emas 12.1.0.6.0

Oracle MOS (My Oracle Support) oracle.sysman.mos  12.1.0.6.0

Oracle Exadata                 oracle.sysman.xa   12.1.0.6.0

Now Login on EM Console and verify