DBA Tips Archive for Oracle

  


[an error occurred while processing this directive]

No Title

[an error occurred while processing this directive]

by Michael New, MichaelNew@earthlink.net, Gradation LLC

Contents

Introduction

There are many scenarios where OEM Database Control is unable to start. Attempting to document every possible issue with OEM not starting and their solution would be a monumental task. However, I recently ran into a scenario that I believe was worth documenting and sharing.

OEM Database Control was configured on several Oracle database severs and had been running successfully for several years. One day, Database Control was not able to start on any of these database servers. Absolutely no changes were made to the environment (Oracle Database, OS, Network, Security, etc.). I still followed through a series of troubleshooting steps like verifying the correct hostname and reverse name lookup (nslookup). But knowing that nothing changed in the environment, I wasn't convinced the problem was being caused by a setup or configuration issue. I then tried several times to drop the OEM Database Control application and re-create it. Database Control successfully dropped and re-created, however, I was still not able to start the OEM application.


============================================================= C:\>set oracle_sid=TESTDB ============================================================= C:\>emca -deconfig dbcontrol db -repos drop STARTED EMCA at Mar 2, 2011 4:45:41 PM EM Configuration Assistant, Version 10.2.0.1.0 Production Copyright (c) 2003, 2005, Oracle. All rights reserved. Enter the following information: Database SID: TESTDB Listener port number: 1521 Password for SYS user: xxxxxxxx Password for SYSMAN user: xxxxxxxx Password for SYSMAN user: xxxxxxxx Do you wish to continue? [yes(Y)/no(N)]: y Mar 2, 2011 4:46:04 PM oracle.sysman.emcp.EMConfig perform INFO: This operation is being logged at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\TESTDB\emca_2011-03-02_04-45-41-PM.log. Mar 2, 2011 4:46:07 PM oracle.sysman.emcp.util.DBControlUtil stopOMS INFO: Stopping Database Control (this may take a while) ... Mar 2, 2011 4:46:11 PM oracle.sysman.emcp.EMReposConfig dropRepository INFO: Dropping the EM repository (this may take a while) ... Mar 2, 2011 4:46:42 PM oracle.sysman.emcp.EMReposConfig invoke INFO: Repository successfully dropped Enterprise Manager configuration completed successfully FINISHED EMCA at Mar 2, 2011 4:46:42 PM ============================================================= C:\>emca -config dbcontrol db -repos create STARTED EMCA at Mar 2, 2011 4:46:59 PM EM Configuration Assistant, Version 10.2.0.1.0 Production Copyright (c) 2003, 2005, Oracle. All rights reserved. Enter the following information: Database SID: TESTDB Listener port number: 1521 Password for SYS user: xxxxxxxx Password for DBSNMP user: xxxxxxxx Password for SYSMAN user: xxxxxxxx Email address for notifications (optional): <ENTER> Outgoing Mail (SMTP) server for notifications (optional): <ENTER> ----------------------------------------------------------------- You have specified the following settings Database ORACLE_HOME ................ C:\oracle\product\10.2.0\db_1 Database hostname ................ testnode1.idevelopment.info Listener port number ................ 1521 Database SID ................ TESTDB Email address for notifications ............... Outgoing Mail (SMTP) server for notifications ............... ----------------------------------------------------------------- Do you wish to continue? [yes(Y)/no(N)]: y Mar 2, 2011 4:47:18 PM oracle.sysman.emcp.EMConfig perform INFO: This operation is being logged at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\TESTDB\emca_2011-03-02_04-46-59-PM.log. Mar 2, 2011 4:47:26 PM oracle.sysman.emcp.EMReposConfig createRepository INFO: Creating the EM repository (this may take a while) ... Mar 2, 2011 4:48:26 PM oracle.sysman.emcp.EMReposConfig invoke INFO: Repository successfully created Mar 2, 2011 4:48:33 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole INFO: Securing Database Control (this may take a while) ... Mar 2, 2011 4:48:45 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole INFO: Database Control secured successfully. Mar 2, 2011 4:48:45 PM oracle.sysman.emcp.util.DBControlUtil startOMS INFO: Starting Database Control (this may take a while) ... Mar 2, 2011 4:57:25 PM oracle.sysman.emcp.EMConfig perform SEVERE: Error starting Database Control Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\TESTDB\emca_2011-03-02_04-46-59-PM.log for more details. Could not complete the configuration. Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\TESTDB\emca_2011-03-02_04-46-59-PM.log for more details. ============================================================= C:\>type C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\TESTDB\emca_2011-03-02_04-46-59-PM.log ... <SNIP> ... SEVERE: Error starting Database Control Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\PROD\emca_2011-02-15_06-56-03-PM.log for more details. Feb 15, 2011 7:07:44 PM oracle.sysman.emcp.EMConfig perform CONFIG: Stack Trace: oracle.sysman.emcp.exception.EMConfigException: Error starting Database Control at oracle.sysman.emcp.EMDBPostConfig.performConfiguration(EMDBPostConfig.java:646) at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:224) at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:193) at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:184) at oracle.sysman.emcp.EMConfigAssistant.invokeEMCA(EMConfigAssistant.java:486) at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1142) at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:470) at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:419)

Solution

After an exhaustive search and several hours of troubleshooting, I finally found the problem with this particular case. It turns out that OEM Database Control was hard-coded with a 10 year certificate that expired in this environment. I believe this is the certificate used by the agent to communicate with the application server.

The solution was to apply interim patch for base bug: 8350262 to each Oracle database server. No database outage is required to apply this patch — the database and listener can remain online while applying this patch.


Recommended Database patch for new/renew/secure DB Console with 10gR4 or 10gR5 Database: application of Patch 8350262 will prevent problems. **See Box in Note 278100.1: ~~~~~~~~~~~~ "ATTENTION! After 31-Dec-2010, creating/recreating/securing 10.2.0.4/10.2.0.5 EM DB Control will fail..." ~~~~~~~~~~~~ ========================================================================= Interim Patch for Base Bug: 8350262 ========================================================================= Date: Sep 10, 2010 ------------------------------------------------------------------------- Platform Patch for : Generic Product Patched : 10.2.0.4.0 Product Version : Enterprise Manager Database Control Bugs Fixed by this patch: ------------------------- 8350262:CREATE DBCONSOLE CERT WITH 10YEAR VALIDITY =========================================================================

The patch will update the following files:


Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\fsc\FSWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\rep\RepWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\util\RootCert.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\util\SecConstants.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\fsc\FSWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\rep\RepWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\RootCert.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\SecConstants.class"

After applying the patch, drop and re-create OEM Database Control.

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Saturday, 25-Feb-2012 16:43:59 EST
Page Count: 4592