DBA Tips Archive for Oracle

  


[an error occurred while processing this directive]

No Title

[an error occurred while processing this directive]

by Michael New, MichaelNew@earthlink.net, Gradation LLC

Contents

Introduction

This article highlights some of the common bugs you may encounter while using Oracle Enterprise Manager 10g. While not meant to be a substitute for the official Oracle documentation or the notes provided on My Oracle Support, this guide does attempt to document a list of symptoms and possible resolutions to aid users while troubleshooting their Oracle Enterprise Manger 10g environment.

Clearing the State of the Management Agent

Symptoms

The target database instance and listener are up. The management agent for the target database is up and running. After several minutes, Grid Control Console shows the following error for the target database:

The instance has been started in no-mount state.

It also shows:

Database Instance: Status Open
Listener: Status Up

"The database target is currently unavailable. The state of the components are listed below."
Agent Connection to Instance: Status Unavailable Details

Checked the status of the dbsnmp database user and it was NOT locked.

You have already "Configured the Target Database Grid Control Console" and verified and tested the dbsnmp user's password was correct.

Solution

Log in to the node that hosts the target database. Switch your SHELL environment to the management agent and run "emctl clearstate agent":


[oracle@vmlinux1 ~]$ emctl clearstate agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. EMD clearstate completed successfully [oracle@vmlinux1 ~]$ emctl upload agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully

EMAgent is Thrashing. Exiting watchdog

Symptoms

Oracle Grid Control has been working for several months with no issue (GC 10.2.0.5). No changes and/or patches have been applied to the environment. On a recent reboot of the OEM Grid Control management server, the Oracle agent is throwing the following message when trying to start:


[oracle@oemprod ~]$ . oraenv ORACLE_SID = [oms10g] ? agent10g The Oracle base for ORACLE_HOME=/u01/app/oracle/product/agent10g is /u01/app/oracle [oracle@oemprod ~]$ emctl start agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. Starting agent ................................. failed. EMAgent is Thrashing. Exiting watchdog Consult the log files in: /u01/app/oracle/product/agent10g/sysman/log

Solution

Reviewing the emagent.trc and emdctl.trc files did not provide any useful trace infomation to diagnose the problem in this case. I did find several notes on Metalink regarding this error, however, none of the solutions applied to my configuration.

I then decided to check the dbsnmp login credentials to the OEM Grid Control Repository. The database password in this case was valid, however the password was about to expire:


[oracle@oemprod log]$ sqlplus dbsnmp/<dbsnmp_password> SQL*Plus: Release 11.1.0.7.0 - Production on Mon May 10 11:46:58 2010 Copyright (c) 1982, 2008, Oracle. All rights reserved. ERROR: ORA-28002: the password will expire within 7 days Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production With the Partitioning, Oracle Label Security, OLAP, Data Mining, Oracle Database Vault and Real Application Testing options SQL>

The solution in this case was to change the password for the dbsnmp user account. Note that I used the same password value when changing the dbsnmp password to avoid any further complications:


SQL> alter user dbsnmp identified by <same_dbsnmp_password>; User altered.

After updating the dbsnmp password, the agent was able to start successfully:


[oracle@oemprod ~]$ . oraenv ORACLE_SID = [emrep] ? agent10g The Oracle base for ORACLE_HOME=/u01/app/oracle/product/agent10g is /u01/app/oracle [oracle@oemprod ~]$ emctl start agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. Starting agent .......... started.

EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

Symptoms

After upgrading or recoverying the OMS and Agent to version 10.2.0.5, it is possible to recieve the error in <AGENT_HOME>/sysman/log/emagent.trc:


2010-05-10 12:28:56,007 Thread-149101456 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"

'emctl upload agent' fails with:


[oracle@oemprod ~]$ emctl upload agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

Cause

The Agent Re-synchronization is a new feature in 10.2.0.5 Grid Control which verifies whether an Agent that been uploading earlier to the OMS is re-installed or restored from a backup. If yes, the OMS blocks further updates from this Agent until the information about the Agent and all its targets are synchronized between the Repository and the Agent.

Factors which can cause the Agent to go out-of-sync with the Repository:

Solution

To solve the issue you have to perform following from the OEM grid control console:

  1. Login to 10.2.0.5 Grid Console as sysman user
  2. Navigate to Setup > Agents > [Click on problematic Agent name].
  3. In Agent home page click on "Resycnhronize" Button on right hand top of the page. Choose the 'Unblock Agent' option and click [Continue].

  Ensure that the Agent is up and running when attempting the Re-synchronization.

Once the re-sycnchronization is completed successfully, the Agent should be able to communicate with the OMS:


[oracle@oemprod ~]$ emctl upload agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully

Re-synchronization Failure

It is possible for the re-synchornization operation to complete with the following error:


Agent Operation completed with errors. For those targets that could not be saved, please go to the target's monitoring configuration page to save them. All other targets have been saved successfully. Agent has not been unblocked.

  1. Error saving target database1.domain:oracle_database - Skipping target {database1.domain, oracle_database}: Missing properties - UserName, password
  2. Error saving target database2.domain:oracle_database - Skipping target {database2.domain, oracle_database}: Missing properties - UserName, password
  3. ...

OR

Error saving target EnterpriseManager0.omsmachine.domain_Web
Cache:oracle_webcache - Skipping target
{EnterpriseManager0.omsmachine.domain_Web Cache, oracle_webcache}:
Missing properties - authpwd, authuser

The above indicates that the Repository has been unable to completely save the details of the all targets that the Agent currently has. In this case, the Agent was re-installed and has newly discovered all the Database targets and iAS targets, which are not completely configured as yet (i.e the monitoring password is missing).

To resolve the above:

  1. Navigate to the Agent home page in the Grid Console (Setup > Agents). Choose one database at a time and click on the [Configure] button.
  2. In the "Monitoring Configuration" page, enter the password for the monitoring user (the dbsnmp user) and save.
  3. Perform the same for any other target that is shown in the result from the Agent re-synchronization.
  4. Once, all the targets are configured in this manner go to: Setup > Agents > [choose this Agent name] and click on the [Unblock] button.

Enterprise Manager Firewall Port Requirements

If your Enterprise Manager grid environment is making use of firewalls, ensure you specify the appropriate ports.

The following figure provides a topology of an Enterprise Manager grid environment that is using a firewall, and also illustrates the appropriate ports that you must specify:

The conventions used in the preceding illustration are as follows:

Conventions
Convention Description
C Is the entity that is making the call.
* Enterprise Manager will default to the first available port within an Enterprise Manager set range.
** Enterprise Manager will default to the first available port.
*** Are the Database listener ports.

Note:

Port Description

These are the ports that need to open for bi-directional data communication between the OEM Grid Control management server and the target agent:

OEM Management Server / Agent
Port(s) Required Description
1521 TNS Listener
3872 OMS / Agent Data Communications
1830-1849  
1159 Enterprise Manager Central Console Secure Port
4898-4989   Console and OMS / Agent Data Communications
4889 OMS / Agent Data Communications
4890-4897  
7200-7210   Oracle HTTP Server Diagnostic port

The ports that should be open on a VPN tunnel to the OEM management console:

OEM Management Server / Agent
Port(s) Required Description
4889 Enterprise Manager Central Console Port
4444   Oracle HTTP Server Listen (SSL) port
1156-1158   Application Server Control port, Oracle Management
Agent Port, Enterprise Manager Central Console Secure Port
1159
7200   Oracle HTTP Server Diagnostic port
7777   Oracle HTTP Server port
7778   Oracle HTTP Server Listen port
8250   Web Cache HTTP Listen (SSL) port
9400-9410   Oracle Web Cache Administration, Console, and Diagnostics

Resolve Duplicate CRS Cluster Name in EM Grid Control - (Grid Control 10g)

During the installation of Oracle Clusterware, you are asked to specify a name for the cluster:

    

Figure 1: Specify Cluster Configuration - Cluster Name

If you're like me, you may opt to keep the CRS Cluster Name set to its default value of "crs". Even when configuring a second or third Oracle RAC within an organization, we many times simply leave the cluster name for the new clusters set to their default value of crs. Although I believe it is bad practice to configure multiple Oracle RAC clusters with the same CRS cluster name, it doesn't necessarily cause any conflicts given these clusters don't interact with each other. The clustered databases work independently from each other without incident. That is, until you register multiple clustered databases with the same CRS cluster name in Oracle Enterprise Manager Grid Control!

An issue can arise when multiple Oracle RAC clusters with the same CRS cluster name are registered as targets in EM Grid Control. While not causing a problem with the cluster itself, it does cause EM Grid Control to think both clusters are the same.

The following article will explain the steps used to change the EM cluster target name(s) in Grid Control from their default name of "crs" to avoid conflict with other cluster targets. In this article, a new EM cluster target name will be created in EM Grid Control for two Oracle RAC database environments that were registered in EM Grid Control with the same EM cluster target name.

Resolve Duplicate CRS Cluster Name in EM Grid Control - (Grid Control 10g)

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Friday, 01-Jun-2012 18:09:46 EDT
Page Count: 27468