DBA Tips Archive for Oracle

  


Resolve Duplicate CRS Cluster Name in EM Grid Control - (Grid Control 10g)

by Jeff Hunter, Sr. Database Administrator

Contents

Introduction

During the installation of Oracle Clusterware, you are asked to specify a name for the cluster:

    

Figure 1: Specify Cluster Configuration - Cluster Name

If you're like me, you may opt to keep the CRS Cluster Name set to its default value of "crs". Even when configuring a second or third Oracle RAC within an organization, we many times simply leave the cluster name for the new clusters set to their default value of crs. Although I believe it is bad practice to configure multiple Oracle RAC clusters with the same CRS cluster name, it doesn't necessarily cause any conflicts given these clusters don't interact with each other. That is, until you register multiple clustered databases with the same CRS cluster name in Oracle Enterprise Manager Grid Control!

 

To verify the cluster name for an Oracle Clusterware installation:

<CRS_HOME>/bin/cemutlo -n

For example:

[oracle@thing1 ~]$ $ORA_CRS_HOME/bin/cemutlo -n
crs

[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/cemutlo -n
crs

Consider the following two Oracle RAC database environments:

Cluster Name Cluster Database Name Machine Name Oracle Instance Name
crs thingdb.idevelopment.info thing1.idevelopment.info thingdb1
thing2.idevelopment.info thingdb2
crs racdb.idevelopment.info racnode1.idevelopment.info racdb1
racnode2.idevelopment.info racdb2

The example above shows two different Oracle RAC database environments configured with the same CRS cluster name; namely crs. Although both clusters share the same CRS cluster name, the clustered databases work independently without any problems.

One issue can arise, however, when they are both registered as targets in EM Grid Control. While not causing a problem with the cluster itself, it does cause EM Grid Control to think both clusters are the same:

    
Figure 2: Oracle RAC / EM Grid Control Example Environment

    
Figure 3: Cluster Name Configuration Conflict in EM Grid Control

The above illustration clearly shows how, during discovery, EM Grid Control configured both Oracle RAC environments under a single EM cluster target name; namely crs.

The goal of this article will be to explain how to change the EM cluster target name(s) in Grid Control from their default target name of crs to avoid conflict with other cluster targets.

In this article, a new EM cluster target name will be created in EM Grid Control for each of the two Oracle RAC database environments as follows:

New EM Cluster Target Name Cluster Database Name Machine Name Oracle Instance Name
thingdb_crs thingdb.idevelopment.info thing1.idevelopment.info thingdb1
thing2.idevelopment.info thingdb2
racdb_crs racdb.idevelopment.info racnode1.idevelopment.info racdb1
racnode2.idevelopment.info racdb2

Please note that the cluster name will only be changed in EM Grid Control. The CRS cluster name for the actual clusters will remain with the same name (e.g. crs).

 

Oracle strongly recommends that all default Oracle Clusterware installations with the default cluster name of crs be renamed in Grid Control so that configuration conflicts do not occur!

The first step will be to remove all associations of Cluster and Oracle RAC targets in EM Grid Control. Next, you will choose one of two methods to re-add both cluster targets to EM Grid Control with their new EM cluster target name. The two methods available to re-add both clusters to EM Grid Control are the (1) Manual Method or the (2) agentca Method. My personal preference is to use the manual method as I've not been able to consistently get the agentca method to work every time.

This document applies to Oracle Enterprise Manager Grid Control - Version: 10.2.0.1 to 10.2.0.5. The two Oracle RAC database environments registered in EM Grid Control use the same version of Oracle Clusterware and Database software; namely Oracle 10g Release 2 (10.2.0.4)

Remove all Associations of Cluster

As mentioned in the Introduction, this article will demonstrate how to change the EM cluster target name for two Oracle RAC database environments in EM Grid Control from their default name of crs to avoid conflict with other cluster targets. For the purpose of this article, I have two, 2-node Oracle RAC 10g database environments whose CRS cluster names are the same (crs). Both clusters have been successfully registered in EM Grid Control.

The first step is to remove all associations of cluster and Oracle RAC targets in EM Grid Control.

 

Any historical data will be lost for the deleted targets!

From within EM Grid Control click:

Setup > Agents

    
Figure 4: EM Grid Control — Setup / Agents

Follow the link for the first Oracle RAC node/agent which in this example is racnode1.idevelopment.info:3872. This will bring up configuration information and all monitored targets for this agent.

From the Agent Setup screen, look for and remove any targets of Type:

    
Figure 5: EM Grid Control — Remove Cluster / Cluster Database Target Types

 

For each agent/node, only remove targets of type Cluster and Cluster Database. Do not remove any other target types like Automatic Storage Management, Listener, Database Instance, Host, etc.

 

If the first target you remove is the shared cluster target (crs), this will remove any cluster databases associated with it. In this case, you would not need to continue searching for and removing the clustered databases mentioned above.

After removing any Cluster and Cluster Database targets from the first node/agent, go back to the Setup > Agents screen and select the next node/agent (i.e. racnode2.idevelopment.info:3872). Continue through all remaining nodes/agents until you have verified all cluster and cluster database targets have been removed.

For the purpose of this example, there will be a total of three targets to remove. Note that the cluster target (crs) will only be associated with one of the node/agent targets; which for this example was racnode1.idevelopment.info:3872:

Target Target Type
crs Cluster
racdb.idevelopment.info Cluster Database
thingdb.idevelopment.info Cluster Database

Re-Add Cluster Targets with New Cluster Target Name

Overview

After removing all associations of both cluster and Oracle RAC targets from within EM Grid Control, the final step is to re-add both cluster targets to EM Grid Control with their new EM cluster target name. As mentioned in the Introduction section, there two methods that can be used to re-add both clusters, both of which are described below.

As previously mentioned, my personal preference is to use the manual method as I've not been able to consistently get the agentca method to work every time.

 

Before adding back the Cluster and Cluster Database targets, check Setup > Management Services and Repository > Overview > Deleted Targets: n

Click on the number (n) and verify that "Time delete completed" (or "Completion time") for all the deleted targets is not null.

Also, make certain that a valid completion date/time is shown for each target deleted in the "Time delete completed" column. The target name/target type cannot be reused until deletion completes. If the "Time delete completed" field shows any target with a value of "Deletion has not completed", log in to that target (node) and manually upload the agent data using emctl:

[oracle@racnode1 ~]$ emctl upload agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

[oracle@racnode2 ~]$ emctl upload agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

[oracle@thing1 ~]$ emctl upload agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

[oracle@thing2 ~]$ emctl upload agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Do not continue with this section until the "Time delete completed" field has a valid date/time value for each deleted target!

Re-Add Cluster Targets with New Cluster Target Name - (Manual Method)

The first method that can be used to re-add all cluster targets back into EM Grid Control with new EM cluster target names is to manually add the targets from within EM Grid Control.

From within EM Grid Control click:

Targets > All Targets

Using the pull-down menu near Add, select "Cluster" and hit the [Go] button:

    
Figure 6: EM Grid Control — Targets / All Targets / Add New Cluster

From the Add Target: Cluster screen, start by adding the new EM cluster target information for the first Oracle RAC cluster (thingdb_crs). Click the [Add] button when finished:

Cluster Name

thingdb_crs

Clusterware Home

/u01/app/crs

Selected Hosts

thing1.idevelopment.info
thing2.idevelopment.info

    
Figure 7: EM Grid Control — Add Target: Cluster

After the "Add Target: Cluster" screen completes, go back and perform the same actions to add the second Oracle RAC cluster (racdb_crs):

Cluster Name

racdb_crs

Clusterware Home

/u01/app/crs

Selected Hosts

racnode1.idevelopment.info
racnode2.idevelopment.info

After adding the new EM cluster target names for both Oracle RAC environments, the final step is to add the appropriate Oracle clustered database to each new EM cluster target. For example, I will be adding the clustered database thingdb.idevelopment.info (and its Oracle instances) to the new thingdb_crs EM cluster target and the racdb.idevelopment.info clustered database to the new racdb_crs EM cluster target. Although this process takes several steps, it is a relatively simple task!

From within EM Grid Control click:

Targets > All Targets

Using the pull-down menu near Add, select "Database Instance" and hit the [Go] button:

    
Figure 8: EM Grid Control — Targets / All Targets / Add New Database Instance

From the Add Database Instance Target: Specify Host screen, start by selecting one of the nodes in one of the Oracle RAC configurations. In this example, I want to add the thingdb.idevelopment.info clustered database first (no reasons in particular) so I'll pick either node thing1.idevelopment.info or thing2.idevelopment.info — either one will work. I'll select thing1.idevelopment.info.

    
Figure 9: EM Grid Control — Add Database Instance Target: Specify Host

Now for the cool part! EM Grid Control identifies that the node being used as the source for the new database is already a member of a cluster! It lists the members of the cluster and asks where to look for the databases to add to Enterprise Manager. Select the radio button option for on all hosts in the cluster and click the [Continue] button.

    
Figure 10: EM Grid Control — Add Database: Specify Source

EM Grid Control will now indicate that the new clustered database is only partially configured. Click the icon in the Configure field to enter the password for the database monitor user (dbsnmp).

    
Figure 11: EM Grid Control — Targets Discovered on Cluster: thingdb_crs

Enter the password for the dbsnmp user, test the connection, and finally click the [Next] button for this screen and the [Ok] button on the next screen.

    
Figure 12: EM Grid Control — Targets Discovered on Cluster: thingdb_crs

This will return you to the Targets Discovered on Cluster: thingdb_crs screen. Click [Ok] to save the new clustered database for the provided EM cluster target.

After the first clustered database has been added, go back and perform the same actions to add the second Oracle RAC cluster database (racdb.idevelopment.info) to the EM cluster target (racdb_crs).

Re-Add Cluster Targets with New Cluster Target Name - (agentca Method)

The second method that can be used to re-add all cluster targets back into EM Grid Control with new EM cluster target names uses the agentca command from one of the nodes in each Oracle RAC configuration. The general syntax is:

<AGENT_HOME>/bin/agentca -d -n <NEW_CLUSTER_TARGET_NAME> -c <NODE1>,<NODE2>,<NODE3> ... etc

For example:

racdb


[oracle@racnode1 ~]$ . oraenv ORACLE_SID = [racdb1] ? agent10g [oracle@racnode1 ~]$ agentca -d -n racdb_crs -c racnode1.idevelopment.info,racnode2.idevelopment.info

thingdb


[oracle@thing1 ~]$ . oraenv ORACLE_SID = [thingdb1] ? agent10g [oracle@thing1 ~]$ agentca -d -n thingdb_crs -c thing1,thing2

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Wednesday, 28-Dec-2011 13:45:28 EST
Page Count: 8122