DBA Tips Archive for Oracle

  


Remove a Node from an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 4.5)

by Jeff Hunter, Sr. Database Administrator


Contents

  1. Overview
  2. Remove the Oracle Instance on the Node to be Deleted
  3. ASM Instance Clean-Up Procedures
  4. Remove the Listener From the Node to be Deleted
  5. Remove the Oracle Database Software From the Node to be Deleted
  6. Remove the Node to be Deleted from Oracle Clusterware
  7. OCFS2 Clean-Up Procedures
  8. Remove Access Permissions on Openfiler for the Node to be Deleted
  9. About the Author



Overview

Although not as exciting as building an Oracle RAC or adding a new node and instance to a clustered database; removing a node from a clustered environment is just as important to understand for a DBA managing Oracle RAC 10g. While it is true that most of the attention in a clustered database environment is focused on extending the database tier to support increased demand, the exact opposite is just as likely to be encountered where the DBA needs to remove a node from an existing RAC environment. It may be that a server is being underutilized in the database cluster and could be better served in another business unit. Another scenario is a node failure. In this case, a node can be removed from the cluster while the remaining nodes continue to service ongoing requests.

This document is an extension to two of my articles: "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". Contained in this new article are the steps required to remove a single node from an already running and configured three-node Oracle RAC 10g Release 2 environment on the CentOS 32-bit (x86) platform. The node being removed is the third node I added in the second article. Although this article was written and tested on CentOS 4.5 Linux, it should work unchanged with Red Hat Enterprise Linux 4 Update 5.

When removing the Oracle Cluster and Oracle Database software, you must first remove the Oracle Database software and then remove the Oracle Clusterware from the node that you are deleting. In other words, you remove the software components from the node you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components. It is important that you perform each step contained this article in the order provided.

This article assumes the following:

The reader has already built and configured a three-node Oracle RAC 10g Release 2 environment using the articles "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". These two articles provide comprehensive instructions for building a two-node RAC cluster and then adding a third node to the cluster named linux3. All Oracle RAC nodes are single processor machines running CentOS 4.5, Oracle RAC 10g Release 2, OCFS2, and ASMLib 2.0. The current three-node RAC environment actually consists of four machines — three named linux1, linux2, and linux3 which each run an Oracle10g instance and a forth node to run the network storage server named openfiler1.

Note: The current three-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to version 10.2.0.3.0 by applying the 5337014 patchset (p5337014_10203_LINUX.zip). The patchset was applied to Oracle Clusterware and the Oracle Database software. I also applied the one-off patchset - "BUNDLE Patch for Base Bug 6000740" (MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle Database software. The procedures for installing both patchsets are not included in any of the parent article(s).

The third node in the current Oracle RAC environment named linux3 (running the orcl3 instance) will be removed from the cluster making it a two-node cluster.

Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current three-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software.

The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system will be included.

Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to removing the ASM instance from the deleted node, we will also be removing ASMLib from this node.

During the creation of the existing two-node cluster and the addition of the third node, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster — namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that the oracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely removed from the node to be deleted (linux3). This task will need to be performed from a node that is to remain a part of the RAC cluster database environment (linux1 in this example) using ssh and scp without being prompted for a password.

All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). While removing the node linux3 from the Oracle RAC cluster, we will also be removing access permissions to the Openfiler server from linux3.

The following is a conceptual look at what the environment will look like after removing the third Oracle RAC node (linux3) from the cluster. Click on the graphic below to enlarge the image:

Oracle RAC 10g Release 2 Environment

Figure 1: Removing linux3 from the current Oracle RAC 10g Release 2 Environment


  While this article provides comprehensive instructions for successfully removing a node from an existing Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, and administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com.

  Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux
  Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g Release 2 (10.2)
  2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)



Remove the Oracle Instance on the Node to be Deleted


  Remove the Oracle instance running on the node to be deleted using DBCA!


Overview

Before removing any of the Oracle software from the node to be deleted (linux3), the first step is to remove the Oracle instance (orcl3) from the RAC database. This task can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA.

The DBCA will need to be run from a node other than the node that hosts the instance you want to delete. The suggested node to run the DBCA from is the same node used to install the original Oracle RAC environment from which for this example is linux1.


Verifying Terminal Shell Environment

Before starting the DBCA, you should first verify you are logged onto the server you will be running the DBCA from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster:

Verify Server and Enable X Server Access

# hostname
linux1

# xhost +
access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1

$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2

$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3


Remove Database Instance

To start the database instance removal process, run the following from linux1:

$ dbca &
Screen Name Response
Welcome Screen Select Oracle Real Application Clusters database.
Operations Select Instance Management.
Instance Management Select Delete an instance.
List of cluster databases Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is orcl. Select this clustered database.

At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges":

    Username: sys
    Password: <sys_password>

Click Next to continue.

List of cluster
database instances
This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on.

Select the node:instance to delete (linux3:orcl3) and Click Next to continue.

Database Services If the current clustered database has any database services defined, the next screen allows the DBA to re-configure those database services to account for the instance being removed. In this example, the existing clustered database has one service defined named orcl_taf. With the "orcl_taf" database service selected, change the details to Not used for the instance to be removed (orcl3) and leave the "TAF Policy" set to Basic.

Click Finish to remove the instance.

Database Configuration
Assistant: Summary
After verifying the instance removal options in the summary dialog, Click OK to begin the instance management process.
Database Configuration Assistant A second dialog box is presented which includes the warning:

"The Database Configuration Assistance will delete the Oracle instance and its associated OFA directory structure. All information about this instance will be deleted. Do you want to proceed?"

Click OK to acknowledge this dialog.

Database Configuration Assistant
Progress Screen
A progress bar is display while the instance is being deleted. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform another operation?" Click No to end and exit the DBCA utility.

When the Oracle Database Configuration Assistant has completed, the orcl3 instance (and associated services) will have been successfully removed from linux3!


Verify Database Instance and Associated Objects Were Removed

Check Cluster Services

$ $ORA_CRS_HOME/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    linux1
ora....X1.lsnr application    ONLINE    ONLINE    linux1
ora.linux1.gsd application    ONLINE    ONLINE    linux1
ora.linux1.ons application    ONLINE    ONLINE    linux1
ora.linux1.vip application    ONLINE    ONLINE    linux1
ora....SM2.asm application    ONLINE    ONLINE    linux2
ora....X2.lsnr application    ONLINE    ONLINE    linux2
ora.linux2.gsd application    ONLINE    ONLINE    linux2
ora.linux2.ons application    ONLINE    ONLINE    linux2
ora.linux2.vip application    ONLINE    ONLINE    linux2
ora....SM3.asm application    ONLINE    ONLINE    linux3
ora....X3.lsnr application    ONLINE    ONLINE    linux3
ora.linux3.gsd application    ONLINE    ONLINE    linux3
ora.linux3.ons application    ONLINE    ONLINE    linux3
ora.linux3.vip application    ONLINE    ONLINE    linux3
ora.orcl.db    application    ONLINE    ONLINE    linux1
ora....l1.inst application    ONLINE    ONLINE    linux1
ora....l2.inst application    ONLINE    ONLINE    linux2
ora...._taf.cs application    ONLINE    ONLINE    linux2
ora....cl1.srv application    ONLINE    ONLINE    linux1
ora....cl2.srv application    ONLINE    ONLINE    linux2

- or -

$ rac_crs_stat
HA Resource                                   Target     State
-----------                                   ------     -----
ora.linux1.ASM1.asm                           ONLINE     ONLINE on linux1
ora.linux1.LISTENER_LINUX1.lsnr               ONLINE     ONLINE on linux1
ora.linux1.gsd                                ONLINE     ONLINE on linux1
ora.linux1.ons                                ONLINE     ONLINE on linux1
ora.linux1.vip                                ONLINE     ONLINE on linux1
ora.linux2.ASM2.asm                           ONLINE     ONLINE on linux2
ora.linux2.LISTENER_LINUX2.lsnr               ONLINE     ONLINE on linux2
ora.linux2.gsd                                ONLINE     ONLINE on linux2
ora.linux2.ons                                ONLINE     ONLINE on linux2
ora.linux2.vip                                ONLINE     ONLINE on linux2
ora.linux3.ASM3.asm                           ONLINE     ONLINE on linux3
ora.linux3.LISTENER_LINUX3.lsnr               ONLINE     ONLINE on linux3
ora.linux3.gsd                                ONLINE     ONLINE on linux3
ora.linux3.ons                                ONLINE     ONLINE on linux3
ora.linux3.vip                                ONLINE     ONLINE on linux3
ora.orcl.db                                   ONLINE     ONLINE on linux1
ora.orcl.orcl1.inst                           ONLINE     ONLINE on linux1
ora.orcl.orcl2.inst                           ONLINE     ONLINE on linux2
ora.orcl.orcl_taf.cs                          ONLINE     ONLINE on linux2
ora.orcl.orcl_taf.orcl1.srv                   ONLINE     ONLINE on linux1
ora.orcl.orcl_taf.orcl2.srv                   ONLINE     ONLINE on linux2


Verify Instance we Removed

Login to one of the instances and query the gv$instance view:
SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS')
  2  from gv$instance order by inst_id;

   INST_ID INSTANCE_NAME    STATUS       TO_CHAR(STARTUP_TIME
---------- ---------------- ------------ --------------------
         1 orcl1            OPEN         27-FEB-2008 01:26:52
         2 orcl2            OPEN         27-FEB-2008 01:27:22


Verify Database Files / Tablespaces have been Removed

Check that the UNDO tablespace that was defined for orcl3 (UNDOTBS3) has been removed:
SQL> select tablespace_name from dba_tablespaces order by 1;

TABLESPACE_NAME
------------------
EXAMPLE
SOE
SOEINDEX
SYSAUX
SYSTEM
TEMP
UNDOTBS1
UNDOTBS2
USERS
Check that the online redo log files that were defined for orcl3 have been removed:
SQL> select inst_id, group#, member from gv$logfile order by 1,2,3;

INST_ID GROUP# MEMBER
------- ------ ---------------------------------------------------------
      1      1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
      1      1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
      1      2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
      1      2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
      1      3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
      1      3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
      1      4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
      1      4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299
      2      1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
      2      1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
      2      2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
      2      2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
      2      3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
      2      3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
      2      4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
      2      4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299


Update TNSNAMES

Login to all machines that were accessing the instance removed and update the tnsnames.ora file (if necessary).



ASM Instance Clean-Up Procedures


  Remove ASM instance and ASMLib from the node to be deleted!


Overview

When the clustered database is running on ASM (as is the case in this example), the steps in this section describe how to remove the ASM instance (and ASMLib) from the node to be deleted. If the Oracle RAC database is not running on ASM, this section can be skipped.

The tasks in this section will need to be run from a node other than the node that hosts the ASM instance you want to delete. The suggested node to run these steps from is the same node used to install the original Oracle RAC environment from which for this example is linux1.


Remove the ASM Instance from the Oracle Cluster Registry File

Run the following from linux1 as the oracle user account:
$ srvctl stop asm -n linux3

$ srvctl remove asm -n linux3


Remove ASM Directories from the Node to be Deleted

Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
$ cd $ORACLE_BASE/admin

$ rm -rf +ASM


$ cd $ORACLE_HOME/dbs

$ rm -f *ASM*

  You may also have to edit the /etc/oratab file on the node to be deleted and remove entries beginning with +ASM to delete the references of the ASM instance.


Remove ASMLib from the Node to be Deleted

The Linux nodes in the current Oracle RAC database make use of ASMLib so that the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls.

In this section we will remove the ASMLib kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:

# /etc/init.d/oracleasm stop
Unmounting ASMlib driver filesystem: [  OK  ]
Unloading module "oracleasm": [  OK  ]

# rpm -qa | grep oracleasm
oracleasm-support-2.0.3-1
oracleasm-2.6.9-55.EL-2.0.3-1
oracleasmlib-2.0.2-1

# rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave

# rm -f /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/rc.d/init.d/oracleasm
# rm -f /etc/rc0.d/*oracleasm*
# rm -f /etc/rc1.d/*oracleasm*
# rm -f /etc/rc2.d/*oracleasm*
# rm -f /etc/rc3.d/*oracleasm*
# rm -f /etc/rc4.d/*oracleasm*
# rm -f /etc/rc5.d/*oracleasm*
# rm -f /etc/rc6.d/*oracleasm*



Remove the Listener From the Node to be Deleted


  Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will remove the node-specific listener from the clustered configuration for the node to be deleted.


Overview

Each of the nodes in the current Oracle RAC environment use a node-specific listener named LISTENER_nodename which runs out of the local Oracle home.

In this section, you will use the Network Configuration Assistant (NETCA) to delete the TNS listener for the node to be deleted from the existing clustered configuration. The NETCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Although it is possible to re-create this listener in another Oracle home (on a different node), we will simply be removing it from the cluster.


Verifying Terminal Shell Environment

As was performed earlier, the terminal shell environment needs to be configured for remote access and user equivalence to all Oracle RAC nodes before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable:

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1

$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2

$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3


Run the Network Configuration Assistant

To start the NETCA, run the following from linux1:
$ netca &

The following table walks you through the process of reconfiguring the clustered TNS listeners to remove the node-specific listener from the node to be deleted.

Screen Name Response
Select the Type of Oracle
Net Services Configuration
Select Cluster configuration
Select the nodes to configure Only select the Oracle RAC node to remove the listener from: linux3.
Type of Configuration Select Listener configuration.
Listener Configuration
Next 4 Screens
The following screens are now like any other normal listener configuration when removing a listener:
   What do you want to do: Delete
   Select a listener you want to delete: LISTENER
   Acknowledge warning dialog: Yes
   Listener deleted [ Next ]
   Listener configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Click Finish to exit the NETCA.


Verify TNS Listener Configuration

The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC cluster (linux1 and linux2):
$ hostname
linux1

$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX1

$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr
NAME=ora.linux1.LISTENER_LINUX1.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux1

=====================

$ hostname
linux2

$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX2

$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr
NAME=ora.linux2.LISTENER_LINUX2.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux2

=====================

$ hostname
linux3

$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'

$ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.



Remove the Oracle Database Software From the Node to be Deleted


  Remove the Oracle Database software and update the Node List for the node to be deleted!


Overview

In this section, you will be removing the node to be deleted (linux3) from the Node List for its Oracle inventory. Next will be to remove the Oracle Database software from the node to be deleted using the Oracle Universal Installer (OUI). Finally, update the Oracle inventory on all remaining Oracle RAC nodes to reflect the nodes which will remain in the cluster.


Update Node List for Oracle Database Software - (Remove linux3)

From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Database software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES="" -local
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.

  Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!


De-install Oracle Database Software

Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Database software. Make certain that you choose the home to be removed and not just the products under that home.

From linux3 as the oracle user account, run the following:

$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller
Screen Name Response
Welcome Screen Click the Installed Products button.
Inventory: Contents Tab Check the Oracle home to deleted (OraDb10g_home1) and click the Remove button.
Confirmation Acknowledge the warning dialog by clicking Yes.
Deinstallation Process A progress bar is displayed while the Oracle Database software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Database software (Oracle home) was successfully removed, Click Close to exit this dialog.
Welcome Screen Click Cancel to exit the OUI.


Update Node List for Remaining Nodes in the Cluster

Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={linux1,linux2}"
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.

  Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!



Remove the Node to be Deleted from Oracle Clusterware


  Remove the node to be deleted from Oracle Clusterware and update the Node List for all remaining nodes!


Overview

After successfully removing the Oracle Database software from the node to be deleted, the next major step is to remove the node to be deleted from Oracle Clusterware.


Remove Node-Specific Interface Configuration

Run the following commands to remove node-specific interface configurations from the node to be deleted. For this example, run these commands from linux3 as the oracle user account:
$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200

  In the above command, replace 6200 with the port number you get in the remoteport section of the ons.config file found in the $ORA_CRS_HOME/opmn/conf directory:
$ grep '^remoteport' $ORA_CRS_HOME/opmn/conf/ons.config
remoteport=6200

$ $ORA_CRS_HOME/bin/oifcfg delif -node linux3
PROC-4: The cluster registry key to be operated on does not exist.
PRIF-11: cluster registry error

  From the research that I have performed, I believe the above error can be safely ignored. The "oifcfg delif" command attempts to delete a node-specific interface defined for node linux3 in the Oracle Cluster Registry file (OCR). This means it is looking for the presence of a key in the OCR file named "SYSTEM.css.interfaces.nodelinux3". If you were to dump the contents of the OCR file, you would quickly see that no key is present with that name:
$ $ORA_CRS_HOME/bin/ocrdump
$ grep 'SYSTEM.css.interfaces.nodelinux3' OCRDUMPFILE
After examining the contents of the OCRDUMPFILE, interfaces are indeed defined, however they are identified as global:
$ grep 'SYSTEM.css.interfaces.global' OCRDUMPFILE
[SYSTEM.css.interfaces.global]
[SYSTEM.css.interfaces.global.eth0]
[SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0]
[SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0.1]
[SYSTEM.css.interfaces.global.eth2]
[SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0]
[SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0.1]
Given there are no node-specific interface entries defined and we obviously don't want to delete the interface keys configured globally, the oifcfg delif command and its errors can be safely ignored!


Disable Oracle Clusterware Applications

From the node you are deleting from the cluster (linux3), run the script $ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle Clusterware applications that are on the node. This script should only be run once. Given the Clusterware software install is on local disk (non-shared), make certain to use the nosharedhome argument. The default for this script is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user account.

Running this script will stop the CRS stack and delete the ocr.loc file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file sytem.

While logged into linux3 as the root user account, run the following:

$ su
# cd $ORA_CRS_HOME/install
# ./rootdelete.sh local nosharedvar nosharedhome
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'


Delete Node from Cluster and Update OCR

Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script to delete the node (linux3) from the Oracle cluster and to update the Oracle Cluster Registry (OCR). This script should be run from a pre-existing / available node in the cluster (linux1) as the root user account:

Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from linux1:

$ $ORA_CRS_HOME/bin/olsnodes -n
linux1  1
linux2  2
linux3  3

From the listing above, the node number for linux3 is 3.

While logged into linux1 as the root user account, run the following using the name linux3 and the node number 3:

$ su
# cd $ORA_CRS_HOME/install
# ./rootdeletenode.sh linux3,3
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
CRS-0210: Could not find resource 'ora.linux3.ons'.
CRS-0210: Could not find resource 'ora.linux3.vip'.
CRS-0210: Could not find resource 'ora.linux3.gsd'.
CRS-0210: Could not find resource ora.linux3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'linux3,3' deleted successfully

To verify that the node was successfully removed, use the following as either the oracle or root user:

$ $ORA_CRS_HOME/bin/olsnodes -n
linux1  1
linux2  2


Update Node List for Oracle Clusterware Software - (Remove linux3)

From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Clusterware software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES="" -local CRS=true
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.

  Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!


De-install Oracle Clusterware Software

Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Clusterware software. Make certain that you choose the home to be removed and not just the products under that home.

From linux3 as the oracle user account, run the following:

$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller
Screen Name Response
Welcome Screen Click the Installed Products button.
Inventory: Contents Tab Check the Oracle home to deleted (OraCrs10g_home) and click the Remove button.
Confirmation Acknowledge the warning dialog by clicking Yes to remove the Oracle Clusterware software and to remove the /u01/app/crs directory.
Deinstallation Process A progress bar is displayed while the Oracle Clusterware software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Clusterware software (Clusterware home) was successfully removed, Click Close to exit this dialog.
Welcome Screen Click Cancel to exit the OUI.


Update Node List for Remaining Nodes in the Cluster

Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY

$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={linux1,linux2}" CRS=true
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.

  Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!


Verify Node to be Deleted is no Longer a Member of the Cluster

Run the following commands to verify that the node to be deleted from the Oracle RAC cluster is no longer a member of the cluster and to verify that the Oracle Clusterware components have been successfully removed from that node.

Run the following commands from linux1 as the oracle user account:


$ srvctl status nodeapps -n linux3
PRKC-1056 : Failed to get the hostname for node linux3
PRKH-1001 : HASContext Internal Error
  [OCR Error(Native: getHostName:[21])]
The error above indicates that linux3 is no longer a member of the cluster.


$ $ORA_CRS_HOME/bin/crs_stat | grep -i linux3
You should not see any output from the above command


$ $ORA_CRS_HOME/bin/olsnodes -n
linux1  1
linux2  2
You should see the present node list without the deleted node (that is linux1 and linux2 only).


Remove/Rename any Remaining Oracle Files from Node to be Deleted

From the node to be deleted (linux3), remove/rename any remaining Oracle files while logged in as the root user account:
# mv -f /etc/inittab.no_crs /etc/inittab
# rm -f /etc/inittab.orig
# rm -f /etc/inittab.crs

# rm -rf /etc/oracle
# rm -f /etc/oratab
# rm -f /etc/oraInst.loc
# rm -rf /etc/ORCLcluster
# rm -rf /u01/app/oracle
# rm -rf /u01/app/crs
# rm -f /usr/local/bin/coraenv
# rm -f /usr/local/bin/dbhome
# rm -f /usr/local/bin/oraenv
Finally, remove the Oracle user account and all associated UNIX groups from linux3:
# userdel -r oracle
# groupdel oinstall
# groupdel dba



OCFS2 Clean-Up Procedures


  Remove OCFS2 from the node to be deleted!


Overview

The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. It is also used to store the shared SPFILE for ASM. This section includes instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system.


Remove OCFS2 from the Node to be Deleted

In this section we will remove the OCFS2 kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload

# rpm -qa | grep ocfs2
ocfs2-tools-1.2.4-1
ocfs2-2.6.9-55.EL-1.2.5-6
ocfs2console-1.2.4-1

# rpm -ev ocfs2-tools-1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1
warning: /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave

# rm -f /etc/sysconfig/o2cb.rpmsave
# rm -rf /etc/ocfs2


Remove Node to be Deleted from the OCFS2 Cluster

The next step is to remove the Oracle RAC node being deleted (linux3) from the "live" OCFS2 cluster. This entails running the o2cb_ctl command-line utility from the remaining two RAC nodes linux1 and linux2.

As root, run the following from linux1 and then linux2:

[root@linux1 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported

[root@linux2 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported

o2cb_ctl parameters:

-D  : Delete an object from the existing OCFS2 Cluster Configuration.
-u  : Valid only with -D. When deleting something (node or cluster),
      it will also remove it from the live cluster (/config). If the 
      parameter is not specified, then only update the 
      /etc/ocfs2/cluster.conf.
-n  : Object name which is usually the node name or cluster name.

  At the time of this writing, the latest version of OCFS2 (namely the o2cb cluster stack) only supports "adding" nodes dynamically. Deleting a node requires the OCFS2 cluster to be down which means a database outage. This is a serious shortcoming with OCFS2 that hasn't been addressed for some time. Currently, an enhancement request (oss.oracle.com Bugzilla Bug 694) has been submitted to Oracle, however no target date or version has been set for this enhancement.

Removing linux3 from the OCFS2 cluster will have to be performed manually during the next scheduled database outage. After bringing the clustered database and all Oracle Clusterware services down on both linux1 and linux2, remove the following stanza from the /etc/ocfs2/cluster.conf file on both Oracle RAC nodes:

node:
        ip_port = 7777
        ip_address = 192.168.2.107
        number = 2
        name = linux3
        cluster = ocfs2
Before closing out the /etc/ocfs2/cluster.conf file, change the value of node_count to 2 on both Oracle RAC nodes:
node_count = 2

After modifying the /etc/ocfs2/cluster.conf file on both Oracle RAC nodes, unmount the OCFS2 file system and restart the o2cb cluster stack on both nodes:

# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload

# /etc/init.d/o2cb load
# /etc/init.d/o2cb online ocfs2

# mount /u02
After mounting the OCFS2 file system on both Oracle RAC nodes, verify that linux3 does not exist in the live OCFS2 cluster - only linux1 and linux2 should be present:
[root@linux1 node]# ls -l /config/cluster/ocfs2/node
total 0
drwxr-xr-x  2 root root 0 Mar  1 01:32 linux1
drwxr-xr-x  2 root root 0 Mar  1 01:32 linux2

[root@linux2 ~]# ls -l /config/cluster/ocfs2/node
total 0
drwxr-xr-x  2 root root 0 Mar  1 01:33 linux1
drwxr-xr-x  2 root root 0 Mar  1 01:33 linux2


Remove OCFS2 Mount Options from /etc/fstab

Finally, remove the following entry from the /etc/fstab from linux3:
LABEL=oracrsfiles  /u02   ocfs2   _netdev,datavolume,nointr     0 0



Remove Access Permissions on Openfiler for the Node to be Deleted


  Remove network access permissions in Openfiler for linux3 and disable auto-discovery of iSCSI volumes from linux3!


Overview

At this point, linux3 has been successfully removed from the Oracle RAC configuration. Remember that in the existing Oracle RAC configuration that all shared disk storage is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). When we added linux3 to the current Oracle RAC configuration in the article "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)", we needed to configure network access in Openfiler so that linux3 had permissions to the shared iSCSI volumes. We also configured the iSCSI initiator on linux3 to automatically discover all available iSCSI volumes on each boot.

In this final section of the article, we will be revoking network access in Openfiler for the node linux3 and lastly, modifying the iSCSI initiator on linux3 to disable automatic discovery of iSCSI volumes from Openfiler. For the purpose of this example, all iSCSI traffic was configured to use the private network interface eth1 which in this article is on the 192.168.2.0 network.

Revoking Openfiler network access permissions from linux3 can be done using the Openfiler Storage Control Center — a browser based tool over an https connection on port 446. For example:

https://openfiler1:446/
From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:

The first page the administrator sees is the [Accounts] / [Authentication] screen. Configuring user accounts and groups is not necessary for this article and will therefore not be discussed.


Revoke Access from linux3 to each iSCSI Volume

The first step in this process is to remove access from linux3 to each individual iSCSI volume within Openfiler.

The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1 which is used for the existing Oracle RAC configuration.

iSCSI / Logical Volumes in Volume Group rac1
Volume Name Volume Description Required Space (MB) Filesystem Type
crs Oracle Clusterware 2,048 iSCSI
asm1 Oracle ASM Volume 1 118,720 iSCSI
asm2 Oracle ASM Volume 2 118,720 iSCSI
asm3 Oracle ASM Volume 3 118,720 iSCSI
asm4 Oracle ASM Volume 4 118,720 iSCSI

To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. There we will see all five logical volumes within the volume group rac1:


Figure 2: Current Logical (iSCSI) Volumes

For each of the five logical volumes, click on the 'Edit' link (under the Properties column). This will bring up the 'Edit properties' screen for that volume. Scroll to the bottom of this screen; change the access for host linux3-priv from 'Allow' to 'Deny' and click the 'Update' button. Perform this task for all five logical volumes.


Figure 3: Deny Host Access to Logical (iSCSI) Volumes


Remove linux3 from Openfiler Local Network Configuration

The next step is to modify the local network configuration within Openfiler by removing the node linux3 (which was actually entered as node linux3-priv).

This task can be completed using the Openfiler Storage Control Center by navigating to [General] / [Local Networks]. The Local Networks screen allows an administrator to setup (add/remove) networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we want to remove linux3 from the Openfiler local network configuration.

Simply, check the linux3-priv / 192.168.2.107 row and click the [Update] button at the bottom of the screen to remove linux3 from the Openfiler local network configuration.

The following image shows the step required to remove linux3 and update the local network configuration within Openfiler:


Figure 4: Configure Openfiler Host Access for deleted Oracle RAC Node


Disable iSCSI Initiator Service on linux3

The final step in this process is to modify the iSCSI initiator service on linux3 so it will not automatically start and therefore will not attempt to discover iSCSI volumes from the Openfiler server.

First, stop the iscsi initiator service on linux3:

# service iscsi stop
Searching for iscsi-based multipath maps
Found 0 maps
Stopping iscsid: [  OK  ]
Removing iscsi driver: [  OK  ]
Next, edit the file /etc/iscsi.conf on linux3 and comment out (or remove) the DiscoveryAddress entry which specifies the hostname of the Openfiler network storage server. In our example, use a # at the beginning of DiscoveryAddress entry as shown below:
...
# DiscoveryAddress=openfiler1-priv
...
Finally, configure the iSCSI service to not be active across machine reboots on linux3. The Linux command chkconfig can be used to achieve that as follows:
# chkconfig --level 345 iscsi off



About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Saturday, 18-Sep-2010 17:44:12 EDT
Page Count: 66314