Oracle DBA Tips Corner
Return to the Oracle DBA Tips Corner.
|
Oracle DBA Tips Corner
|
Removing a Node from an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)
by Jeff Hunter, Sr. Database Administrator
Contents
Overview
This document is an extension to two of my articles:
"Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)"
and
"Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)".
Contained in this new article are the steps required to remove a single node
from an already running and configured three-node Oracle RAC 10g Release 2 environment on the
CentOS 32-bit (x86) platform. The node being removed is the third node I added in the
second article.
Although this article was written and tested on CentOS 4.5 Linux, it should work unchanged with
Red Hat Enterprise Linux 4 Update 5.
When removing the Oracle Cluster and Oracle Database software,
you must first remove the Oracle Database software and then remove the Oracle Clusterware
from the node that you are deleting. In other words, you remove the software
components from the node you are deleting in the reverse order that you
originally installed the clusterware and Oracle database software components. It
is important that you perform each step contained this article in the order provided.
This article assumes the following:
Note: The current three-node Oracle RAC
environment has been upgraded from its base release (10.2.0.1.0) to
version 10.2.0.3.0 by applying the 5337014 patchset (p5337014_10203_LINUX.zip).
The patchset was applied to Oracle Clusterware and the Oracle Database
software. I also applied the one-off patchset - "BUNDLE Patch for Base Bug 6000740"
(MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle
Database software. The procedures for installing both patchsets are
not included in any of the parent article(s).
The following is a conceptual look at what the environment will look like
after removing the third Oracle RAC node (linux3) from the cluster. Click on the
graphic below to enlarge the image:
Figure 1: Removing linux3 from the current Oracle RAC 10g Release 2 Environment
Remove the Oracle Instance on the Node to be Deleted
The DBCA will need to be run from a node other than the node that hosts
the instance you want to delete. The suggested node to run the DBCA from
is the same node used to install the original Oracle RAC environment from which for this
example is linux1.
Verify Server and Enable X Server Access
Login as the oracle User Account and Set DISPLAY (if necessary)
Verify Remote Access / User Equivalence
When using the secure shell method, user equivalence
will need to be enabled on any new terminal shell session
before attempting to run the DBCA. To enable
user equivalence for the current terminal shell session, perform the
following steps remembering to enter the pass phrase for the RSA key
you generated when prompted:
At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA
system privileges":
Username: sys
Click Next to continue.
Select the node:instance to delete (linux3:orcl3) and
Click Next to continue.
Click Finish to remove the instance.
"The Database Configuration Assistance will delete the Oracle instance
and its associated OFA directory structure. All information about this instance will
be deleted. Do you want to proceed?"
Click OK to acknowledge this dialog.
When the Oracle Database Configuration Assistant has completed, the
orcl3 instance (and associated services) will have been
successfully removed from linux3!
Check Cluster Services
ASM Instance Clean-Up Procedures
The tasks in this section will need to be run from a node other than
the node that hosts the ASM instance you want to delete. The suggested node
to run these steps from is the same node used to install the original
Oracle RAC environment from which for this example is linux1.
In this section we will remove the ASMLib kernel driver, supporting software,
and associated directories. Run the following from the node to be
deleted (linux3) as the root user account:
Remove the Listener From the Node to be Deleted
In this section, you will use the Network Configuration Assistant (NETCA)
to delete the TNS listener for the node to be deleted from the
existing clustered configuration. The NETCA program will be run from
linux1 with user equivalence enabled to all nodes in the cluster.
Although it is possible to re-create this listener in another Oracle home
(on a different node), we will simply be removing it from the cluster.
Login as the oracle User Account and Set DISPLAY (if necessary)
Verify Remote Access / User Equivalence
When using the secure shell method, user equivalence
will need to be enabled on any new terminal shell session
before attempting to run the NETCA. To enable
user equivalence for the current terminal shell session, perform the
following steps remembering to enter the pass phrase for the RSA key
you generated when prompted:
The following table walks you through the process of reconfiguring the clustered TNS
listeners to remove the node-specific listener from the node to be deleted.
Remove the Oracle Database Software From the Node to be Deleted
From linux3 as the oracle user account, run the following:
Remove the Node to be Deleted from Oracle Clusterware
Running this script will stop the CRS stack and delete the ocr.loc
file on the node to be removed. The nosharedvar option
assumes the ocr.loc file is not on a shared file sytem.
While logged into linux3 as the root user account, run
the following:
Before executing rootdeletenode.sh, we need to know the
node number associated with the node name to be deleted from
the cluster. To determine the node number, run the following
command as the oracle user account from linux1:
From the listing above, the node number for linux3 is 3.
While logged into linux1 as the root user account, run
the following using the name linux3 and the node number 3:
To verify that the node was successfully removed, use the following as
either the oracle or root user:
From linux3 as the oracle user account, run the following:
Run the following commands from linux1 as the oracle user
account:
OCFS2 Clean-Up Procedures
As root, run the following from linux1 and then linux2:
o2cb_ctl parameters:
Removing linux3 from the OCFS2 cluster will
have to be performed manually during the next scheduled database outage.
After bringing the clustered database and all Oracle Clusterware services down on both
linux1 and linux2, remove the following stanza from the
/etc/ocfs2/cluster.conf file on both Oracle RAC nodes:
After modifying the /etc/ocfs2/cluster.conf file on both
Oracle RAC nodes, unmount the OCFS2 file system
and restart the o2cb cluster stack on both nodes:
Remove Access Permissions on Openfiler for the Node to be Deleted
In this final section of the article, we will be revoking network access in Openfiler
for the node linux3 and lastly, modifying the iSCSI initiator
on linux3 to disable automatic discovery of iSCSI volumes from
Openfiler. For the purpose of this example, all iSCSI traffic was configured to
use the private network interface eth1 which in this article
is on the 192.168.2.0 network.
Revoking Openfiler network access permissions from linux3 can be done
using the Openfiler Storage Control Center —
a browser based tool over an https connection on port 446. For example:
The first page the administrator sees is the
[Accounts] / [Authentication] screen. Configuring user accounts
and groups is not necessary for this article and will
therefore not be discussed.
The current Openfiler configuration contains five logical iSCSI volumes
in a single volume group named rac1 which is used for
the existing Oracle RAC configuration.
To view the available iSCSI volumes from within the Openfiler Storage Control Center,
navigate to [Volumes] / [List of Existing Volumes]. There we will see all five
logical volumes within the volume group rac1:
For each of the five logical volumes, click on the 'Edit' link (under
the Properties column). This will bring up the 'Edit properties' screen for that
volume. Scroll to the bottom of this screen; change the access for host linux3-priv from
'Allow' to 'Deny' and click the 'Update' button. Perform this task for all five logical volumes.
This task can be completed using the Openfiler Storage Control Center
by navigating to [General] / [Local Networks]. The Local Networks screen allows
an administrator to setup (add/remove) networks and/or hosts that will be allowed to
access resources exported by the Openfiler appliance. For the purpose of this
article, we want to remove linux3 from the Openfiler local network configuration.
Simply, check the linux3-priv / 192.168.2.107 row and click the [Update]
button at the bottom of the screen to remove linux3 from the Openfiler local
network configuration.
The following image shows the step required to remove linux3
and update the local network configuration within Openfiler:
First, stop the iscsi initiator service on linux3:
All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter
and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express,
prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.
I have made every effort and taken great care in making sure that the material included on my web site is technically accurate,
but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from
relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Although not as exciting as building an Oracle RAC or adding a new node and instance
to a clustered database; removing a node from a clustered environment is just as important to
understand for a DBA managing Oracle RAC 10g. While it is true that most
of the attention in a clustered database environment is focused on extending the database
tier to support increased demand, the exact opposite is just as likely to be encountered where
the DBA needs to remove a node from an existing RAC environment. It may be that
a server is being underutilized in the database cluster and could be better served in another business unit.
Another scenario is a node failure. In this case, a node can be removed from the cluster
while the remaining nodes continue to service ongoing requests.
The reader has already built and configured a three-node
Oracle RAC 10g Release 2 environment using the articles
"Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)"
and
"Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)".
These two articles provide comprehensive instructions for building a
two-node RAC cluster and then adding a third node to the cluster named linux3.
All Oracle RAC nodes are single processor machines
running CentOS 4.5, Oracle RAC 10g Release 2,
OCFS2, and ASMLib 2.0. The current three-node RAC environment
actually consists of four machines — three named
linux1, linux2, and linux3 which each run
an Oracle10g instance and a forth node to
run the network storage server named openfiler1.
The third node in the current Oracle RAC environment named linux3 (running the orcl3 instance)
will be removed from the cluster making it a two-node cluster.
Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software
installed on their local disks. The current three-node Oracle RAC environment does not
use shared Oracle homes for the Clusterware or Database software.
The existing Oracle RAC 10
Automatic Storage Management (ASM) is being used as the file system and volume manager
for all Oracle physical database files (data, online redo logs, control files, archived redo logs)
and a Flash Recovery Area. In addition to removing the ASM instance from the deleted node,
we will also be removing ASMLib from this node.
During the creation of the existing two-node cluster and the addition of the third node,
the installation of Oracle Clusterware and the Oracle Database software were only
performed from one node in the
RAC cluster — namely from linux1 as the oracle user account.
The Oracle Universal Installer (OUI) on that particular node would then use the
ssh and scp commands to run remote commands on and
copy files (the Oracle software) to all other nodes within the RAC cluster.
The oracle user account on the node running the OUI
(runInstaller) had to be trusted by all other nodes in the
RAC cluster. This meant that the oracle user account had to
run the secure shell commands (ssh or scp) on the
Linux server executing the OUI (linux1) against all other Linux servers in
the cluster without being prompted for a password. The same security requirements hold
true for this article. User equivalence will be configured so that the
Oracle Clusterware and Oracle Database software will be securely removed from the
node to be deleted (linux3). This task will need to be performed from a node that
is to remain a part of the RAC cluster database environment (linux1 in this example)
using ssh and scp without being prompted for a password.
All shared disk storage for the existing Oracle RAC is based on
iSCSI
using a Network Storage Server; namely
Openfiler Release 2.2 (respin 2). While removing the node linux3 from the
Oracle RAC cluster, we will also be removing access permissions to the Openfiler server
from linux3.
)
While this article provides comprehensive instructions for successfully removing a node from
an existing Oracle RAC 10g system, it is by no means a substitute for the official
Oracle documentation. In addition to this article, users should also consult the following
Oracle documents to gain a full understanding of alternative configuration options, installation, and
administration with Oracle RAC 10g. Oracle's official documentation site is
docs.oracle.com.
Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux
Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g Release 2 (10.2)
2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)
Remove the Oracle instance running on the node to be deleted using DBCA!
Overview
Before removing any of the Oracle software from the node to be deleted (linux3),
the first step is to remove the Oracle instance (orcl3) from the RAC database.
This task can be performed using either Enterprise Manager or the
Database Configuration Assistant (DBCA). For the purpose of this article,
I am opting to use the DBCA.
Verifying Terminal Shell Environment
Before starting the DBCA, you should first verify
you are logged onto the server you will be running the DBCA from
(i.e. linux1) then run
the xhost command as root from the console to
allow X Server connections. Next, login as the oracle user account.
If you are using a remote client to connect to the node performing the
installation (SSH / Telnet to linux1 from a workstation configured with
an X Server), you will need to set the DISPLAY variable to point to your
local workstation.
Finally, verify remote access / user equivalence to all nodes in the cluster:
# hostname
linux1
# xhost +
access control disabled, clients can connect from any host
# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY
Verify you are able to run the secure shell
commands (ssh or scp) on the Linux server
you will be running the DBCA from against all other Linux servers
in the cluster without being prompted for a password.
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1
$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2
$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3
Remove Database Instance
To start the database instance removal process, run the
following from linux1:
$ dbca &
Screen Name
Response
Welcome Screen
Select Oracle Real Application Clusters database.
Operations
Select Instance Management.
Instance Management
Select Delete an instance.
List of cluster databases
Provides a list of clustered databases running on the node. For the purpose of this
example, the clustered database running on node linux1 is orcl.
Select this clustered database.
Password: <sys_password>
List of cluster
database instances
This screen provides a list of all instances currently available on the cluster, their status,
and which node they reside on.
Database Services
If the current clustered database has any database services defined, the next
screen allows the DBA to re-configure those database services to account for the
instance being removed. In this example, the existing clustered database has one service
defined named orcl_taf.
With the "orcl_taf" database service selected, change the details to
Not used for the
instance to be removed (orcl3) and leave the "TAF Policy" set to Basic.
Database Configuration
Assistant: Summary
After verifying the instance removal options in the summary dialog,
Click OK to begin the instance management process.
Database Configuration Assistant
A second dialog box is presented which includes the warning:
Database Configuration Assistant
Progress Screen
A progress bar is display while the instance is being deleted. Once the instance management
process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform
another operation?" Click No to end and exit the DBCA utility.
Verify Database Instance and Associated Objects Were Removed
$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE linux1
ora....X1.lsnr application ONLINE ONLINE linux1
ora.linux1.gsd application ONLINE ONLINE linux1
ora.linux1.ons application ONLINE ONLINE linux1
ora.linux1.vip application ONLINE ONLINE linux1
ora....SM2.asm application ONLINE ONLINE linux2
ora....X2.lsnr application ONLINE ONLINE linux2
ora.linux2.gsd application ONLINE ONLINE linux2
ora.linux2.ons application ONLINE ONLINE linux2
ora.linux2.vip application ONLINE ONLINE linux2
ora....SM3.asm application ONLINE ONLINE linux3
ora....X3.lsnr application ONLINE ONLINE linux3
ora.linux3.gsd application ONLINE ONLINE linux3
ora.linux3.ons application ONLINE ONLINE linux3
ora.linux3.vip application ONLINE ONLINE linux3
ora.orcl.db application ONLINE ONLINE linux1
ora....l1.inst application ONLINE ONLINE linux1
ora....l2.inst application ONLINE ONLINE linux2
ora...._taf.cs application ONLINE ONLINE linux2
ora....cl1.srv application ONLINE ONLINE linux1
ora....cl2.srv application ONLINE ONLINE linux2
- or -
$ rac_crs_stat
HA Resource Target State
----------- ------ -----
ora.linux1.ASM1.asm ONLINE ONLINE on linux1
ora.linux1.LISTENER_LINUX1.lsnr ONLINE ONLINE on linux1
ora.linux1.gsd ONLINE ONLINE on linux1
ora.linux1.ons ONLINE ONLINE on linux1
ora.linux1.vip ONLINE ONLINE on linux1
ora.linux2.ASM2.asm ONLINE ONLINE on linux2
ora.linux2.LISTENER_LINUX2.lsnr ONLINE ONLINE on linux2
ora.linux2.gsd ONLINE ONLINE on linux2
ora.linux2.ons ONLINE ONLINE on linux2
ora.linux2.vip ONLINE ONLINE on linux2
ora.linux3.ASM3.asm ONLINE ONLINE on linux3
ora.linux3.LISTENER_LINUX3.lsnr ONLINE ONLINE on linux3
ora.linux3.gsd ONLINE ONLINE on linux3
ora.linux3.ons ONLINE ONLINE on linux3
ora.linux3.vip ONLINE ONLINE on linux3
ora.orcl.db ONLINE ONLINE on linux1
ora.orcl.orcl1.inst ONLINE ONLINE on linux1
ora.orcl.orcl2.inst ONLINE ONLINE on linux2
ora.orcl.orcl_taf.cs ONLINE ONLINE on linux2
ora.orcl.orcl_taf.orcl1.srv ONLINE ONLINE on linux1
ora.orcl.orcl_taf.orcl2.srv ONLINE ONLINE on linux2
Verify Instance we Removed
Login to one of the instances and query the gv$instance view:
SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS')
2 from gv$instance order by inst_id;
INST_ID INSTANCE_NAME STATUS TO_CHAR(STARTUP_TIME
---------- ---------------- ------------ --------------------
1 orcl1 OPEN 27-FEB-2008 01:26:52
2 orcl2 OPEN 27-FEB-2008 01:27:22
Verify Database Files / Tablespaces have been Removed
Check that the UNDO tablespace that was defined for orcl3 (UNDOTBS3)
has been removed:
SQL> select tablespace_name from dba_tablespaces order by 1;
TABLESPACE_NAME
------------------
EXAMPLE
SOE
SOEINDEX
SYSAUX
SYSTEM
TEMP
UNDOTBS1
UNDOTBS2
USERS
Check that the online redo log files that were defined for orcl3
have been removed:
SQL> select inst_id, group#, member from gv$logfile order by 1,2,3;
INST_ID GROUP# MEMBER
------- ------ ---------------------------------------------------------
1 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
1 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
1 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
1 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
1 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
1 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
1 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
1 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299
2 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
2 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
2 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
2 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
2 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
2 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
2 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
2 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299
Update TNSNAMES
Login to all machines that were accessing the instance removed and update
the tnsnames.ora file (if necessary).
Remove ASM instance and ASMLib from the node to be deleted!
Overview
When the clustered database is running on ASM
(as is the case in this example), the steps in this section
describe how to remove the ASM instance (and ASMLib) from the node to be deleted.
If the Oracle RAC database is not running on ASM, this section can be
skipped.
Remove the ASM Instance from the Oracle Cluster Registry File
Run the following from linux1 as the oracle user account:
$ srvctl stop asm -n linux3
$ srvctl remove asm -n linux3
Remove ASM Directories from the Node to be Deleted
Remove all ASM directories from the node to be deleted (linux3)
as the oracle user account:
$ cd $ORACLE_BASE/admin
$ rm -rf +ASM
$ cd $ORACLE_HOME/dbs
$ rm -f *ASM*
You may also have to edit the /etc/oratab file on the node to be deleted
and remove entries beginning with +ASM to delete the references of the
ASM instance.
Remove ASMLib from the Node to be Deleted
The Linux nodes in the current Oracle RAC database make use of ASMLib so that
the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls.
# /etc/init.d/oracleasm stop
Unmounting ASMlib driver filesystem: [ OK ]
Unloading module "oracleasm": [ OK ]
# rpm -qa | grep oracleasm
oracleasm-support-2.0.3-1
oracleasm-2.6.9-55.EL-2.0.3-1
oracleasmlib-2.0.2-1
# rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/rc.d/init.d/oracleasm
# rm -f /etc/rc0.d/*oracleasm*
# rm -f /etc/rc1.d/*oracleasm*
# rm -f /etc/rc2.d/*oracleasm*
# rm -f /etc/rc3.d/*oracleasm*
# rm -f /etc/rc4.d/*oracleasm*
# rm -f /etc/rc5.d/*oracleasm*
# rm -f /etc/rc6.d/*oracleasm*
Perform the following configuration procedures from
only one of the Oracle RAC nodes in the cluster (linux1)!
The Network Configuration Assistant (NETCA) will remove the
node-specific listener from the clustered configuration for
the node to be deleted.
Overview
Each of the nodes in the current Oracle RAC environment use a node-specific listener named
LISTENER_nodename which runs out of the local Oracle home.
Verifying Terminal Shell Environment
As was performed earlier, the terminal shell environment needs to
be configured for remote access and user equivalence to all Oracle RAC nodes
before running the NETCA. Note
that you can utilize the same terminal shell session used in the previous section
which in this case, you do not have to perform any of the actions described below
with regards to setting up remote
access and the DISPLAY variable:
# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY
Verify you are able to run the secure shell
commands (ssh or scp) on the Linux server
you will be running the NETCA from against all other Linux servers
in the cluster without being prompted for a password.
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1
$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2
$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3
Run the Network Configuration Assistant
To start the NETCA, run the following from linux1:
$ netca &
Screen Name
Response
Select the Type of Oracle
Net Services Configuration
Select Cluster configuration
Select the nodes to configure
Only select the Oracle RAC node to remove the listener from: linux3.
Type of Configuration
Select Listener configuration.
Listener Configuration
Next 4 Screens
The following screens are now like any other normal listener configuration
when removing a listener:
What do you want to do: Delete
Select a listener you want to delete: LISTENER
Acknowledge warning dialog: Yes
Listener deleted [ Next ]
Listener configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration
Click Finish to exit the NETCA.
Verify TNS Listener Configuration
The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC
cluster (linux1 and linux2):
$ hostname
linux1
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX1
$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr
NAME=ora.linux1.LISTENER_LINUX1.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux1
=====================
$ hostname
linux2
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX2
$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr
NAME=ora.linux2.LISTENER_LINUX2.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux2
=====================
$ hostname
linux3
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
$ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
Remove the Oracle Database software and update the Node List
for the node to be deleted!
Overview
In this section, you will be removing the node to be deleted (linux3) from
the Node List for its Oracle inventory. Next will be to remove the
Oracle Database software from the node to be deleted using the
Oracle Universal Installer (OUI). Finally, update the Oracle inventory on all remaining
Oracle RAC nodes to reflect the nodes which will remain in the cluster.
Update Node List for Oracle Database Software - (Remove linux3)
From the node to be deleted (linux3), run the OUI
as the oracle user account to update the inventory node list
for the Oracle Database software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES="" -local
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Although the OUI does not launch an installer GUI, the DISPLAY
environment variable still needs to be set!
De-install Oracle Database Software
Next, run the OUI from the node to be deleted (linux3) to
de-install the Oracle Database software. Make certain that you choose
the home to be removed and not just the products under that home.
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller
Screen Name
Response
Welcome Screen
Click the Installed Products button.
Inventory: Contents Tab
Check the Oracle home to deleted (OraDb10g_home1)
and click the Remove button.
Confirmation
Acknowledge the warning dialog by clicking Yes.
Deinstallation Process
A progress bar is displayed while the Oracle Database software is being removed.
Once this process has completed, you are returned to the "Inventory: Contents Tab"
dialog. After confirming the Oracle Database software (Oracle home) was successfully removed,
Click Close to exit this dialog.
Welcome Screen
Click Cancel to exit the OUI.
Update Node List for Remaining Nodes in the Cluster
Finally, from linux1 logged in as the oracle user account
(and user equivalence enabled), update the Oracle Clusterware inventory
node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={linux1,linux2}"
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Although the OUI does not launch an installer GUI, the DISPLAY
environment variable still needs to be set!
Remove the node to be deleted from Oracle Clusterware
and update the Node List for all remaining nodes!
Overview
After successfully removing the Oracle Database software from the node to be deleted,
the next major step is to remove the node to be deleted from Oracle Clusterware.
Remove Node-Specific Interface Configuration
Run the following commands to remove node-specific interface configurations from the
node to be deleted. For this example, run these commands from linux3
as the oracle user account:
$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200
In the above command, replace 6200 with the
port number you get in the remoteport section of the ons.config
file found in the $ORA_CRS_HOME/opmn/conf directory:
$ grep '^remoteport' $ORA_CRS_HOME/opmn/conf/ons.config
remoteport=6200
$ $ORA_CRS_HOME/bin/oifcfg delif -node linux3
PROC-4: The cluster registry key to be operated on does not exist.
PRIF-11: cluster registry error
From the research that I have performed, I believe the above error
can be safely ignored. The "oifcfg delif" command attempts to
delete a node-specific interface defined for node linux3
in the Oracle Cluster Registry file (OCR). This means it is looking for
the presence of a key in the OCR file named "SYSTEM.css.interfaces.nodelinux3".
If you were to dump the contents of the OCR file, you would quickly see that
no key is present with that name:
$ $ORA_CRS_HOME/bin/ocrdump
$ grep 'SYSTEM.css.interfaces.nodelinux3' OCRDUMPFILE
After examining the contents of the OCRDUMPFILE, interfaces are
indeed defined, however they are identified as global:
$ grep 'SYSTEM.css.interfaces.global' OCRDUMPFILE
[SYSTEM.css.interfaces.global]
[SYSTEM.css.interfaces.global.eth0]
[SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0]
[SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0.1]
[SYSTEM.css.interfaces.global.eth2]
[SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0]
[SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0.1]
Given there are no node-specific interface entries defined and we
obviously don't want to delete the interface keys configured globally,
the oifcfg delif command and its errors can be safely ignored!
Disable Oracle Clusterware Applications
From the node you are deleting from the cluster (linux3), run the script
$ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle
Clusterware applications that are on the node. This script should
only be run once. Given the Clusterware software install is on local
disk (non-shared), make certain to use the nosharedhome
argument. The default for this script is sharedhome which
prevents you from updating the permissions of local files such that
they can be removed by the oracle user account.
$ su
# cd $ORA_CRS_HOME/install
# ./rootdelete.sh local nosharedvar nosharedhome
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
Delete Node from Cluster and Update OCR
Upon successful completion of the rootdelete.sh script, run
the rootdeletenode.sh script to delete the node (linux3)
from the Oracle cluster and to update the Oracle Cluster Registry (OCR).
This script should be run from a pre-existing / available node
in the cluster (linux1) as the root user account:
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
linux3 3
$ su
# cd $ORA_CRS_HOME/install
# ./rootdeletenode.sh linux3,3
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
CRS-0210: Could not find resource 'ora.linux3.ons'.
CRS-0210: Could not find resource 'ora.linux3.vip'.
CRS-0210: Could not find resource 'ora.linux3.gsd'.
CRS-0210: Could not find resource ora.linux3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'linux3,3' deleted successfully
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
Update Node List for Oracle Clusterware Software - (Remove linux3)
From the node to be deleted (linux3), run the OUI
as the oracle user account to update the inventory node list
for the Oracle Clusterware software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES="" -local CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Although the OUI does not launch an installer GUI, the DISPLAY
environment variable still needs to be set!
De-install Oracle Clusterware Software
Next, run the OUI from the node to be deleted (linux3) to
de-install the Oracle Clusterware software. Make certain that you choose
the home to be removed and not just the products under that home.
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller
Screen Name
Response
Welcome Screen
Click the Installed Products button.
Inventory: Contents Tab
Check the Oracle home to deleted (OraCrs10g_home)
and click the Remove button.
Confirmation
Acknowledge the warning dialog by clicking Yes
to remove the Oracle Clusterware software and to remove the /u01/app/crs
directory.
Deinstallation Process
A progress bar is displayed while the Oracle Clusterware software is being removed.
Once this process has completed, you are returned to the "Inventory: Contents Tab"
dialog. After confirming the Oracle Clusterware software (Clusterware home) was successfully removed,
Click Close to exit this dialog.
Welcome Screen
Click Cancel to exit the OUI.
Update Node List for Remaining Nodes in the Cluster
Finally, from linux1 logged in as the oracle user account
(and user equivalence enabled), update the Oracle Clusterware inventory
node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={linux1,linux2}" CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Although the OUI does not launch an installer GUI, the DISPLAY
environment variable still needs to be set!
Verify Node to be Deleted is no Longer a Member of the Cluster
Run the following commands to verify that the node to be deleted from the Oracle RAC
cluster is no longer a member of the cluster and to verify that the Oracle Clusterware
components have been successfully removed from that node.
$ srvctl status nodeapps -n linux3
PRKC-1056 : Failed to get the hostname for node linux3
PRKH-1001 : HASContext Internal Error
[OCR Error(Native: getHostName:[21])]
The error above indicates that linux3 is no longer a member
of the cluster.
$ $ORA_CRS_HOME/bin/crs_stat | grep -i linux3
You should not see any output from the above command
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
You should see the present node list without the deleted node
(that is linux1 and linux2 only).
Remove/Rename any Remaining Oracle Files from Node to be Deleted
From the node to be deleted (linux3), remove/rename any remaining
Oracle files while logged in as the root user account:
# mv -f /etc/inittab.no_crs /etc/inittab
# rm -f /etc/inittab.orig
# rm -f /etc/inittab.crs
# rm -rf /etc/oracle
# rm -f /etc/oratab
# rm -f /etc/oraInst.loc
# rm -rf /etc/ORCLcluster
# rm -rf /u01/app/oracle
# rm -rf /u01/app/crs
# rm -f /usr/local/bin/coraenv
# rm -f /usr/local/bin/dbhome
# rm -f /usr/local/bin/oraenv
Finally, remove the Oracle user account and all associated UNIX groups from linux3:
# userdel -r oracle
# groupdel oinstall
# groupdel dba
Remove OCFS2 from the node to be deleted!
Overview
The existing Oracle RAC 10
Remove OCFS2 from the Node to be Deleted
In this section we will remove the OCFS2 kernel driver, supporting software, and
associated directories. Run the following from the node to be deleted
(linux3) as the root user account:
# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# rpm -qa | grep ocfs2
ocfs2-tools-1.2.4-1
ocfs2-2.6.9-55.EL-1.2.5-6
ocfs2console-1.2.4-1
# rpm -ev ocfs2-tools-1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1
warning: /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave
# rm -f /etc/sysconfig/o2cb.rpmsave
# rm -rf /etc/ocfs2
Remove Node to be Deleted from the OCFS2 Cluster
The next step is to remove the Oracle RAC node being deleted (linux3) from the "live"
OCFS2 cluster. This entails running the o2cb_ctl command-line
utility from the remaining two RAC nodes linux1 and linux2.
[root@linux1 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported
[root@linux2 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported
-D : Delete an object from the existing OCFS2 Cluster Configuration.
-u : Valid only with -D. When deleting something (node or cluster),
it will also remove it from the live cluster (/config). If the
parameter is not specified, then only update the
/etc/ocfs2/cluster.conf.
-n : Object name which is usually the node name or cluster name.
At the time of this writing, the latest version of OCFS2
(namely the o2cb cluster stack) only supports "adding"
nodes dynamically. Deleting a node requires the OCFS2 cluster to be down
which means a database outage.
This is a serious shortcoming with OCFS2 that hasn't been addressed
for some time. Currently, an enhancement request
(oss.oracle.com Bugzilla Bug 694)
has been submitted to Oracle, however no target date or version has been set for this
enhancement.
node:
ip_port = 7777
ip_address = 192.168.2.107
number = 2
name = linux3
cluster = ocfs2
Before closing out the /etc/ocfs2/cluster.conf file,
change the value of node_count
to 2 on both Oracle RAC nodes:
node_count = 2
# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb load
# /etc/init.d/o2cb online ocfs2
# mount /u02
After mounting the OCFS2 file system on both Oracle RAC nodes,
verify that linux3 does not exist in the live OCFS2
cluster - only linux1 and linux2 should be
present:
[root@linux1 node]# ls -l /config/cluster/ocfs2/node
total 0
drwxr-xr-x 2 root root 0 Mar 1 01:32 linux1
drwxr-xr-x 2 root root 0 Mar 1 01:32 linux2
[root@linux2 ~]# ls -l /config/cluster/ocfs2/node
total 0
drwxr-xr-x 2 root root 0 Mar 1 01:33 linux1
drwxr-xr-x 2 root root 0 Mar 1 01:33 linux2
Remove OCFS2 Mount Options from /etc/fstab
Finally, remove the following entry from the /etc/fstab from
linux3:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0
Remove network access permissions in Openfiler for
linux3 and disable auto-discovery of iSCSI volumes
from linux3!
Overview
At this point, linux3 has been successfully removed from the
Oracle RAC configuration. Remember that in the existing Oracle RAC configuration
that all shared disk storage is based on iSCSI using a Network Storage Server;
namely Openfiler Release 2.2 (respin 2). When we added linux3
to the current Oracle RAC configuration in the article
"Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)",
we needed to configure network access in Openfiler so that linux3
had permissions to the shared iSCSI volumes. We also configured the
iSCSI initiator on linux3 to automatically discover all available
iSCSI volumes on each boot.
https://openfiler1:446/
From the Openfiler Storage Control Center home page, login as
an administrator. The default administration login credentials
for Openfiler are:
Revoke Access from linux3 to each iSCSI Volume
The first step in this process is to remove access from linux3 to each individual
iSCSI volume within Openfiler.
iSCSI / Logical Volumes in Volume Group rac1
Volume Name
Volume Description
Required Space (MB)
Filesystem Type
crs
Oracle Clusterware
2,048
iSCSI
asm1
Oracle ASM Volume 1
118,720
iSCSI
asm2
Oracle ASM Volume 2
118,720
iSCSI
asm3
Oracle ASM Volume 3
118,720
iSCSI
asm4
Oracle ASM Volume 4
118,720
iSCSI
Figure 2: Current Logical (iSCSI) Volumes
Figure 3: Deny Host Access to Logical (iSCSI) Volumes
Remove linux3 from Openfiler Local Network Configuration
The next step is to modify the local network configuration within Openfiler
by removing the node linux3 (which was actually entered as node linux3-priv).
Figure 4: Configure Openfiler Host Access for deleted Oracle RAC Node
Disable iSCSI Initiator Service on linux3
The final step in this process is to modify the iSCSI initiator service on linux3 so it
will not automatically start and therefore will not attempt to discover iSCSI
volumes from the Openfiler server.
# service iscsi stop
Searching for iscsi-based multipath maps
Found 0 maps
Stopping iscsid: [ OK ]
Removing iscsi driver: [ OK ]
Next, edit the file /etc/iscsi.conf on linux3 and
comment out (or remove) the DiscoveryAddress entry which specifies
the hostname of the Openfiler network storage server.
In our example, use a # at the beginning of DiscoveryAddress entry
as shown below:
...
# DiscoveryAddress=openfiler1-priv
...
Finally, configure the iSCSI service to not be active across machine reboots on linux3.
The Linux command chkconfig can be used to achieve that as follows:
# chkconfig --level 345 iscsi off
Jeffrey Hunter graduated from Stanislaus State University in Turlock,
California, with a Bachelor's degree in Computer Science. Jeff is an
Oracle Certified Professional, Java Development Certified Professional,
Author, and currently works as a Senior Database Administrator for
The DBA Zone, Inc..
His work includes advanced performance tuning, Java and PL/SQL programming, capacity
planning, database security, and physical / logical database design in a UNIX,
Linux, and Windows server environment. Jeff's other interests include mathematical
encryption theory, programming language processors (compilers and interpreters)
in Java and C, LDAP, writing web-based database administration tools, and of
course Linux. Jeff has been a Sr. Database Administrator and Software Engineer
for over 13 years and maintains his own website site at:
http://www.iDevelopment.info.
Tuesday, 04-Mar-2008 17:31:28 EST
Page Count: 3041