Oracle DBA, a practical approach !!: September 2012

Adding a node to an existing Oracle RAC cluster on 11.2.0.2 Grid Infrastructure

Environment:

SUSE Linux Enterprise Server 10 (x86_64) ; VERSION = 10 ; PATCHLEVEL = 2

Oracle Grid Infrastructure 11.2.0.2

Oracle Database Server 11.2.0.2

Enterprise Manager Agent 10.2.0.3

Existing cluster has 7 nodes (racprd01, racprd02, racprd03, racprd04, racprd05, racprd08, racprd09); Task is to add an 8th node to the cluster which will be called racprd10

I will be using one of the existing nodes(racprd05) as reference node and the new node will be racprd10

Reference Oracle Documentation:

Oracle® Clusterware Administration and Deployment Guide - 11g Release 2 (11.2)

http://docs.oracle.com/cd/E11882_01/rac.112/e16794/adddelclusterware.htm#CHDJFEEA

Prerequisite Steps for Adding Cluster Nodes:

Make physical connections.

Connect the nodes' hardware to the network infrastructure of your cluster. This includes establishing electrical connections, configuring network interconnects, configuring shared disk subsystem connections, and so on.

This was done by the system engineers along with datacenter support team.

Install the operating system.

Install a cloned image of the operating system that matches the operating system on the other nodes in your cluster. This includes installing required service patches, updates, and drivers.

I’m going to use one of the nodes (racprd05) as the reference node or peer node for the new racprd10 server. Hence, I requested the system engineers to use racprd05 as the source and clone new node racprd10 from the source server. The reason I chose racprd05 as reference node is because the server hardware is the same model that I’m going to add.

Also, we use nfs cluster file system. So all existing shared volumes mounted on the existing nodes in the cluster were mounted on the new server with the same mount options. This includes OCR, voting disks, shared storage for oradata, redo, temp, archive volumes, etc.

The binaries stay on the local disk of each server. So apart from the software binaries, everything else is shared and mounted on all the servers.

Create Oracle users.

Since the new server was cloned from an existing server in the cluster, the ‘oracle’ user and the required groups came over from the source server. The UID’s and GID’s were the same which is a requirement.

Ensure that SSH is configured on the node.

Use steps from the below link to manually add SSH configuration on the new node. This includes generating the rsa/dsa keys, adding them to the authorized_keys file (all keys from all servers to a common authorized_keys file) and copying them over to all the servers. Enabling passwordless SSH between the servers.

http://docs.oracle.com/cd/E11882_01/install.112/e22489/manpreins.htm#BABJBAEB

The above link has steps for manually setting up SSH for all nodes in the cluster (a new RAC setup). We only want to do the steps required to enable the new node into the SSH configuration.

At a high level,

Copy authorized_keys file from existing node to new node

Generate rsa/dsa keys on the new node

Add the newly generated keys on the new node to the authorized keys file obtained from the existing node. So, at this point, the new node’s authorized_keys file will have keys from all the existing nodes and the new node too.

Copy this authorized_keys file from the new node to all the existing nodes. So, at this point, all nodes authorized_keys file will have keys from all other nodes participating in the cluster.

Enabling SSH User Equivalency on Cluster Nodes or password less SSH between new node to all the other nodes. And, from all the nodes to the new node. At this point, you can SSH without needing to use password between any/all nodes in the cluster

Verify the hardware and operating system installations with the Cluster Verification Utility (CVU).

Run the following commands to verify that the nodes you want to add are reachable by other nodes in the cluster. This command is used to verify user equivalence to all given nodes from the local node, node connectivity among all of the given nodes, accessibility to shared storage from all of the given nodes, and so on.

From GI_HOME/bin on an existing node, run

$ cluvfy stage -post hwos -n node_list | all [-verbose]

Using keyboard-interactive authentication.

Password:

Last login: Thu Jul 19 15:11:32 2012 from racprd10.imycompany.com

racprd05 | ORA1020 | /export/oracle

> . oraenv

ORACLE_SID = [ORA1020] ? CRS

The Oracle base remains unchanged with value /export/oracle

racprd05 | CRS | /export/oracle

> export PATH=$ORACLE_HOME/bin:$PATH

racprd05 | CRS | /export/oracle

> which cluvfy

/export/11.2.0.2/bin/cluvfy

racprd05 | CRS | /export/oracle

> cluvfy stage -post hwos -n racprd10 -verbose

Performing post-checks for hardware and operating system setup

Checking node reachability...

Check: Node reachability from node "racprd05"

Destination Node Reachable?

------------------------------------ ------------------------

racprd10 yes

Result: Node reachability check passed from node "racprd05"

Checking user equivalence...

Check: User equivalence for user "oracle"

Node Name Comment

------------------------------------ ------------------------

racprd10 passed

Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Node Name Status Comment

------------ ------------------------ ------------------------

racprd10 passed

Verification of the hosts config file successful

Interface information for node "racprd10"

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

------ --------------- --------------- --------------- --------------- ----------------- ------

bond0 10.120.20.99 10.120.20.0 0.0.0.0 10.120.20.1 00:5B:21:42:6D:F0 1500

bond1 192.168.20.103 192.168.20.0 0.0.0.0 10.120.20.1 00:5B:21:42:6C:84 1500

bond2 10.120.20.109 10.120.20.0 0.0.0.0 10.120.20.1 00:5B:21:42:6C:81 1500

Check: Node connectivity of subnet "10.120.20.0"

Source Destination Connected?

------------------------------ ------------------------------ ----------------

racprd10[10.120.20.99] racprd10[10.120.20.109] yes

Result: Node connectivity passed for subnet "10.120.20.0" with node(s) racprd10

Check: TCP connectivity of subnet "10.120.20.0"

Source Destination Connected?

------------------------------ ------------------------------ ----------------

racprd05:10.120.20.146 racprd10:10.120.20.99 passed

racprd05:10.120.20.146 racprd10:10.120.20.109 passed

Result: TCP connectivity check passed for subnet "10.120.20.0"

Check: Node connectivity of subnet "192.168.20.0"

Result: Node connectivity passed for subnet "192.168.20.0" with node(s) racprd10

Check: TCP connectivity of subnet "192.168.20.0"

Source Destination Connected?

------------------------------ ------------------------------ ----------------

racprd05:10.120.20.146 racprd10:192.168.20.103 passed

Result: TCP connectivity check passed for subnet "192.168.20.0"

Interfaces found on subnet "10.120.20.0" that are likely candidates for VIP are:

racprd10 bond0:10.120.20.99

Interfaces found on subnet "10.120.20.0" that are likely candidates for VIP are:

racprd10 bond2:10.120.20.109

Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are:

racprd10 bond1:192.168.20.103

Result: Node connectivity check passed

Checking for multiple users with UID value 0

Result: Check for multiple users with UID value 0 passed

Check: Time zone consistency

Result: Time zone consistency check passed

Checking shared storage accessibility...

Disk Sharing Nodes (1 in count)

------------------------------------ ------------------------

/dev/sda racprd10

Disk Sharing Nodes (1 in count)

------------------------------------ ------------------------

/dev/sdb racprd10

Shared storage check was successful on nodes "racprd10"

Post-check for hardware and operating system setup was successful.

racprd05 | CRS | /export/oracle

Next is Peer check.

From GI_HOME/bin on an existing node, run the CVU command to obtain a detailed comparison of the properties of the reference node with all of the other nodes that are part of your current cluster environment.

$ cluvfy comp peer [-refnode ref_node] -n node_list

[-orainv orainventory_group] [-osdba osdba_group] [-verbose]

> cluvfy comp peer -refnode racprd05 -n racprd10 -orainv dba -osdba dba -verbose

Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: racprd05]