Sunday, April 16, 2017

Add Node to RAC cluster 12cR1 Manually Silent Mode



Adding a node to a 12cR1 RAC cluster Manually


This post is for how to add node to 12cR1 cluster manually in CLI mode.

Here I have used Oracle Linux 6.5 with UEK2 server as a host. Below are my existing 2 node configuration details.

rac121:ccbdb1>. oraenv
ORACLE_SID = [ccbdb1] ? +ASM1
The Oracle base has been changed from /u01/app/oracle to /grid/app/oracle

rac121:+ASM1>oifcfg getif
eth0  192.168.1.0  global  public
eth1  10.10.10.0  global  cluster_interconnect
eth2  20.20.20.0  global  cluster_interconnect
eth3  30.30.30.0  global  cluster_interconnect


Here I am using oracle user for both Grid and RDBMS Binary. Also make sure the new host will be identical to  configuration with existing host.

Adding the node

High level steps need to follow in order to adding node to the cluster are:

·         Install Operating System OEL 6.5 x64 UEK2
·         Install required rpm as per existing 2 node host
·         Configure network (Public and Private)
·         Configure kernel parameters, storage (asmlib) and DNS
·         First Adding Grid Binary then after RDBMS Binary and register to OCR

Configure yum and install required oracle related package using oracle-rdbms-server-12cR1-preinstall package.

Also validate the below configuration as per existing 2 nodes:

·         configure necessary required rules in firewall to allow access or disable the firewall as your configuration.
·         Change the setting in /etc/sysconfig/selinux to permissive
·         Here we are using CTS not NTP, so make sure need to remove or rename the /etc/ntp.conf and disable the service.
·         Mount all share storage drive in new host with required permission.
·         Create the directory structure as per existing node configuration.
·         Configure user equivalence between all nodes.


rac121:ccbdb1> cd /ora_global_nfs/software/Oracle/database/DB_12.1.0.2/grid/sshsetup

rac121:ccbdb1>./sshUserSetup.sh -user oracle -hosts "rac121 rac123" -noPromptPassphrase
The output of this script is also logged into /tmp/sshUserSetup_2017-04-06-14-01-20.log
Hosts are rac121 rac123
user is oracle
Platform:- Linux
Checking if the remote hosts are reachable
PING rac121.ora.com (192.168.1.121) 56(84) bytes of data.
64 bytes from rac121.ora.com (192.168.1.121): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from rac121.ora.com (192.168.1.121): icmp_seq=2 ttl=64 time=0.024 ms
....
....
------------------------------------------------------------------------
--rac121:--
Running /usr/bin/ssh -x -l oracle rac121 date to verify SSH connectivity has been setup from local host to rac121.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Thu Apr  6 14:01:39 IST 2017
------------------------------------------------------------------------
--rac123:--
Running /usr/bin/ssh -x -l oracle rac123 date to verify SSH connectivity has been setup from local host to rac123.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Thu Apr  6 14:01:39 IST 2017
------------------------------------------------------------------------
SSH verification complete.
rac121:ccbdb1>

Run the cluster verification utility to check the node configuration between nodes before adding nodes.

rac121:ccbdb1>cluvfy stage -pre nodeadd -n rac123 -fixup -fixupnoexec

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "rac121"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking CRS integrity...

CRS integrity check passed

....
....

Starting check for Reverse path filter setting ...

Check for Reverse path filter setting passed

Starting check for Network interface bonding status of private interconnect network interfaces ...

Check for Network interface bonding status of private interconnect network interfaces passed

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was successful.



Since we have successfully validated the cluvfy, now it's high time to add our 3rd node to our existing 2 node cluster.

rac121:ccbdb1>./addnode.sh -silent "CLUSTER_NEW_NODES={rac123}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac123-vip}"

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 33259 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 7997 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /grid/app/oraInventory/logs/addNodeActions2017-04-06_02-41-43PM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /grid/app/12.1.0.1/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /grid/app/oraInventory/orainstRoot.sh
        2. /grid/app/12.1.0.1/grid/root.sh

Execute /grid/app/oraInventory/orainstRoot.sh on the following nodes:
[rac123]
Execute /grid/app/12.1.0.1/grid/root.sh on the following nodes:
[rac123]

The scripts can be executed in parallel on all the nodes.

..........
Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.

Execute the orainstRoot.sh and root.sh at node 3 as privilege user.

rac123:ccbdb3>sudo /grid/app/oraInventory/orainstRoot.sh

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for oracle:
Changing permissions of /grid/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /grid/app/oraInventory to oinstall.
The execution of the script is complete.


rac123:ccbdb3>sudo /grid/app/12.1.0.1/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/12.1.0.1/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /grid/app/12.1.0.1/grid/crs/install/crsconfig_params
2017/04/06 14:56:07 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2017/04/06 14:56:42 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

OLR initialization - successful
2017/04/06 14:57:38 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac123'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac123'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac123' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac123' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac123'
CRS-2672: Attempting to start 'ora.evmd' on 'rac123'
CRS-2676: Start of 'ora.mdnsd' on 'rac123' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac123'
CRS-2676: Start of 'ora.gpnpd' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac123'
CRS-2676: Start of 'ora.gipcd' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac123'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac123'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac123'
CRS-2676: Start of 'ora.diskmon' on 'rac123' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac123'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac123'
CRS-2676: Start of 'ora.ctssd' on 'rac123' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac123'
CRS-2676: Start of 'ora.asm' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac123'
CRS-2676: Start of 'ora.storage' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac123'
CRS-2676: Start of 'ora.crf' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac123'
CRS-2676: Start of 'ora.crsd' on 'rac123' succeeded
CRS-6017: Processing resource auto-start for servers: rac123
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac121'
CRS-2672: Attempting to start 'ora.net1.network' on 'rac123'
CRS-2672: Attempting to start 'ora.FLASH.dg' on 'rac123'
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac123'
CRS-2676: Start of 'ora.net1.network' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rac123'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac121'
CRS-2676: Start of 'ora.DATA.dg' on 'rac123' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac121' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac123'
CRS-2676: Start of 'ora.FLASH.dg' on 'rac123' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac123'
CRS-2676: Start of 'ora.ons' on 'rac123' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac123' succeeded
CRS-6016: Resource auto-start has completed for server rac123
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/04/06 15:01:18 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
2017/04/06 15:01:46 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Seems we have added node to our cluster successfully.  Need to validate the cluster.

rac123:+ASM3>crsctl check cluster -all
******************************************************
rac121:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************
rac122:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************
rac123:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************

rac123:+ASM3>crsctl stat res -t
------------------------------------------------------------
Name           Target  State    Server    State details
------------------------------------------------------------
Local Resources
------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
ora.DG_OCR1.dg
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
ora.FLASH.dg
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
ora.asm
               ONLINE  ONLINE   rac121    Started,STABLE
               ONLINE  ONLINE   rac122    Started,STABLE
               ONLINE  ONLINE   rac123    Started,STABLE
ora.net1.network
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
ora.ons
               ONLINE  ONLINE   rac121    STABLE
               ONLINE  ONLINE   rac122    STABLE
               ONLINE  ONLINE   rac123    STABLE
------------------------------------------------------------
Cluster Resources
------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE   rac122    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE   rac123    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE   rac121    STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE   rac121    169.254.26.48 10.10.
                                          10.121 20.20.20.121,
                                          STABLE
ora.ccbdb.db
      1        ONLINE  ONLINE   rac121    Open,STABLE
      2        ONLINE  ONLINE   rac122    Open,STABLE
ora.cvu
      1        ONLINE  ONLINE   rac121    STABLE
ora.mgmtdb
      1        ONLINE  ONLINE   rac121    Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE   rac121    STABLE
ora.rac121.vip
      1        ONLINE  ONLINE   rac121    STABLE
ora.rac122.vip
      1        ONLINE  ONLINE   rac122    STABLE
ora.rac123.vip
      1        ONLINE  ONLINE   rac123    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE   rac122    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE   rac123    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE   rac121    STABLE
------------------------------------------------------------


Adding RDBMS nodes in silent mode manually

Now adding the RDBMS home to the server.

rac121:ccbdb1>./addnode.sh -silent "CLUSTER_NEW_NODES={rac123}"
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 32896 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 7992 MB    Passed

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   8% Done.
You can find the log of this install session at:
 /grid/app/oraInventory/logs/addNodeActions2017-04-07_12-10-53AM.log

Instantiate files in progress.

Instantiate files successful.
..................................................   14% Done.

Copying files to node in progress.

Copying files to node successful.
..................................................   73% Done.

Saving cluster inventory in progress.
..................................................   80% Done.

Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.1/DB_1 was successful.
Please check '/tmp/silentInstall.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   88% Done.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0.1/DB_1/root.sh

Execute /u01/app/oracle/product/12.1.0.1/DB_1/root.sh on the following nodes:
[rac123]


..........
Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.

Execute the root.sh as privileges user on node3.

rac123:ccbdb3>sudo /u01/app/oracle/product/12.1.0.1/DB_1/root.sh
Check /u01/app/oracle/product/12.1.0.1/DB_1/install/root_rac123.ora.com_2017-04-07_01-02-18.log for the output of root script


Since GRID/RDBMS has been added to cluster, now time to add the instance in OCR.

rac123:+ASM3>srvctl config database -d ccbdb
Database unique name: ccbdb
Database name: ccbdb
Oracle home: /u01/app/oracle/product/12.1.0.1/DB_1
Oracle user: oracle
Spfile: +DATA/CCBDB/PARAMETERFILE/spfile.276.919855143
Password file: +DATA/CCBDB/PASSWORD/pwdccbdb.256.919854693
Domain: ora.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: ccbdb1,ccbdb2
Configured nodes: rac121,rac122
Database is administrator managed

Create ifile (pfile) on node3 which point the spfile in diskgroup.

Add the below parameter in SPFILE.

ccbdb3.instance_number=3
ccbdb3.thread=3
ccbdb3.undo_tablespace='UNDOTBS3'

Create Undo tablespace  and redo logs for node 3 .


SQL> create undo tablespace UNDOTBS3  datafile size 1g autoextend on  ;

Tablespace created.

alter database add logfile thread 3 ;
alter database add logfile thread 3 ;


Enable Public Thread for node 3.


SQL> alter database enable public thread 3 ;


Add instance to cluster.

rac123:+ASM3>srvctl add instance -d ccbdb -n rac123 -i ccbdb3

rac123:+ASM3>srvctl config database -d ccbdb
Database unique name: ccbdb
Database name: ccbdb
Oracle home: /u01/app/oracle/product/12.1.0.1/DB_1
Oracle user: oracle
Spfile: +DATA/CCBDB/PARAMETERFILE/spfile.276.919855143
Password file: +DATA/CCBDB/PASSWORD/pwdccbdb.256.919854693
Domain: ora.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: ccbdb1,ccbdb2,ccbdb3
Configured nodes: rac121,rac122,rac123
Database is administrator managed


rac121:ccbdb1>srvctl start database -d ccbdb


Validate the database

rac121:ccbdb1>srvctl status database -d ccbdb

Instance ccbdb1 is running on node rac121 
Instance ccbdb2 is running on node rac122 
Instance ccbdb3 is running on node rac123


SQL> select distinct name, instance_name,host_name, open_mode from gv$database a, gv$instance b ;

NAME      INSTANCE_NAME    HOST_NAME       OPEN_MODE
--------- ---------------- --------------- --------------------
CCBDB     ccbdb3           rac123.ora.com  READ WRITE
CCBDB     ccbdb1           rac121.ora.com  READ WRITE
CCBDB     ccbdb2           rac122.ora.com  READ WRITE



Hope this help...:)

1 comment: