Adding a node to a 12cR1 RAC cluster
Manually
This post is for how to add node to 12cR1 cluster
manually in CLI mode.
Here I have used Oracle Linux 6.5 with UEK2
server as a host. Below are my existing 2 node configuration details.
rac121:ccbdb1>.
oraenv
ORACLE_SID =
[ccbdb1] ? +ASM1
The Oracle base has
been changed from /u01/app/oracle to /grid/app/oracle
rac121:+ASM1>oifcfg
getif
eth0 192.168.1.0
global public
eth1 10.10.10.0
global cluster_interconnect
eth2 20.20.20.0
global cluster_interconnect
eth3 30.30.30.0
global cluster_interconnect
Here I am using oracle user for both Grid
and RDBMS Binary. Also make sure the new host will be identical to configuration with existing host.
Adding the node
High level steps need to follow in order to
adding node to the cluster are:
·
Install Operating System OEL 6.5 x64 UEK2
·
Install required rpm as per existing 2 node
host
·
Configure network (Public and Private)
·
Configure kernel parameters, storage (asmlib)
and DNS
·
First Adding Grid Binary then after RDBMS
Binary and register to OCR
Configure yum and install required oracle
related package using oracle-rdbms-server-12cR1-preinstall package.
Also validate the below configuration as
per existing 2 nodes:
·
configure necessary required rules in
firewall to allow access or disable the firewall as your configuration.
·
Change the setting in /etc/sysconfig/selinux
to permissive
·
Here we are using CTS not NTP, so make sure
need to remove or rename the /etc/ntp.conf and disable the service.
·
Mount all share storage drive in new host
with required permission.
·
Create the directory structure as per
existing node configuration.
·
Configure user equivalence between all nodes.
rac121:ccbdb1> cd
/ora_global_nfs/software/Oracle/database/DB_12.1.0.2/grid/sshsetup
rac121:ccbdb1>./sshUserSetup.sh
-user oracle -hosts "rac121 rac123" -noPromptPassphrase
The output of this
script is also logged into /tmp/sshUserSetup_2017-04-06-14-01-20.log
Hosts are rac121
rac123
user is oracle
Platform:- Linux
Checking if the
remote hosts are reachable
PING rac121.ora.com
(192.168.1.121) 56(84) bytes of data.
64 bytes from
rac121.ora.com (192.168.1.121): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from rac121.ora.com
(192.168.1.121): icmp_seq=2 ttl=64 time=0.024 ms
....
....
------------------------------------------------------------------------
--rac121:--
Running
/usr/bin/ssh -x -l oracle rac121 date to verify SSH connectivity has been setup
from local host to rac121.
IF YOU SEE ANY
OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR
A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that
being prompted for a passphrase may be OK but being prompted for a password is
ERROR.
Thu Apr 6 14:01:39 IST 2017
------------------------------------------------------------------------
--rac123:--
Running
/usr/bin/ssh -x -l oracle rac123 date to verify SSH connectivity has been setup
from local host to rac123.
IF YOU SEE ANY
OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR
A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that
being prompted for a passphrase may be OK but being prompted for a password is
ERROR.
Thu Apr 6 14:01:39 IST 2017
------------------------------------------------------------------------
SSH verification
complete.
rac121:ccbdb1>
Run
the cluster verification utility to check the node configuration between nodes
before adding nodes.
rac121:ccbdb1>cluvfy stage
-pre nodeadd -n rac123 -fixup -fixupnoexec
Performing
pre-checks for node addition
Checking node
reachability...
Node reachability
check passed from node "rac121"
Checking user
equivalence...
User equivalence
check passed for user "oracle"
Checking CRS
integrity...
CRS integrity check
passed
....
....
Starting check for
Reverse path filter setting ...
Check for Reverse
path filter setting passed
Starting check for
Network interface bonding status of private interconnect network interfaces ...
Check for Network
interface bonding status of private interconnect network interfaces passed
Starting check for
/dev/shm mounted as temporary file system ...
Check for /dev/shm
mounted as temporary file system passed
Starting check for
/boot mount ...
Check for /boot
mount passed
Starting check for
zeroconf check ...
Check for zeroconf
check passed
Pre-check for
cluster services setup was successful.
Since
we have successfully validated the cluvfy, now it's high time to add our 3rd node
to our existing 2 node cluster.
rac121:ccbdb1>./addnode.sh
-silent "CLUSTER_NEW_NODES={rac123}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac123-vip}"
Starting Oracle
Universal Installer...
Checking Temp
space: must be greater than 120 MB.
Actual 33259 MB Passed
Checking swap
space: must be greater than 150 MB.
Actual 7997 MB Passed
Prepare
Configuration in progress.
Prepare
Configuration successful.
.................................................. 8% Done.
You can find the
log of this install session at:
/grid/app/oraInventory/logs/addNodeActions2017-04-06_02-41-43PM.log
Instantiate files
in progress.
Instantiate files
successful.
.................................................. 14% Done.
Copying files to
node in progress.
Copying files to
node successful.
.................................................. 73% Done.
Saving cluster
inventory in progress.
.................................................. 80% Done.
Saving cluster
inventory successful.
The Cluster Node
Addition of /grid/app/12.1.0.1/grid was successful.
Please check
'/tmp/silentInstall.log' for more details.
Setup Oracle Base
in progress.
Setup Oracle Base
successful.
.................................................. 88% Done.
As a root user, execute
the following script(s):
1.
/grid/app/oraInventory/orainstRoot.sh
2. /grid/app/12.1.0.1/grid/root.sh
Execute
/grid/app/oraInventory/orainstRoot.sh on the following nodes:
[rac123]
Execute
/grid/app/12.1.0.1/grid/root.sh on the following nodes:
[rac123]
The scripts can be
executed in parallel on all the nodes.
..........
Update Inventory in
progress.
.................................................. 100% Done.
Update Inventory
successful.
Successfully Setup Software.
Execute
the orainstRoot.sh and root.sh at node 3 as privilege user.
rac123:ccbdb3>sudo
/grid/app/oraInventory/orainstRoot.sh
We trust you have received the usual lecture
from the local System
Administrator. It usually boils down to these
three things:
#1)
Respect the privacy of others.
#2)
Think before you type.
#3)
With great power comes great responsibility.
[sudo] password for oracle:
Changing permissions of
/grid/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for
world.
Changing groupname of /grid/app/oraInventory
to oinstall.
The
execution of the script is complete.
rac123:ccbdb3>sudo
/grid/app/12.1.0.1/grid/root.sh
Performing root user operation.
The following environment variables are set
as:
ORACLE_OWNER= oracle
ORACLE_HOME=
/grid/app/12.1.0.1/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file
as needed by
Database Configuration Assistant when a
database is created
Finished running generic part of root script.
Now product-specific root actions will be
performed.
Relinking oracle with rac_on option
Using configuration parameter file:
/grid/app/12.1.0.1/grid/crs/install/crsconfig_params
2017/04/06 14:56:07 CLSRSC-4001: Installing
Oracle Trace File Analyzer (TFA) Collector.
2017/04/06 14:56:42 CLSRSC-4002: Successfully
installed Oracle Trace File Analyzer (TFA) Collector.
OLR initialization - successful
2017/04/06 14:57:38 CLSRSC-330: Adding
Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services
has been stopped.
CRS-4123: Oracle High Availability Services
has been started.
CRS-4133: Oracle High Availability Services
has been stopped.
CRS-4123: Oracle High Availability Services
has been started.
CRS-2791: Starting shutdown of Oracle High
Availability Services-managed resources on 'rac123'
CRS-2673: Attempting to stop
'ora.drivers.acfs' on 'rac123'
CRS-2677: Stop of 'ora.drivers.acfs' on
'rac123' succeeded
CRS-2793: Shutdown of Oracle High
Availability Services-managed resources on 'rac123' has completed
CRS-4133: Oracle High Availability Services
has been stopped.
CRS-4123: Starting Oracle High Availability
Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on
'rac123'
CRS-2672: Attempting to start 'ora.evmd' on
'rac123'
CRS-2676: Start of 'ora.mdnsd' on 'rac123'
succeeded
CRS-2676: Start of 'ora.evmd' on 'rac123'
succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on
'rac123'
CRS-2676: Start of 'ora.gpnpd' on 'rac123'
succeeded
CRS-2672: Attempting to start 'ora.gipcd' on
'rac123'
CRS-2676: Start of 'ora.gipcd' on 'rac123'
succeeded
CRS-2672: Attempting to start
'ora.cssdmonitor' on 'rac123'
CRS-2676: Start of 'ora.cssdmonitor' on
'rac123' succeeded
CRS-2672: Attempting to start 'ora.cssd' on
'rac123'
CRS-2672: Attempting to start 'ora.diskmon'
on 'rac123'
CRS-2676: Start of 'ora.diskmon' on 'rac123'
succeeded
CRS-2676: Start of 'ora.cssd' on 'rac123'
succeeded
CRS-2672: Attempting to start
'ora.cluster_interconnect.haip' on 'rac123'
CRS-2672: Attempting to start 'ora.ctssd' on
'rac123'
CRS-2676: Start of 'ora.ctssd' on 'rac123'
succeeded
CRS-2676: Start of
'ora.cluster_interconnect.haip' on 'rac123' succeeded
CRS-2672: Attempting to start 'ora.asm' on
'rac123'
CRS-2676: Start of 'ora.asm' on 'rac123'
succeeded
CRS-2672: Attempting to start 'ora.storage'
on 'rac123'
CRS-2676: Start of 'ora.storage' on 'rac123'
succeeded
CRS-2672: Attempting to start 'ora.crf' on
'rac123'
CRS-2676: Start of 'ora.crf' on 'rac123'
succeeded
CRS-2672: Attempting to start 'ora.crsd' on
'rac123'
CRS-2676: Start of 'ora.crsd' on 'rac123'
succeeded
CRS-6017: Processing resource auto-start for
servers: rac123
CRS-2673: Attempting to stop
'ora.LISTENER_SCAN2.lsnr' on 'rac121'
CRS-2672: Attempting to start
'ora.net1.network' on 'rac123'
CRS-2672: Attempting to start 'ora.FLASH.dg'
on 'rac123'
CRS-2672: Attempting to start 'ora.DATA.dg'
on 'rac123'
CRS-2676: Start of 'ora.net1.network' on
'rac123' succeeded
CRS-2672: Attempting to start 'ora.ons' on
'rac123'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr'
on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip'
on 'rac121'
CRS-2676: Start of 'ora.DATA.dg' on 'rac123'
succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac121'
succeeded
CRS-2672: Attempting to start 'ora.scan2.vip'
on 'rac123'
CRS-2676: Start of 'ora.FLASH.dg' on 'rac123'
succeeded
CRS-2676: Start of 'ora.scan2.vip' on
'rac123' succeeded
CRS-2672: Attempting to start
'ora.LISTENER_SCAN2.lsnr' on 'rac123'
CRS-2676: Start of 'ora.ons' on 'rac123'
succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr'
on 'rac123' succeeded
CRS-6016: Resource auto-start has completed
for server rac123
CRS-6024: Completed start of Oracle Cluster
Ready Services-managed resources
CRS-4123: Oracle High Availability Services
has been started.
2017/04/06 15:01:18 CLSRSC-343: Successfully
started Oracle Clusterware stack
clscfg: EXISTING configuration version 5
detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp
'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
2017/04/06 15:01:46 CLSRSC-325:
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Seems
we have added node to our cluster successfully.
Need to validate the cluster.
rac123:+ASM3>crsctl check
cluster -all
******************************************************
rac121:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************
rac122:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************
rac123:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
******************************************************
rac123:+ASM3>crsctl stat res
-t
------------------------------------------------------------
Name Target State
Server State
details
------------------------------------------------------------
Local Resources
------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE rac123
STABLE
ora.DG_OCR1.dg
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE
rac123 STABLE
ora.FLASH.dg
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE
rac123 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE
rac123 STABLE
ora.asm
ONLINE ONLINE
rac121 Started,STABLE
ONLINE ONLINE
rac122 Started,STABLE
ONLINE ONLINE
rac123 Started,STABLE
ora.net1.network
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE
rac123 STABLE
ora.ons
ONLINE ONLINE
rac121 STABLE
ONLINE ONLINE
rac122 STABLE
ONLINE ONLINE
rac123 STABLE
------------------------------------------------------------
Cluster Resources
------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE
ONLINE rac122
STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE
ONLINE rac123
STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE
ONLINE rac121
STABLE
ora.MGMTLSNR
1 ONLINE
ONLINE rac121
169.254.26.48 10.10.
10.121 20.20.20.121,
STABLE
ora.ccbdb.db
1 ONLINE
ONLINE rac121
Open,STABLE
2 ONLINE
ONLINE rac122
Open,STABLE
ora.cvu
1 ONLINE
ONLINE rac121
STABLE
ora.mgmtdb
1 ONLINE
ONLINE rac121
Open,STABLE
ora.oc4j
1 ONLINE
ONLINE rac121
STABLE
ora.rac121.vip
1 ONLINE
ONLINE rac121
STABLE
ora.rac122.vip
1 ONLINE
ONLINE rac122
STABLE
ora.rac123.vip
1 ONLINE
ONLINE rac123
STABLE
ora.scan1.vip
1 ONLINE
ONLINE rac122
STABLE
ora.scan2.vip
1 ONLINE
ONLINE rac123
STABLE
ora.scan3.vip
1 ONLINE
ONLINE rac121
STABLE
------------------------------------------------------------
Adding RDBMS nodes in silent mode manually
Now
adding the RDBMS home to the server.
rac121:ccbdb1>./addnode.sh
-silent "CLUSTER_NEW_NODES={rac123}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 32896 MB Passed
Checking swap space: must be greater than 150 MB. Actual 7992 MB Passed
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 8% Done.
You can find the log of this install session at:
/grid/app/oraInventory/logs/addNodeActions2017-04-07_12-10-53AM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 14% Done.
Copying files to node in progress.
Copying files to node successful.
.................................................. 73% Done.
Saving cluster inventory in progress.
.................................................. 80% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.1/DB_1
was successful.
Please check '/tmp/silentInstall.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 88% Done.
As a root user, execute
the following script(s):
1.
/u01/app/oracle/product/12.1.0.1/DB_1/root.sh
Execute /u01/app/oracle/product/12.1.0.1/DB_1/root.sh on the
following nodes:
[rac123]
..........
Update Inventory in progress.
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
Execute
the root.sh as privileges user on node3.
rac123:ccbdb3>sudo
/u01/app/oracle/product/12.1.0.1/DB_1/root.sh
Check
/u01/app/oracle/product/12.1.0.1/DB_1/install/root_rac123.ora.com_2017-04-07_01-02-18.log
for the output of root script
Since
GRID/RDBMS has been added to cluster, now time to add the instance in OCR.
rac123:+ASM3>srvctl config
database -d ccbdb
Database unique name: ccbdb
Database name: ccbdb
Oracle home: /u01/app/oracle/product/12.1.0.1/DB_1
Oracle user: oracle
Spfile: +DATA/CCBDB/PARAMETERFILE/spfile.276.919855143
Password file: +DATA/CCBDB/PASSWORD/pwdccbdb.256.919854693
Domain: ora.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances:
ccbdb1,ccbdb2
Configured nodes:
rac121,rac122
Database is administrator managed
Create
ifile (pfile) on node3 which point the spfile in diskgroup.
Add
the below parameter in SPFILE.
ccbdb3.instance_number=3
ccbdb3.thread=3
ccbdb3.undo_tablespace='UNDOTBS3'
Create
Undo tablespace and redo logs for node 3
.
SQL> create undo tablespace
UNDOTBS3 datafile size 1g autoextend
on ;
Tablespace created.
alter database add logfile thread 3 ;
alter database add logfile thread 3 ;
Enable
Public Thread for node 3.
SQL> alter database enable public thread 3 ;
Add
instance to cluster.
rac123:+ASM3>srvctl add
instance -d ccbdb -n rac123 -i ccbdb3
rac123:+ASM3>srvctl config database -d ccbdb
Database unique name: ccbdb
Database name: ccbdb
Oracle home: /u01/app/oracle/product/12.1.0.1/DB_1
Oracle user: oracle
Spfile: +DATA/CCBDB/PARAMETERFILE/spfile.276.919855143
Password file: +DATA/CCBDB/PASSWORD/pwdccbdb.256.919854693
Domain: ora.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: ccbdb1,ccbdb2,ccbdb3
Configured nodes: rac121,rac122,rac123
Database is administrator managed
rac121:ccbdb1>srvctl start
database -d ccbdb
Validate
the database
rac121:ccbdb1>srvctl status
database -d ccbdb
Instance ccbdb1 is running on node rac121
Instance ccbdb2 is running on node rac122
Instance ccbdb3 is running on node rac123
SQL> select distinct name,
instance_name,host_name, open_mode from gv$database a, gv$instance b ;
NAME INSTANCE_NAME HOST_NAME OPEN_MODE
--------- ---------------- --------------- --------------------
CCBDB ccbdb3 rac123.ora.com READ WRITE
CCBDB ccbdb1 rac121.ora.com READ WRITE
CCBDB ccbdb2 rac122.ora.com READ WRITE
Hope this help...:)
