Network setup and configuration
Each node will have two network interfaces. One for the public IP address and another for the private IP address. This install will use a static IP address configuration. Oracle also provides Grid Naming Service for dynamic IP address assignment.
Create the following:
1. A public IP address for each node
2. A private IP address on a separate subnet
3. A virtual IP address for each node in the same subnet as the public IP addresses
4. Three SCAN (single client access name) IP addresses in the public subnet
Note: Oracle recommends the use of network interfaces of at least 1Gbe, with 10Gbe recommended. Infiniband is an alternative recommended network interface.
If a NAS server, accessed by ethernet, is used, a third network interface for the NAS I/O is recommended.
Update /etc/hosts file with node names and IP addresses
# vi /etc/hosts
. . .
# Public IP addresses for the RAC nodes
192.168.2.147 linuxdb3.localdomain linuxdb43
192.168.2.145 linuxdb4.localdomain linuxdb44
# Private IP addresses for the RAC nodes
10.5.1.20 linuxdb3-rac1-priv.localdomain linuxdb43-rac1-priv
10.5.1.30 linuxdb4-rac2-priv.localdomain linuxdb44-rac2-priv
# Virtual IP addresses for the RAC nodes
192.168.2.125 linuxdb3-rac1-vip.localdomain linuxdb43-rac1-vip
192.168.2.126 linuxdb4-rac2-vip.localdomain linuxdb44-rac2-vip
# SCAN IP addresses for the RAC nodes
192.168.2.135 linuxdb-rac-scan.localdomain linuxdb-rac-scan
192.168.2.136 linuxdb-rac-scan.localdomain linuxdb-rac-scan
192.168.2.137 linuxdb-rac-scan.localdomain linuxdb-rac-scan
Setup DNS
Setup DNS server on a separate server or one of the cluster servers. See Appendix A
Create required users and groups
# groupadd oinstall
# groupadd dba
# groupadd oper
# groupadd asmadmin
# groupadd asmdba
# groupadd asmoper
# groupadd backupdba
# groupadd dgdba
# groupadd kmdba
# groupadd racdba
# useradd grid -g oinstall -G asmadmin,asmdba,asmoper
# useradd oracle -g oinstall -G dba,oper,asmdba
# passwd grid
Changing password for user grid.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully
Create Grid and Database Home directories
# mkdir -p /u01/app/18.3.0/grid
# mkdir -p /u01/app/grid
# mkdir -p /u01/app/oracle
# chown -R grid:oinstall /u01
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/
Install and configure ssh, scp
1. Login as the grid user
2. Create the directory .ssh
$ mkdir ~/.ssh
$ chmod 755 ~/.ssh
3. Generate an RSA key. At the prompts accept the default location and ignore password request
$ /usr/bin/ssh-keygen -t rsa
A public key is written to ~/.ssh/id_rsa.pub and a private key is written to the file ~/.ssh/id_rsa.
4. Generate a DSA key.
$ /usr/bin/ssh-keygen -t dsa
A public key is written to ~/.ssh/id_dsa.pub and a private key is written to the file ~/.ssh/id_dsa.
l
5. Copy the contents of the ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub files to the ~/.ssh/authorized_keys file on this node and to the same file on all other
cluster nodes.
Note: The ~/.ssh/authorized_keys file on every node must contain the contents from all of the ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub files that you generated on all cluster nodes.
6. Change the permissions on the ~/.ssh/authorized_keys file on all cluster
nodes:
$ chmod 644 ~/.ssh/authorized_keys
7. If you have configured SSH correctly, you can now use the ssh or scp commands without being prompted for a password.
To test the SSH configuration, enter the following commands:
$ ssh nodename1 uname -n
$ ssh nodename2 uname –n
$ scp test1 node2:test1
You should see the name of the server displayed without a request for a password. If any node prompts for a password, verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys.
Note: The first time you use SSH to connect to a node from a particular system, you might see a message stating that the authenticity of the host could not be established. Enter yes at the prompt to continue. The node is now added to the ~/.ssh/authorized_hosts file. Connect to each node using the node names and IP addresses so that this prompt will not occur during installation.
8. To ensure that X11 forwarding will not cause the installation to fail, create the ~oracle/.ssh/config file. Put the following text into the file:
Host *
ForwardX11 no
Install the Oracle Linux packages needed, set resource usage and kernel parameters.
Install Packages
# yum install oracle-database-preinstall-18c -y
Loaded plugins: langpacks, ulninfo
ol7_UEKR4 | 1.2 kB 00:00:00
ol7_latest | 1.4 kB 00:00:00
(1/2): ol7_UEKR4/x86_64/updateinfo | 82 kB 00:00:00
(2/2): ol7_UEKR4/x86_64/primary | 2.0 MB 00:00:00
ol7_UEKR4 103/103
Resolving Dependencies
--> Running transaction check
---> Package oracle-database-preinstall-18c.x86_64 0:1.0-1.el7 will be installed
--> Processing Dependency: ksh for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64
--> Processing Dependency: libaio-devel for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64
--> Running transaction check
---> Package ksh.x86_64 0:20120801-137.0.1.el7 will be installed
---> Package libaio-devel.x86_64 0:0.3.109-13.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================
Package Arch Version Repository Size
=============================================================================================
Installing:
oracle-database-preinstall-18c x86_64 1.0-1.el7 ol7_latest 18 k
Installing for dependencies:
ksh x86_64 20120801-137.0.1.el7 ol7_latest 881 k
libaio-devel x86_64 0.3.109-13.el7 ol7_latest 12 k
Transaction Summary
=============================================================================================
Install 1 Package (+2 Dependent packages)
Total download size: 911 k
Installed size: 3.2 M
Downloading packages:
(1/3): libaio-devel-0.3.109-13.el7.x86_64.rpm | 12 kB 00:00:00
(2/3): ksh-20120801-137.0.1.el7.x86_64.rpm | 881 kB 00:00:00
(3/3): oracle-database-preinstall-18c-1.0-1.el7.x86_64.rpm | 18 kB 00:00:02
---------------------------------------------------------------------------------------------
Total 304 kB/s | 911 kB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : ksh-20120801-137.0.1.el7.x86_64 1/3
Installing : libaio-devel-0.3.109-13.el7.x86_64 2/3
Installing : oracle-database-preinstall-18c-1.0-1.el7.x86_64 3/3
Verifying : oracle-database-preinstall-18c-1.0-1.el7.x86_64 1/3
Verifying : libaio-devel-0.3.109-13.el7.x86_64 2/3
Verifying : ksh-20120801-137.0.1.el7.x86_64 3/3
Installed:
oracle-database-preinstall-18c.x86_64 0:1.0-1.el7
Dependency Installed:
ksh.x86_64 0:20120801-137.0.1.el7 libaio-devel.x86_64 0:0.3.109-13.el7
Complete!
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Do a full update to be sure
# yum update -y
Check log file to review kernel changes
# vi /var/log/oracle-database-preinstall-18c/backup/your_timestamp/orakernel.log
Set resource usage settings for user grid and oracle
The resource usage should have been set by the oracle-database-preinstall-18c
package in a new file that bypasses the old ‘/etc/security/limits.conf’ file. Check the files and update them manually if necessary.
# cat /etc/security/limits.d/oracle-database-preinstall-18c.conf
...
# oracle-database-preinstall-18c setting for nofile soft limit is 1024
oracle soft nofile 1024
# oracle-database-preinstall-18c setting for nofile hard limit is 65536
oracle hard nofile 65536
# oracle-database-preinstall-18c setting for nproc soft limit is 16384
# refer orabug15971421 for more info.
oracle soft nproc 16384
# oracle-database-preinstall-18c setting for nproc hard limit is 16384
oracle hard nproc 16384
# oracle-database-preinstall-18c setting for stack soft limit is 10240KB
oracle soft stack 10240
# oracle-database-preinstall-18c setting for stack hard limit is 32768KB
oracle hard stack 32768
# oracle-database-preinstall-18c setting for memlock hard limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
oracle hard memlock 134217728
# oracle-database-preinstall-18c setting for memlock soft limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90% of RAM
oracle soft memlock 134217728
The file above has no entries for the grid user. Udate the /etc/security/limits.conf file with the same information from above and include limits for the grid user.
# vi /etc/security/limits.conf
...
grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid soft memlock unlimited
grid hard memlock unlimited
Verify that system kernel parameters are correctly set
# cat /etc/sysctl.conf
# oracle-database-preinstall-18c setting for fs.file-max is 6815744
fs.file-max = 6815744
# oracle-database-preinstall-18c setting for kernel.sem is '250 32000 100 128'
kernel.sem = 250 32000 100 128
# oracle-database-preinstall-18c setting for kernel.shmmni is 4096
kernel.shmmni = 4096
# oracle-database-preinstall-18c setting for kernel.shmall is 1073741824 on x86_64
kernel.shmall = 1073741824
# oracle-database-preinstall-18c setting for kernel.shmmax is 4398046511104 on x86_64
kernel.shmmax = 4398046511104
# oracle-database-preinstall-18c setting for kernel.panic_on_oops is 1 per Orabug 19212317
kernel.panic_on_oops = 1
# oracle-database-preinstall-18c setting for net.core.rmem_default is 262144
net.core.rmem_default = 262144
# oracle-database-preinstall-18c setting for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# oracle-database-preinstall-18c setting for net.core.wmem_default is 262144
net.core.wmem_default = 262144
# oracle-database-preinstall-18c setting for net.core.wmem_max is 1048576
net.core.wmem_max = 1048576
# oracle-database-preinstall-18c setting for net.ipv4.conf.all.rp_filter is 2
net.ipv4.conf.all.rp_filter = 2
# oracle-database-preinstall-18c setting for net.ipv4.conf.default.rp_filter is 2
net.ipv4.conf.default.rp_filter = 2
# oracle-database-preinstall-18c setting for fs.aio-max-nr is 1048576
fs.aio-max-nr = 1048576
# oracle-database-preinstall-18c setting for net.ipv4.ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
If sysctl.conf file is manually updated then run the following command to update the kernel parameters
# /sbin/sysctl -p
Disable the Firewall
# systemctl stop firewalld
# systemctl disable firewalld
Disable Avahi-daemon
# systemctl stop avahi-daemon
Warning: Stopping avahi-daemon.service, but it can still be activated by:
avahi-daemon.socket
# systemctl disable avahi-daemon
Removed symlink /etc/systemd/system/multi-user.target.wants/avahi-daemon.service.
Removed symlink /etc/systemd/system/sockets.target.wants/avahi-daemon.socket.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service.
Install and set NTP Clock synchronization
On one of the servers update the chrony.conf file with ntp server list. A public time server or a separate server in your domain can be used as a time server.
Open /etc/chrony.conf file
# vi /etc/chrony.conf
...
server 0.north-america.pool.ntp.org iburst
server 1.north-america.pool.ntp.org iburst
server 2.north-america.pool.ntp.org iburst
server 3.north-america.pool.ntp.org iburst
# Allow NTP client access from local network.
allow 192.168/16
...
On the other servers update the chrony.conf file with the name of the server accessing the public time servers.
...
server linuxdb3 iburst
...
Start or restart NTP service.
# systemctl enable chronyd.service
# systemctl start chronyd.service
