Introduction and Storage Setup
Introduction:
This document is designed as a step by step explanation of the setup of an Oracle 18c RAC database on an Oracle Linux 7 system using an iSCSI interface to a NAS storage array.
Note:
This document should be used in conjunction with the following documents:
1. Oracle Grid Infrastructure Installation and Upgrade Guide, 18c for Linux.
( July 2018. part no. E84125-02)
2. Oracle Real Application Clusters Installation Guide, 18c for Linux and UNIX
( July 2018. part no. E84128-02)
3. Oracle Database Installation Guide, 18c Release 3 (18.3) for Linux.
( July 2018 part no. E83745-02)
.
Software used:
1. Oracle Database 18c Grid Infrastructure (18.3) for Linux x86-64
2. Oracle Database 18c (18.3) for Linux x86-64
3. Oracle Linux 7
Overview:
1. Setup iSCSI.
2. Create mount points for CRS
3. Prepare servers for Oracle Grid installation
4. Setup Udev access to NAS storage
4. Install Oracle Grid
5. Install Oracle ASM
6. Install Oracle 18c
Step 1: Install the iSCSI software
# yum install iscsi-initiator-utils
Step 2: Start iSCSI services
# systemctl start iscsi.service
# systemctl start iscsid.service
Step 3: Connect to the NAS via iSCSI
Discover iSCSI targets
[root@linuxdb3 ~]# iscsiadm -m discovery -t sendtargets -p nas1
10.5.1.106:3260,257 iqn.2005-10.org.nas.ctl:crs1
10.5.1.106:3260,257 iqn.2005-10.org.nas.ctl:data1
10.5.1.106:3260,257 iqn.2005-10.org.nas.ctl:fra1
On node startup connect automatically to all iSCSI disks
[root@linuxdb3 ~]# iscsiadm -m node --op update -n node.startup -v automatic
Login to NAS and map iSCSI disks
[root@linuxdb3 ~]# iscsiadm -m node -p nas1 --login
.
.
.
Login to [iface: default, target: iqn.2005-10.org.nas.ctl:crs1, portal: 10.5.1.106,3260] successful.
Login to [iface: default, target: iqn.2005-10.org.nas.ctl:data1, portal: 10.5.1.106,3260] successful.
Login to [iface: default, target: iqn.2005-10.org.nas.ctl:fra1, portal: 10.5.1.106,3260] successful.
View the target disks
[root@linuxdb3 ~]# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:crs1-lun-1 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:crs1-lun-1-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 9 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:data1-lun-2 -> ../../sdd
lrwxrwxrwx. 1 root root 10 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:data1-lun-2-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root 9 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:fra1-lun-3 -> ../../sde
lrwxrwxrwx. 1 root root 10 Jun 12 18:18 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:fra1-lun-3-part1 -> ../../sde1
# lsscsi
.
.
.
[3:0:0:1] disk NAS iSCSI Disk 0123 /dev/sdc
[4:0:0:2] disk NAS iSCSI Disk 0123 /dev/sdd
[5:0:0:3] disk NAS iSCSI Disk 0123 /dev/sde
Step 4: Format iSCSI disks
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x405ec941.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): p
Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 1048576 bytes
Disk label type: dos
Disk identifier: 0x9265447a
Device Boot Start End Blocks Id System
/dev/sdc1 2048 20971519 10484736 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Check result of disk format
# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jun 12 18:28 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:crs1-lun-1 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Jun 12 18:28 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:crs1-lun-1-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 9 Jun 12 18:30 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:data1-lun-2 -> ../../sdd
lrwxrwxrwx. 1 root root 10 Jun 12 18:30 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:data1-lun-2-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root 9 Jun 12 18:31 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:fra1-lun-3 -> ../../sde
lrwxrwxrwx. 1 root root 10 Jun 12 18:31 ip-10.5.1.106:3260-iscsi-iqn.2005-10.org.nas.ctl:fra1-lun-3-part1 -> ../../sde1
.
.
.
Clean the headers of the block devices if they already existed before the install
# dd if=/dev/zero of=/dev/sdb1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0381133 s, 13.4 MB/s
# dd if=/dev/zero of=/dev/sdc1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0381133 s, 13.4 MB/s
# dd if=/dev/zero of=/dev/sdd1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0381133 s, 13.4 MB/s
Update the other nodes
# partprobe
Step 5: Configure UDEV rules
Create /etc/scsi_id.config file if it doesn’r exist.
add line:
options=-g
Get the SCSI ID for the iscsi disks
# /usr/lib/udev/scsi_id -g -u -d /dev/sdb1
36589cfc00000061e57a2c37cc800c105
# /usr/lib/udev/scsi_id -g -u -d /dev/sdc1
36589cfc000000528b60b31dca0da3712
# /usr/lib/udev/scsi_id -g -u -d /dev/sdd1
36589cfc0000005a5aa7bedd4ba7cd334
# /usr/lib/udev/scsi_id -g -u -d /dev/sde1
36589cfc000000ebd5355d5fec47b5986
Using the above values create or edit the /etc/udev/rules.d/99-oracle-asmdevices.rules file.
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc00000061e57a2c37cc800c105", SYMLINK+="oracleasm/asmdisk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000528b60b31dca0da3712", SYMLINK+="oracleasm/asmdisk2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc0000005a5aa7bedd4ba7cd334", SYMLINK+="oracleasm/asmdisk3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000ebd5355d5fec47b5986", SYMLINK+="oracleasm/asmdisk4", OWNER="grid", GROUP="asmadmin", MODE="0660"
The /dev/oracleasm/asmdisk* links provide a persistent name for the block devices. With each boot the order in which device nodes are added is arbitrary and can result in device names being assigned to different disks on each boot. Persistent naming solves this problem.
Copy or create the /etc/udev/rules.d/99-oracle-asmdevices.rules file on the other servers.
Load the block partition tables on each server
# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# /sbin/partprobe /dev/sdd1
# /sbin/partprobe /dev/sde1
Test the rules
# /sbin/udevadm test /block/sdb/sdb1
Reload the UDEV rules:
# /sbin/udevadm control --reload-rules
The links to the disks should now be visible:
# ls -al /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Sep 24 18:55 /dev/oracleasm/asmdisk1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Sep 24 18:53 /dev/oracleasm/asmdisk2 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Sep 24 18:53 /dev/oracleasm/asmdisk3 -> ../sdd1
lrwxrwxrwx. 1 root root 7 Sep 24 18:53 /dev/oracleasm/asmdisk4 -> ../sde1
The links are owned by root but the devices should have the correct ownership which was defined in the 99-oracle-asmdevices.rules file
# ls -al /dev/sd*1
brw-rw----. 1 root disk 8, 1 Sep 24 17:01 /dev/sda1
brw-rw----. 1 grid asmadmin 8, 17 Sep 24 18:55 /dev/sdb1
brw-rw----. 1 grid asmadmin 8, 33 Sep 24 18:53 /dev/sdc1
brw-rw----. 1 grid asmadmin 8, 49 Sep 24 18:53 /dev/sdd1
brw-rw----. 1 grid asmadmin 8, 65 Sep 24 18:53 /dev/sde1
Note: Direct use of block or raw devices are not supported by Oracle Universal Installer (OUI) or Database Configuration Assistant (DBCA). They must be registered with ASM.
