This is a small article on using UDev on RHEL 7. I have a virtual box with RHEL 7 and I am configuring ASM diskgroups.
Before we configure ASM diskgroups, we need to have disks/partitions available at OS level and those should be recognized by ASM. Oracle provides a utility called ASMLib which can be installed and used very easily to configure disk/partitions at OS level.
ASMLib stamps the header of partitions/disk at OS level with Oracle format making them easily detected by ASM instance doing disk discovery. You can go through ASMLib installation, configuration and usage in an article write by my friend Anand Prakash – https://aprakash.wordpress.com/2016/05/19/oracle-asmlib
In this article we will focus on using UDev for setting up ASM disks.
Let’s start with some information on UDev
What is Udev?
Udev is the device manager for the Linux 2.6 kernel that creates/removes device nodes in the /dev directory dynamically. It is the successor of devfs and hotplug. It runs in userspace and the user can change device names using Udev rules.
Why Do We Need It ?
In the older kernels, the /dev directory contained statics device files. But with dynamic device creation, device nodes for only those devices which are actually present in the system are created. Let us see the disadvantages of the static /dev directory, which led to the development of Udev.
Problems Identifying the Exact Hardware Device for a Device Node in /dev
The kernel will assign a major/minor number pair when it detects a hardware device while booting the system. Let us consider two hard disks. The connection/alignment is in such a way that one is connected as a master and the other, as a slave. The Linux system will call them, /dev/hda and /dev/hdb. Now, if we interchange the disks the device name will change. This makes it difficult to identify the correct device that is related to the available static device node. The condition gets worse when there are a bunch of hard disks connected to the system.
Udev provides a persistent device naming system through the /dev directory, making it easier to identify the device.
Huge Number of Device Nodes in /dev
In the static model of device node creation, no method was available to identify the hardware devices actually present in the system. So, device nodes were created for all the devices that Linux was known to support at the time. The huge mess of device nodes in /dev made it difficult to identify the devices actually present in the system.
Not Enough Major/Minor Number Pairs
The number of static device nodes to be included increased a lot in recent times and the 8-bit scheme, that was used, proved to be insufficient for handling all the devices. As a result the major/minor number pairs started running out.
Working of Udev
The Udev daemon listens to the netlink socket that the kernel uses for communicating with user space applications. The kernel will send a bunch of data through the netlink socket when a device is added to, or removed from a system. The Udev daemon catches all this data and will do the rest, i.e., device node creation, module loading etc.
Kernel Device Event Management
- When bootup is initialized, the /dev directory is mounted in tmpfs.
- After that, Udev will copy the static device nodes from /lib/udev/devices to the /dev directory.
- The Udev daemon then runs and collects uevents from the kernel, for all the devices connected to the system.
- The Udev daemon will parse the uevent data and it will match the data with the rules specified in /etc/udev/rules.d.
- It will create the device nodes and symbolic links for the devices as specified in the rules.
The Udev daemon reads the rules from /etc/udev/rules.d/*.rules and stores them in the memory. - Udev will receive an inotify event, if any rules were changed. It will read the changes and will update the memory.
Let’s start by creating disks at OS level. VirtualBox has commands which creates block devices at OS level.
Using Udev to configure disks
Step 1) Creating Block devices at OS level
I am creating 4 disks, each of 10GB using following command
VBoxManage createhd --filename asm1.vdi --size 10240 --format VDI --variant Fixed VBoxManage createhd --filename asm2.vdi --size 10240 --format VDI --variant Fixed VBoxManage createhd --filename asm3.vdi --size 10240 --format VDI --variant Fixed VBoxManage createhd --filename asm4.vdi --size 10240 --format VDI --variant Fixed
Step 2) Attaching the disk to the correct storage
VBoxManage storageattach 12102 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable VBoxManage storageattach 12102 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable VBoxManage storageattach 12102 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable VBoxManage storageattach 12102 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
12102 is the name of virtual machine. You can see the virtual machine name as shown in below screen shot.
Image may be NSFW.
Clik here to view.
Step 3) (optional) Making disk sharable
This is optional step and is required only if you are using ASM cluster. In that case ASM disks should be made shareable as should be attached to all machines in cluster. You can use following commands to make the disks sharable
VBoxManage modifyhd asm1.vdi --type shareable VBoxManage modifyhd asm2.vdi --type shareable VBoxManage modifyhd asm3.vdi --type shareable VBoxManage modifyhd asm4.vdi --type shareable
Step 4) Check if the disks are now shown in your virtual box
Following screen shot shows that disks are now added to correct virtual machine
Image may be NSFW.
Clik here to view.
Also, after you login to virtual box, you should be able to see new devices under /dev directory
[root@advait ~]# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Jul 18 05:36 /dev/sda brw-rw---- 1 root disk 8, 1 Jul 18 05:36 /dev/sda1 brw-rw---- 1 root disk 8, 2 Jul 18 05:36 /dev/sda2 brw-rw---- 1 root disk 8, 16 Jul 18 05:36 /dev/sdb brw-rw---- 1 root disk 8, 32 Jul 18 05:36 /dev/sdc brw-rw---- 1 root disk 8, 48 Jul 18 05:36 /dev/sdd brw-rw---- 1 root disk 8, 64 Jul 18 05:36 /dev/sde
/dev/sda, /dev/sda1 and /dev/sda2 are partitions of main device used for virtual box.
/dev/sdb, /dev/sdc, /dev/sdd and /dev/sde are the devices we added to virtual box as above.
Step 5) Format the new devices and create partitions
You need to create new partitions using fdisk utility. I am showing the command for one partition, you can repeate the same for remaining partitions.
[root@advait ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xbda01838. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-20971519, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): Using default value 20971519 Partition 1 of type Linux and of size 10 GiB is set Command (m for help): p Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xbda01838 Device Boot Start End Blocks Id System /dev/sdb1 2048 20971519 10484736 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
This will create single partition for each of the devices.
[root@advait ~]# ls -rlt /dev/sd*1 brw-rw---- 1 root disk 8, 1 Jul 18 05:36 /dev/sda1 brw-rw---- 1 root disk 8, 17 Jul 18 05:40 /dev/sdb1 brw-rw---- 1 root disk 8, 33 Jul 18 05:40 /dev/sdc1 brw-rw---- 1 root disk 8, 49 Jul 18 05:41 /dev/sdd1 brw-rw---- 1 root disk 8, 65 Jul 18 05:41 /dev/sde1
Step 5) Configure UDev rules
So before creating UDev rule, we need to understand what exactly we want to do. We want to create alias for each of the disk that we created at OS level so that it is always identified the same way, regardless of the device name Linux assigns it. This can be done by recognizing each device based on some unique IDs and assining the alias to that device. UDev can do just that.
Each disk has a unique SCSI ID. We can use this unique ID to recognize the disk and assign the required alias to that disk.
We can get unique SCSI_ID using following command
[root@advait ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb1 1ATA_VBOX_HARDDISK_VB544d069c-abd3901e
scsi_id command is located in /usr/lib/udev/ directory on RHEL 7. But in previous release this used to be in /sbin/ location.
Like wise we can find SCSI_ID for each disk that we added.
Rules are defined in “/etc/udev/rules.d” directory. Udev reads these rules and apply them to devices listed in /dev directory.
Rules looks like below
KERNEL=="sd?1", SUBSYSTEM=="block", - PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", - RESULT=="1ATA_VBOX_HARDDISK_VB88ee7178-f28aa887", - SYMLINK+="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
Following is the explanation of each parameter
- KERNEL==”sd?1″ – This matches the kernel name of the device. In our case all our partitions are having names as sd?1 (sdb1, sdc1 etc). So this match key matches the kernel name of the devices
- SUBSYSTEM==”block” – This match key matches the subsystem of the devices. SUBSYSTEM could be block, scsi, usb etc. We have all block devices.
- PROGRAM==”/usr/lib/udev/scsi_id -g -u -d /dev/$parent” – This will get the unique SCSI_ID for the device searched by first 2 match parameters (KERNEL and SUBSYSTEM)
- RESULT==”1ATA_VBOX_HARDDISK_VB544d069c-abd3901e” – This will match the output of PROGRAM command with RESULT. If the result matches, then further action will be taken
- SYMLINK+=”asm-disk1″ – This parameter is part of action key. If PROGRAM output matches RESULT, then a SYMLINK will be created, which is named asm-disk1 and will be pointing to the device in question.
- OWNER=”oracle” – This parameter is also part of action. This will change the ownership of device to oracle user
- GROUP=”dba” – This parameter changes the group of device to dba group
- MODE=”0660″ – This changes the permission of device file to 0660
So above rule means that the device pointing to the partition “sd*1” on the disk with the SCSI ID of “1ATA_VBOX_HARDDISK_VB88ee7178-f28aa887” will always be referred with symlink “/dev/asm-disk1” pointing to the device, regardless of the letter “?” Linux assigns when the device is discovered. In addition, the device will have the correct ownership and permissions for ASM.
We can create such rule for each of the device or if number of devices are huge, we can use wildcard and matching patterns to create more intelligent rules to search and take required actions.
I have created following rules and we can create a new rule file /etc/udev/rules.d/99-oracle-asm-disks.rules and put following rules, one for each device
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB544d069c-abd3901e", SYMLINK+="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB9a630cc5-d9697727", SYMLINK+="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB446fdf92-8640efff", SYMLINK+="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB3a71b4f2-8c603b78", SYMLINK+="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
Step 6) Load updated block device partition tables
You can use partprobe to load the partition tables for block devices
/sbin/partprobe /dev/sdb1 /sbin/partprobe /dev/sdc1 /sbin/partprobe /dev/sdd1 /sbin/partprobe /dev/sde1
Step 7) Test the rules
This is optional step to check if the rules are working as expected. You can run following commands to test the rules
udevadm test /block/sdb/sdb1 udevadm test /block/sdb/sdc1 udevadm test /block/sdb/sdd1 udevadm test /block/sdb/sde1
The output for one of the above command looks like following
calling: test version 219 This program is for debugging only, it does not run any program specified by a RUN key. It may show incorrect results, because some values may be different, or not available at a simulation run. === trie on-disk === tool version: 219 file size: 6984832 bytes header size 80 bytes strings 1805856 bytes nodes 5178896 bytes Load module index Created link configuration context. timestamp of '/etc/udev/rules.d' changed Reading rules file: /usr/lib/udev/rules.d/10-dm.rules Reading rules file: /usr/lib/udev/rules.d/100-balloon.rules Reading rules file: /usr/lib/udev/rules.d/13-dm-disk.rules Reading rules file: /usr/lib/udev/rules.d/40-redhat.rules Reading rules file: /usr/lib/udev/rules.d/42-usb-hid-pm.rules Reading rules file: /usr/lib/udev/rules.d/50-udev-default.rules Reading rules file: /usr/lib/udev/rules.d/60-alias-kmsg.rules Reading rules file: /usr/lib/udev/rules.d/60-cdrom_id.rules Reading rules file: /usr/lib/udev/rules.d/60-drm.rules Reading rules file: /usr/lib/udev/rules.d/60-keyboard.rules Reading rules file: /usr/lib/udev/rules.d/60-net.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-alsa.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-input.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-serial.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-storage-tape.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-storage.rules Reading rules file: /usr/lib/udev/rules.d/60-persistent-v4l.rules Reading rules file: /usr/lib/udev/rules.d/60-raw.rules Reading rules file: /usr/lib/udev/rules.d/61-accelerometer.rules Reading rules file: /usr/lib/udev/rules.d/64-btrfs.rules Reading rules file: /usr/lib/udev/rules.d/70-mouse.rules Reading rules file: /usr/lib/udev/rules.d/70-power-switch.rules Reading rules file: /usr/lib/udev/rules.d/70-touchpad.rules Reading rules file: /usr/lib/udev/rules.d/70-uaccess.rules Reading rules file: /usr/lib/udev/rules.d/71-biosdevname.rules Reading rules file: /usr/lib/udev/rules.d/71-seat.rules Reading rules file: /usr/lib/udev/rules.d/73-idrac.rules Reading rules file: /usr/lib/udev/rules.d/73-seat-late.rules Reading rules file: /usr/lib/udev/rules.d/75-net-description.rules Reading rules file: /usr/lib/udev/rules.d/75-probe_mtd.rules Reading rules file: /usr/lib/udev/rules.d/75-tty-description.rules Reading rules file: /usr/lib/udev/rules.d/78-sound-card.rules Reading rules file: /usr/lib/udev/rules.d/80-drivers.rules Reading rules file: /usr/lib/udev/rules.d/80-net-name-slot.rules Reading rules file: /usr/lib/udev/rules.d/80-net-setup-link.rules Reading rules file: /usr/lib/udev/rules.d/81-kvm-rhel.rules Reading rules file: /usr/lib/udev/rules.d/85-nm-unmanaged.rules Reading rules file: /usr/lib/udev/rules.d/90-alsa-tools-firmware.rules Reading rules file: /usr/lib/udev/rules.d/90-iprutils.rules Reading rules file: /usr/lib/udev/rules.d/90-vconsole.rules Reading rules file: /usr/lib/udev/rules.d/91-drm-modeset.rules Reading rules file: /usr/lib/udev/rules.d/95-dm-notify.rules Reading rules file: /usr/lib/udev/rules.d/95-udev-late.rules Reading rules file: /usr/lib/udev/rules.d/98-kexec.rules Reading rules file: /usr/lib/udev/rules.d/98-rdma.rules Reading rules file: /etc/udev/rules.d/99-oracle-asm-disks.rules Reading rules file: /usr/lib/udev/rules.d/99-systemd.rules rules contain 24576 bytes tokens (2048 * 12 bytes), 12216 bytes strings 1803 strings (22584 bytes), 1184 de-duplicated (10988 bytes), 620 trie nodes used GROUP 6 /usr/lib/udev/rules.d/50-udev-default.rules:52 LINK 'disk/by-id/ata-VBOX_HARDDISK_VB544d069c-abd3901e-part1' /usr/lib/udev/rules.d/60-persistent-storage.rules:43 IMPORT builtin 'blkid' /usr/lib/udev/rules.d/60-persistent-storage.rules:72 probe /dev/sdb1 raid offset=0 PROGRAM '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' /etc/udev/rules.d/99-oracle-asm-disks.rules:1 starting '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb'(out) '1ATA_VBOX_HARDDISK_VB544d069c-abd3901e' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' [10122] exit with return code 0 OWNER 54321 /etc/udev/rules.d/99-oracle-asm-disks.rules:1 GROUP 54322 /etc/udev/rules.d/99-oracle-asm-disks.rules:1 MODE 0660 /etc/udev/rules.d/99-oracle-asm-disks.rules:1 LINK 'asm-disk1' /etc/udev/rules.d/99-oracle-asm-disks.rules:1 PROGRAM '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' /etc/udev/rules.d/99-oracle-asm-disks.rules:2 starting '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb'(out) '1ATA_VBOX_HARDDISK_VB544d069c-abd3901e' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' [10123] exit with return code 0 PROGRAM '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' /etc/udev/rules.d/99-oracle-asm-disks.rules:3 starting '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb'(out) '1ATA_VBOX_HARDDISK_VB544d069c-abd3901e' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' [10124] exit with return code 0 PROGRAM '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' /etc/udev/rules.d/99-oracle-asm-disks.rules:4 starting '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb'(out) '1ATA_VBOX_HARDDISK_VB544d069c-abd3901e' '/usr/lib/udev/scsi_id -g -u -d /dev/sdb' [10125] exit with return code 0 handling device node '/dev/sdb1', devnum=b8:17, mode=0660, uid=54321, gid=54322 preserve permissions /dev/sdb1, 060660, uid=54321, gid=54322 preserve already existing symlink '/dev/block/8:17' to '../sdb1' found 'b8:17' claiming '/run/udev/links/\x2fasm-disk1' creating link '/dev/asm-disk1' to '/dev/sdb1' preserve already existing symlink '/dev/asm-disk1' to 'sdb1' found 'b8:17' claiming '/run/udev/links/\x2fdisk\x2fby-id\x2fata-VBOX_HARDDISK_VB544d069c-abd3901e-part1' creating link '/dev/disk/by-id/ata-VBOX_HARDDISK_VB544d069c-abd3901e-part1' to '/dev/sdb1' preserve already existing symlink '/dev/disk/by-id/ata-VBOX_HARDDISK_VB544d069c-abd3901e-part1' to '../../sdb1' created db file '/run/udev/data/b8:17' for '/block/sdb/sdb1' ACTION=add DEVLINKS=/dev/asm-disk1 /dev/disk/by-id/ata-VBOX_HARDDISK_VB544d069c-abd3901e-part1 DEVNAME=/dev/sdb1 DEVPATH=/block/sdb/sdb1 DEVTYPE=partition ID_ATA=1 ID_ATA_FEATURE_SET_PM=1 ID_ATA_FEATURE_SET_PM_ENABLED=1 ID_ATA_SATA=1 ID_ATA_SATA_SIGNAL_RATE_GEN2=1 ID_ATA_WRITE_CACHE=1 ID_ATA_WRITE_CACHE_ENABLED=1 ID_BUS=ata ID_MODEL=VBOX_HARDDISK ID_MODEL_ENC=VBOX\x20HARDDISK\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20 ID_PART_ENTRY_DISK=8:16 ID_PART_ENTRY_NUMBER=1 ID_PART_ENTRY_OFFSET=2048 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_SIZE=20969472 ID_PART_ENTRY_TYPE=0x83 ID_PART_TABLE_TYPE=dos ID_REVISION=1.0 ID_SERIAL=VBOX_HARDDISK_VB544d069c-abd3901e ID_SERIAL_SHORT=VB544d069c-abd3901e ID_TYPE=disk MAJOR=8 MINOR=17 SUBSYSTEM=block TAGS=:systemd: USEC_INITIALIZED=404668 Unload module index Unloaded link configuration context.
Step 8) Reload rules of udev
Use following command to reload the rules
udevadm control --reload-rules
Check if the required symlinks and other actions are performed on the devices or not
[root@advait ~]# ls -rlt /dev/asm* lrwxrwxrwx 1 root root 4 Jul 18 06:20 /dev/asm-disk2 -> sdc1 lrwxrwxrwx 1 root root 4 Jul 18 06:20 /dev/asm-disk3 -> sdd1 lrwxrwxrwx 1 root root 4 Jul 18 06:20 /dev/asm-disk4 -> sde1 lrwxrwxrwx 1 root root 4 Jul 18 06:20 /dev/asm-disk1 -> sdb1
Above symlinks are owned by root, but devices will be owned by oracle:dba
[root@advait ~]# ls -rlt /dev/sd?1 brw-rw---- 1 root disk 8, 1 Jul 18 05:36 /dev/sda1 brw-rw---- 1 oracle dba 8, 33 Jul 18 06:20 /dev/sdc1 brw-rw---- 1 oracle dba 8, 49 Jul 18 06:20 /dev/sdd1 brw-rw---- 1 oracle dba 8, 65 Jul 18 06:20 /dev/sde1 brw-rw---- 1 oracle dba 8, 17 Jul 18 06:20 /dev/sdb1
Now ASM can identify the disks as /dev/asm-disk* and these symlinks will persist with host reboot.
Hope this helps !!
References:
https://www.linux.com/news/udev-introduction-device-management-modern-linux-system
http://www.reactivated.net/writing_udev_rules.html
Filed under: Linux System Administration, Oracle ASM, Oracle Database 11g Tagged: asm disks, rhel7, scsi_id, udev Image may be NSFW.
Clik here to view.
Image may be NSFW.Clik here to view.
