Persistent block device naming

In the last couple of months or so, I have been playing with building a hadoop cluster and stuff. This also involved dealing with lots of JBODs and configuring them properly, which is pretty straight forward. To make a new disk mounted permanently, you generally add an entry in fstab, all good till now. So, my fstab entry looked something like below. See the HDFS mount points.

Fig. 1. Entries for the mounted points in the fstab file

The problem started when sometimes I would reboot my servers and some of them won’t come back and stuck at boot time while entering a safemode kind of a thing. Only a root user could then go in and troubleshoot. Most often than not, if I would reboot that server again, it rebooted back just fine. But this was annoying, really, when a bunch of servers won’t come back up every time I rebooted. I decided to do a little research into this.

Storage devices are usually managed by the sd drivers. That’s why you see the naming like sda, sdb, etc for the disks. The letter that comes after “sd” is referred to as a major number and the numeric number that comes after that, (e.g. sda”1″) is referred to as a minor number. Whenever the sd driver detects a device, it assigns an available major number and a minor number to it. What this also means is that when you reboot a server, the sd driver may detect some disk earlier than when it detected it the last time you rebooted that server, and therefore, it can assign it a different major and a minor number this time. Thus, what was sda1 last time may become sdb4 this time and the entries you have in fstab won’t work causing the system to not boot up properly. This can happen because of various reasons mentioned in [1].

For all these reasons, you really shouldn’t use sd names when referring to devices in your fstab file. There are several other ways to map the correct device using:

1. by-label

2. by-uuid

3. by-id and by-path

4. by-partlabel

5. by-partuuid

6. Static device names with Udev

I am not going to discuss all of them here except by-uuid, which is what I ended up using. If you issue an “lsblk -f” on your server, you would be able to see NAME, FSTYPE, LABEL, UUID, MOUNTPOINT. Note down the UUID for the concerned device and add this to your fstab file. It would look something like below.

Fig. 2. Entries for the mounted points in the fstab file using UUID

Now, reboot your server all you want, the UUIDs for the disks won’t change, ever, and you will have persistent device naming. This at least made me realize how small things like these can help save you so much of time and make sure you are not going crazy when sda1 becomes sdb4 on its own 😀

References:

1. RedHat 2017. “PERSISTENT NAMING”. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/persistent_naming. [Online;accessed 22-Nov-2017].