Hi,

I have configured a single host HCI environment through the GUI wizard in 4.3.9.

Initial setup has thai layout of disks, as seen by the operating system:

/dev/sda -> for ovirt-node-ng OS

/dev/nvme0n1 --> for gluster, engine and data volumes

/dev/nvme1n1 --> for gluster, vmstore volume

So far so good and all is ok.

I notice that, even with single path internal disks, at the end oVirt configures the gluster disks as multipath devices and LVM2 PV structure on top of the multipath devices.

Is this for "code optimization" at low level or what is the rationale for that, as with Gluster normally you do use local disks and so single path?

Multipath structure generated:

[root@ovirt ~]# multipath -l
nvme.8086-50484b53373530353031325233373541474e-494e54454c205353 dm-5 NVME,INTEL SSDPED1K375GA                    
size=349G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 0:0:1:0 nvme0n1 259:0 active undef running
eui.01000000010000005cd2e4b5e7db4d51 dm-6 NVME,INTEL SSDPEDKX040T7                    
size=932G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 2:0:1:0 nvme1n1 259:2 active undef running
[root@ovirt ~]#

Anyway, on top of the multipah devices

On /dev/nvme0n1:

gluster_vg_nvme0n1 volume group with gluster_lv_data and gluster_lv_engine

On /dev/nvme1n1

gluster_vg_nvme1n1 volume group with gluster_lv_vmstore logical volume

The problem arises when I add another nvme disk, that, occupying a PCI slot, it seems has always higher priority of the previous /dev/nvme1n1 disk and so takes its name.

After booting the node:

old nvme0n1 --> unmodified name

old nvme1n1 --> becomes nvme2n1

new disk --> gets name nvme1n1

From a funcional point of view I have no problems apart LVM warnings I send below and also because the xfs entries in fstab are with UUID:

UUID=fa5dd3cb-aeef-470e-b982-432ac896d87a /gluster_bricks/engine xfs inode64,noatime,nodiratime 0 0
UUID=43bed7de-66b1-491d-8055-5b4ef9b0482f /gluster_bricks/data xfs inode64,noatime,nodiratime 0 0
UUID=b81a491c-0a4c-4c11-89d8-9db7fe82888e /gluster_bricks/vmstore xfs inode64,noatime,nodiratime 0 0

lvs commands get:

[root@ovirt ~]# lvs
  WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp.
  WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl.
  WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV.
  WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV.
  LV                                  VG                 Attr       LSize   Pool                                Origin    

...

Or, for the old nvme1n1 disk, now nvme2n1 multipath device:

[root@ovirt ~]# pvdisplay /dev/mapper/eui.01000000010000005cd2e4e359284f51
  WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp.
  WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl.
  WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV.
  WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV.
  --- Physical volume ---
  PV Name               /dev/mapper/eui.01000000010000005cd2e4e359284f51
  VG Name               gluster_vg_nvme1n1
  PV Size               931.51 GiB / not usable 1.71 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              238467
  Free PE               0
  Allocated PE          238467
  PV UUID               O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl
   
[root@ovirt ~]#

I'm able to create PV on top of the new multipath device detected by the system (see the nvme1n1 of the underlying disk):

eui.01000000010000005cd2e4b5e7db4d51 dm-6 NVME,INTEL SSDPEDKX040T7          
size=3.6T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 1:0:1:0 nvme1n1 259:1 active undef running

[root@ovirt ~]# pvcreate --dataalignment 256K /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51
  WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp.
  WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl.
  WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV.
  WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV.
  Physical volume "/dev/mapper/eui.01000000010000005cd2e4b5e7db4d51" successfully created.
[root@ovirt ~]#

But then I'm unable to create a VG on top of it:

[root@ovirt ~]# vgcreate gluster_vg_4t /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51
  WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp.
  WARNING: Not using device /dev/nvme1n1 for PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9.
  WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl.
  WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because device is used by LV.
  WARNING: PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9 prefers device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 because device is in dm subsystem.
  WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because device is used by LV.
  WARNING: Not using device /dev/nvme0n1 for PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp.
  WARNING: Not using device /dev/nvme1n1 for PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9.
  WARNING: Not using device /dev/nvme2n1 for PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl.
  WARNING: PV eYfuXw-yaPd-cMUE-0dnA-tVON-uZ9g-5x4BDp prefers device /dev/mapper/nvme.8086-50484b53373530353031325233373541474e-494e54454c20535344504544314b3337354741-00000001 because of previous preference.
  WARNING: PV 56ON99-hFFP-cGpZ-g4MX-GfjW-jXeE-fKZVG9 prefers device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 because of previous preference.
  WARNING: PV O43LFq-46Gc-RRgS-Sk1F-5mFZ-Qw4n-oxXgJl prefers device /dev/mapper/eui.01000000010000005cd2e4e359284f51 because of previous preference.
  Cannot use device /dev/mapper/eui.01000000010000005cd2e4b5e7db4d51 with duplicates.
[root@ovirt ~]#

the same with the "-f" option.

I suspect I can solve the problem filtering out /dev/nvme* devices in lvm.conf, but I'm not sure. The OS disk is seen as sda so it should not have problems with this

Something like this:

filter = [ "r|/dev/nvme|", "a|.*/|" ]

And also I am not sure if I have to rebuild the initrd or not in this case and if so what would be the exact sequence of commands to execute.

Any suggestions?

Thanks in advance,

Gianluca