On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > > I'm following the instructions here: > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/... > > I've also added an extra virtual disk to my host node which appears as > /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. > Is udev broken in oVirt node? > > I cannot see anywhere in the dialog where you specify the name of the > device (eg. "/dev/sdb1"). So how's it supposed to work? > > It doesn't work, giving an information-free error message: > > Error while executing action Add Storage Connection: Problem while trying to mount target You can find more info on the failure in: /var/log/vdsm/supervdsmd.log

Posix compliant is basically NFS without some mount options: https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca972492... Using a local device on a host is less tested path, I'm not QE is testing this (Avihai, please correct me if you do). If you have multiple hosts, this will break if the local device does not have the same name on all hosts (so using /dev/sdb1 is very fragile). If you have one host it can be fine. Any reason to add a device to the vm, instead of using an NFS server? I guess that your purpose is testing virt-v2v with oVirt, so you want to test a common configuration; NFS is very common for oVirt users.

This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently. To avoid this issue with local devices, you need to blacklist the device. Add this file: $ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }

Configuring NFS on some other machine is easy. I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.

On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote: > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote: > > This is expected on oVirt, our multipath configuration is intentionally grabbing > > any device that multipath can work with, even if the device only has one path. > > The motivation is to be able to configure a system when only one path is > > available (maybe you have an hba/network/server issue), and once the other > > paths are available the system will use them transparently. > > > > To avoid this issue with local devices, you need to blacklist the device. > > > > Add this file: > > > > $ cat /etc/multipath/conf.d/local.conf > > blacklist { > > wwid "QEMU HARDDISK" > > } > > Thanks - for the mailing list record the syntax that worked for me is: > > # cat /etc/multipath/conf.d/local.conf > blacklist { > wwid ".*QEMU_HARDDISK.*" > } > > > Configuring NFS on some other machine is easy. > > > > I'm using another VM for this, so I can easily test negative flows like stopping > > or restarting the NFS server while it is being used by vms or storage > > operations. > > I'm using 2G alpine vm for this, it works fine even with 1G memory. > > I think I can get local storage working now (I had it working before). Well finally it fails with: 2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)

I think this feature (local storage) no longer works.

On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote: > On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > > > On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote: > > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote: > > > > This is expected on oVirt, our multipath configuration is intentionally grabbing > > > > any device that multipath can work with, even if the device only has one path. > > > > The motivation is to be able to configure a system when only one path is > > > > available (maybe you have an hba/network/server issue), and once the other > > > > paths are available the system will use them transparently. > > > > > > > > To avoid this issue with local devices, you need to blacklist the device. > > > > > > > > Add this file: > > > > > > > > $ cat /etc/multipath/conf.d/local.conf > > > > blacklist { > > > > wwid "QEMU HARDDISK" > > > > } > > > > > > Thanks - for the mailing list record the syntax that worked for me is: > > > > > > # cat /etc/multipath/conf.d/local.conf > > > blacklist { > > > wwid ".*QEMU_HARDDISK.*" > > > } > > > > > > > Configuring NFS on some other machine is easy. > > > > > > > > I'm using another VM for this, so I can easily test negative flows like stopping > > > > or restarting the NFS server while it is being used by vms or storage > > > > operations. > > > > I'm using 2G alpine vm for this, it works fine even with 1G memory. > > > > > > I think I can get local storage working now (I had it working before). > > > > Well finally it fails with: > > > > 2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16) > > The error "Could not initialize cluster lock" comes from vdsm. Usually > engine log is > not the best way to debug such failures. This is only the starting > point and you need to > go to the host and check vdsm and supervdsm logs in /var/log/vdsm/. I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages. > Since this error > comes from sanlock, we also may have useful info in /var/log/sanlock.log. Interesting: 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. I think it's quite likely that the sanlock daemon does not have access here, since (see below) I choown'd the root of the xfs filesystem to 36:36 (otherwise vdsm complains). > Can you share instructions on how to reproduce this issue? I have one engine and one node (both happen to be VMs, but I don't believe that is relevant here). It's running Version 4.4.10.6-1.el8. I added a second disk to the node, and disabled multipath as previously discussed. The second disk is /dev/sdb1. I formatted it as xfs and chowned the root of the filesystem to 36:36.

In the admin portal, Storage -> Domains -> New domain Storage type: Posix compliant fs Name: ovirt-data Path: /dev/sdb1 VFS type: xfs Hit OK -> Error while executing action AddPosixFsStorageDomain: Unexpected exception

Can be fixed with: $ sudo chcon -R -t nfs_t mnt

On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote: > Can be fixed with: > > $ sudo chcon -R -t nfs_t mnt Yes that did work, thanks.

On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote: > On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > > > On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote: > > > On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote: > > > > This is expected on oVirt, our multipath configuration is intentionally grabbing > > > > any device that multipath can work with, even if the device only has one path. > > > > The motivation is to be able to configure a system when only one path is > > > > available (maybe you have an hba/network/server issue), and once the other > > > > paths are available the system will use them transparently. > > > > > > > > To avoid this issue with local devices, you need to blacklist the device. > > > > > > > > Add this file: > > > > > > > > $ cat /etc/multipath/conf.d/local.conf > > > > blacklist { > > > > wwid "QEMU HARDDISK" > > > > } > > > > > > Thanks - for the mailing list record the syntax that worked for me is: > > > > > > # cat /etc/multipath/conf.d/local.conf > > > blacklist { > > > wwid ".*QEMU_HARDDISK.*" > > > } > > > > > > > Configuring NFS on some other machine is easy. > > > > > > > > I'm using another VM for this, so I can easily test negative flows like stopping > > > > or restarting the NFS server while it is being used by vms or storage > > > > operations. > > > > I'm using 2G alpine vm for this, it works fine even with 1G memory. > > > > > > I think I can get local storage working now (I had it working before). > > > > Well finally it fails with: > > > > 2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16) > > The error "Could not initialize cluster lock" comes from vdsm. Usually > engine log is > not the best way to debug such failures. This is only the starting > point and you need to > go to the host and check vdsm and supervdsm logs in /var/log/vdsm/. I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages. > Since this error > comes from sanlock, we also may have useful info in /var/log/sanlock.log. Interesting: 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. I think it's quite likely that the sanlock daemon does not have access here, since (see below) I choown'd the root of the xfs filesystem to 36:36 (otherwise vdsm complains). > Can you share instructions on how to reproduce this issue? I have one engine and one node (both happen to be VMs, but I don't believe that is relevant here). It's running Version 4.4.10.6-1.el8. I added a second disk to the node, and disabled multipath as previously discussed. The second disk is /dev/sdb1. I formatted it as xfs and chowned the root of the filesystem to 36:36. In the admin portal, Storage -> Domains -> New domain Storage type: Posix compliant fs Name: ovirt-data Path: /dev/sdb1 VFS type: xfs Hit OK -> Error while executing action AddPosixFsStorageDomain: Unexpected exception