I'm following the instructions here: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm... I've also added an extra virtual disk to my host node which appears as /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. Is udev broken in oVirt node? I cannot see anywhere in the dialog where you specify the name of the device (eg. "/dev/sdb1"). So how's it supposed to work? It doesn't work, giving an information-free error message: Error while executing action Add Storage Connection: Problem while trying to mount target Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones <rjones@redhat.com> wrote:
I'm following the instructions here: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
I've also added an extra virtual disk to my host node which appears as /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. Is udev broken in oVirt node?
I cannot see anywhere in the dialog where you specify the name of the device (eg. "/dev/sdb1"). So how's it supposed to work?
It doesn't work, giving an information-free error message:
Error while executing action Add Storage Connection: Problem while trying to mount target
You can find more info on the failure in: /var/log/vdsm/supervdsmd.log Posix compliant is basically NFS without some mount options: https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/... Using a local device on a host is less tested path, I'm not QE is testing this (Avihai, please correct me if you do). If you have multiple hosts, this will break if the local device does not have the same name on all hosts (so using /dev/sdb1 is very fragile). If you have one host it can be fine. Any reason to add a device to the vm, instead of using an NFS server? I guess that your purpose is testing virt-v2v with oVirt, so you want to test a common configuration; NFS is very common for oVirt users. Nir
On Thu, Feb 03, 2022 at 03:07:20PM +0200, Nir Soffer wrote:
On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones <rjones@redhat.com> wrote:
I'm following the instructions here: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
I've also added an extra virtual disk to my host node which appears as /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. Is udev broken in oVirt node?
I cannot see anywhere in the dialog where you specify the name of the device (eg. "/dev/sdb1"). So how's it supposed to work?
It doesn't work, giving an information-free error message:
Error while executing action Add Storage Connection: Problem while trying to mount target
You can find more info on the failure in: /var/log/vdsm/supervdsmd.log
vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'xfs', '/srv', '/rhev/data-center/mnt/_srv'] failed with rc=32 out=b'' err=b'mount: /rhev/data-center/mnt/_srv: /srv is not a block device.\n' I suppose it expects the name of the block device (ie. /dev/sdb) rather than the mount point there. It also turns out the new device has been "captured" by multipathd: # multipath -ll 0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1 dm-0 QEMU,QEMU HARDDISK size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 0:0:0:1 sdb 8:16 active ready running I've so far not found a way to disable multipathd effectively. Even stopping and disabling the service and rebooting doesn't help so I guess something starts it up.
Posix compliant is basically NFS without some mount options: https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/...
Using a local device on a host is less tested path, I'm not QE is testing this (Avihai, please correct me if you do).
If you have multiple hosts, this will break if the local device does not have the same name on all hosts (so using /dev/sdb1 is very fragile). If you have one host it can be fine.
Any reason to add a device to the vm, instead of using an NFS server?
I guess that your purpose is testing virt-v2v with oVirt, so you want to test a common configuration; NFS is very common for oVirt users.
I don't have an NFS server to use for this. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Thu, Feb 3, 2022 at 3:42 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Feb 03, 2022 at 03:07:20PM +0200, Nir Soffer wrote:
On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones <rjones@redhat.com> wrote:
I'm following the instructions here: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
I've also added an extra virtual disk to my host node which appears as /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. Is udev broken in oVirt node?
I cannot see anywhere in the dialog where you specify the name of the device (eg. "/dev/sdb1"). So how's it supposed to work?
It doesn't work, giving an information-free error message:
Error while executing action Add Storage Connection: Problem while trying to mount target
You can find more info on the failure in: /var/log/vdsm/supervdsmd.log
vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'xfs', '/srv', '/rhev/data-center/mnt/_srv'] failed with rc=32 out=b'' err=b'mount: /rhev/data-center/mnt/_srv: /srv is not a block device.\n'
I suppose it expects the name of the block device (ie. /dev/sdb) rather than the mount point there.
It also turns out the new device has been "captured" by multipathd:
# multipath -ll 0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1 dm-0 QEMU,QEMU HARDDISK size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 0:0:0:1 sdb 8:16 active ready running
I've so far not found a way to disable multipathd effectively. Even stopping and disabling the service and rebooting doesn't help so I guess something starts it up.
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently. To avoid this issue with local devices, you need to blacklist the device. Add this file: $ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" } And run (as root): multipathd reconfigure At this point lsblk will show the expected /dev/sdb1 and multipath will never use this device again. Adding a serial to the device in libvirt xml will make it easier to blacklist.
Posix compliant is basically NFS without some mount options: https://github.com/oVirt/vdsm/blob/878407297cb7dc892110ae5d6b0403ca97249247/...
Using a local device on a host is less tested path, I'm not QE is testing this (Avihai, please correct me if you do).
If you have multiple hosts, this will break if the local device does not have the same name on all hosts (so using /dev/sdb1 is very fragile). If you have one host it can be fine.
Any reason to add a device to the vm, instead of using an NFS server?
I guess that your purpose is testing virt-v2v with oVirt, so you want to test a common configuration; NFS is very common for oVirt users.
I don't have an NFS server to use for this.
Configuring NFS on some other machine is easy. I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory. Nir
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is: # cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with: 2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16) I think this feature (local storage) no longer works. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually engine log is not the best way to debug such failures. This is only the starting point and you need to go to the host and check vdsm and supervdsm logs in /var/log/vdsm/. Since this error comes from sanlock, we also may have useful info in /var/log/sanlock.log. Can you share instructions on how to reproduce this issue?
I think this feature (local storage) no longer works.
This is not local storage, local storage is a storage domain using a local directory on a host. This works only when creating a local data center, basically each host has its own data center. Nir
On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually engine log is not the best way to debug such failures. This is only the starting point and you need to go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages.
Since this error comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Interesting: 2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. I think it's quite likely that the sanlock daemon does not have access here, since (see below) I choown'd the root of the xfs filesystem to 36:36 (otherwise vdsm complains).
Can you share instructions on how to reproduce this issue?
I have one engine and one node (both happen to be VMs, but I don't believe that is relevant here). It's running Version 4.4.10.6-1.el8. I added a second disk to the node, and disabled multipath as previously discussed. The second disk is /dev/sdb1. I formatted it as xfs and chowned the root of the filesystem to 36:36. In the admin portal, Storage -> Domains -> New domain Storage type: Posix compliant fs Name: ovirt-data Path: /dev/sdb1 VFS type: xfs Hit OK -> Error while executing action AddPosixFsStorageDomain: Unexpected exception
I think this feature (local storage) no longer works.
This is not local storage, local storage is a storage domain using a local directory on a host. This works only when creating a local data center, basically each host has its own data center.
Nir
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually engine log is not the best way to debug such failures. This is only the starting point and you need to go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages.
Since this error comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Interesting:
2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file.
I think it's quite likely that the sanlock daemon does not have access here, since (see below) I choown'd the root of the xfs filesystem to 36:36 (otherwise vdsm complains).
Can you share instructions on how to reproduce this issue?
I have one engine and one node (both happen to be VMs, but I don't believe that is relevant here). It's running Version 4.4.10.6-1.el8.
I added a second disk to the node, and disabled multipath as previously discussed. The second disk is /dev/sdb1. I formatted it as xfs and chowned the root of the filesystem to 36:36.
Looks right, forgetting to change ownership is a common mistake.
In the admin portal, Storage -> Domains -> New domain
Storage type: Posix compliant fs
Name: ovirt-data
Path: /dev/sdb1
VFS type: xfs
Hit OK -> Error while executing action AddPosixFsStorageDomain: Unexpected exception
I reproduce with vdsm-4.50.0.5-1.el8.x86_64. on RHEL 8.6 nightly. Can you file a oVirt/vdsm bug for this?
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually engine log is not the best way to debug such failures. This is only the starting point and you need to go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages.
Since this error comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Interesting:
2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file.
The issue is selinux: NFS domain: $ ls -lhZ /rhev/data-center/mnt/alpine\:_01/e9467633-ee31-4e15-b3f8-3812b374c764/dom_md/ total 2.3M -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.0M Feb 4 15:32 ids -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Jan 20 23:53 inbox -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 2.0M Jan 20 23:54 leases -rw-r--r--. 1 vdsm kvm system_u:object_r:nfs_t:s0 354 Jan 20 23:54 metadata -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Jan 20 23:53 outbox -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.3M Jan 20 23:53 xleases The posix domain mount (mounted manually): $ ls -lhZ mnt/689c22c4-e264-4873-aa75-1aa4970d4366/dom_md/ total 252K -rw-rw----. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 0 Feb 4 15:23 ids -rw-rw----. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 16M Feb 4 15:23 inbox -rw-rw----. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 0 Feb 4 15:23 leases -rw-r--r--. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 316 Feb 4 15:23 metadata -rw-rw----. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 16M Feb 4 15:23 outbox -rw-rw----. 1 vdsm kvm system_u:object_r:unlabeled_t:s0 1.3M Feb 4 15:23 xleases Can be fixed with: $ sudo chcon -R -t nfs_t mnt $ ls -lhZ mnt/689c22c4-e264-4873-aa75-1aa4970d4366/dom_md/ total 252K -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 0 Feb 4 15:23 ids -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Feb 4 15:23 inbox -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 0 Feb 4 15:23 leases -rw-r--r--. 1 vdsm kvm system_u:object_r:nfs_t:s0 316 Feb 4 15:23 metadata -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 16M Feb 4 15:23 outbox -rw-rw----. 1 vdsm kvm system_u:object_r:nfs_t:s0 1.3M Feb 4 15:23 xleases After this change delete the storage domain directory: $ rm -rf mnt/689c22c4-e264-4873-aa75-1aa4970d4366 Since vdsm will refuse to create a storage domain in a non-empty mount, and recreate the storage domain in engine. Works for me with 4.5: $ mount | grep /dev/sda1 /dev/sda1 on /rhev/data-center/mnt/_dev_sda1 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) Nir
On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote:
Can be fixed with:
$ sudo chcon -R -t nfs_t mnt
Yes that did work, thanks. Is this still a bug? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote:
Can be fixed with:
$ sudo chcon -R -t nfs_t mnt
Yes that did work, thanks.
Is this still a bug?
For NFS this works out of the box - I don't think you need to relebal anything manually. As a user I would expect that mounting posix file system will also work out of the box. For example vdsm ca relabel the mount point when creating a storage domain. Also I'm not sure using nfs_t is the right label to use, we better discuss this with selinux folks. So I think this issue is worth a bug, in the worst case we will close it, and the bug will also document the workaround. Nir
On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote:
Can be fixed with:
$ sudo chcon -R -t nfs_t mnt
Yes that did work, thanks.
Warning: that this configuration is a trap - if you add another host to this system, the system will try to mount the same device (/dev/sdb1) on the new host. Since the other host does not have the same disk, the mount will fail, and the other host will be deactivated, since it cannot access all storage. Nir
On Fri, Feb 04, 2022 at 04:02:56PM +0200, Nir Soffer wrote:
On Fri, Feb 4, 2022 at 3:49 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:47:11PM +0200, Nir Soffer wrote:
Can be fixed with:
$ sudo chcon -R -t nfs_t mnt
Yes that did work, thanks.
Warning: that this configuration is a trap - if you add another host to this system, the system will try to mount the same device (/dev/sdb1) on the new host. Since the other host does not have the same disk, the mount will fail, and the other host will be deactivated, since it cannot access all storage.
Point taken! This is only a test however. Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=2050721 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Fri, Feb 4, 2022 at 3:18 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 03:09:02PM +0200, Nir Soffer wrote:
On Fri, Feb 4, 2022 at 11:16 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Fri, Feb 04, 2022 at 08:42:08AM +0000, Richard W.M. Jones wrote:
On Thu, Feb 03, 2022 at 06:31:52PM +0200, Nir Soffer wrote:
This is expected on oVirt, our multipath configuration is intentionally grabbing any device that multipath can work with, even if the device only has one path. The motivation is to be able to configure a system when only one path is available (maybe you have an hba/network/server issue), and once the other paths are available the system will use them transparently.
To avoid this issue with local devices, you need to blacklist the device.
Add this file:
$ cat /etc/multipath/conf.d/local.conf blacklist { wwid "QEMU HARDDISK" }
Thanks - for the mailing list record the syntax that worked for me is:
# cat /etc/multipath/conf.d/local.conf blacklist { wwid ".*QEMU_HARDDISK.*" }
Configuring NFS on some other machine is easy.
I'm using another VM for this, so I can easily test negative flows like stopping or restarting the NFS server while it is being used by vms or storage operations. I'm using 2G alpine vm for this, it works fine even with 1G memory.
I think I can get local storage working now (I had it working before).
Well finally it fails with:
2022-02-04 09:14:55,779Z ERROR [org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand] (default task-2) [25a32edf] Command 'org.ovirt.engine.core.bll.storage.domain.AddPosixFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Could not initialize cluster lock: (), code = 701 (Failed with error unexpected and code 16)
The error "Could not initialize cluster lock" comes from vdsm. Usually engine log is not the best way to debug such failures. This is only the starting point and you need to go to the host and check vdsm and supervdsm logs in /var/log/vdsm/.
I can't really see anything relevant in supervdsm.log, it's all fairly neutral debug messages.
Since this error comes from sanlock, we also may have useful info in /var/log/sanlock.log.
Interesting:
2022-02-04 13:15:27 16723 [826]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/_dev_sdb1/13a731d2-e1d2-4998-9b02-ac46899e3159/dom_md/ids 2022-02-04 13:15:27 16723 [826]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file.
I think it's quite likely that the sanlock daemon does not have access here, since (see below) I choown'd the root of the xfs filesystem to 36:36 (otherwise vdsm complains).
Can you share instructions on how to reproduce this issue?
I have one engine and one node (both happen to be VMs, but I don't believe that is relevant here). It's running Version 4.4.10.6-1.el8.
I added a second disk to the node, and disabled multipath as previously discussed. The second disk is /dev/sdb1. I formatted it as xfs and chowned the root of the filesystem to 36:36.
In the admin portal, Storage -> Domains -> New domain
Storage type: Posix compliant fs
Name: ovirt-data
Path: /dev/sdb1
VFS type: xfs
Hit OK -> Error while executing action AddPosixFsStorageDomain: Unexpected exception
Avihai, are we testing this configuration? I'm not sure this is useful for real users - you need to have the same device available on multipath hosts, backed up by a shared file system, like GFS2. [1] https://en.wikipedia.org/wiki/GFS2 Nir
On Thu, Feb 3, 2022 at 1:30 PM Richard W.M. Jones <rjones@redhat.com> wrote:
I'm following the instructions here:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
I've also added an extra virtual disk to my host node which appears as /dev/sdb. Although the disk is partitioned, /dev/sdb1 is not created. Is udev broken in oVirt node?
As you write the term "local", if you plan to have only one host and not add any other in the future for this environment, you had better creating a DC/Cluster of type "Local" (versus "Shared") and then a storage domain of type "Local on Host", where you give it the path where you mounted your additional disk (with correct permissions). I've been using it for many months on one environment with only one very big server (1,5Tb of ram and 2 NVME disks, each 2Tb in size), currently with 36 VMs running without problem. Now at 4.4.10 async 2. The host is now based on Rocky Linux 8.5 (previously CentOS 8.4) with a standalone engine, also that one now based on Rocky Linux 8.5) that is a VM on an external vSphere environment. This same environment previously was based on a single host gluster configuration and I converted it to this configuration without loss of VMs/data (I detached and then imported the previously gluster based fs). HIH, Gianluca
participants (3)
-
Gianluca Cecchi -
Nir Soffer -
Richard W.M. Jones