On 08/30/2016 04:47 PM, Nir Soffer wrote:

On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

While rebooting one of the hosts in an oVirt cluster, I noticed that thin_check is run on the thin pool devices of one of the VM's on which the disk is assigned to.

That seems strange to me. I would expect the host to stay clear of any VM disks.

We expect the same thing, but unfortunately systemd and lvm try to auto activate stuff. This may be good idea for desktop system, but probably bad idea for a server and in particular a hypervisor.

We don't have a solution yet, but you can try these:

1. disable lvmetad service

systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket

Edit /etc/lvm/lvm.conf:

use_lvmetad = 0

2. disable lvm auto activation

Edit /etc/lvm/lvm.conf:

auto_activation_volume_list = []

3. both 1 and 2

I've now applied both of the above and regenerated the initramfs and rebooted and the host no longer lists the LV's of the VM. Since I rebooted the host before without this issue, I'm not sure a single reboot is enough to conclude it has fully fixed the issue. You mention that there's no solution yet. Does that mean the above settings are not 100% certain to avoid this behaviour? I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only include the PV's for the hypervisor disks (on which the OS is installed) so the system lvm commands only touches those. Since vdsm is using its own lvm.conf this should be OK for vdsm? Regards, Rik -- Rik Theys System Engineer KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee +32(0)16/32.11.07 ---------------------------------------------------------------- <<Any errors in spelling, tact or fact are transmission errors>>

On Wed, Aug 31, 2016 at 11:07 AM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

On 08/31/2016 09:43 AM, Rik Theys wrote:

...

On 08/30/2016 04:47 PM, Nir Soffer wrote:

...

On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

While rebooting one of the hosts in an oVirt cluster, I noticed that thin_check is run on the thin pool devices of one of the VM's on which the disk is assigned to.

That seems strange to me. I would expect the host to stay clear of any VM disks.

We expect the same thing, but unfortunately systemd and lvm try to auto activate stuff. This may be good idea for desktop system, but probably bad idea for a server and in particular a hypervisor.

We don't have a solution yet, but you can try these:

1. disable lvmetad service

systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket

Edit /etc/lvm/lvm.conf:

use_lvmetad = 0

2. disable lvm auto activation

Edit /etc/lvm/lvm.conf:

auto_activation_volume_list = []

3. both 1 and 2

I've now applied both of the above and regenerated the initramfs and rebooted and the host no longer lists the LV's of the VM. Since I rebooted the host before without this issue, I'm not sure a single reboot is enough to conclude it has fully fixed the issue.

You mention that there's no solution yet. Does that mean the above settings are not 100% certain to avoid this behaviour?

I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only include the PV's for the hypervisor disks (on which the OS is installed) so the system lvm commands only touches those. Since vdsm is using its own lvm.conf this should be OK for vdsm?

This does not seem to work. The host can not be activated as it can't find his volume group(s). To be able to use the global_filter in /etc/lvm/lvm.conf, I believe it should be overridden in vdsm's lvm.conf to revert back to the default.

I've moved my filter from global_filter to filter and that seems to work. When lvmetad is disabled I believe this should have the same effect as global_filter? The comments in /etc/lvm/lvm.conf indicate also udev might ignore the filter setting?

Right, global_filter exist so you can override filter used from the command line. For example, hiding certain devices from vdsm. This is why we are using filter in vdsm, leaving global_filter for the administrator. Can you explain why do you need global_filter or filter for the hypervisor disks? Do you have any issue with the current settings, disabling auto activation and lvmetad? Nir

On Wed, Aug 31, 2016 at 2:30 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

Hi,

On 08/31/2016 11:51 AM, Nir Soffer wrote:

...

On Wed, Aug 31, 2016 at 11:07 AM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

On 08/31/2016 09:43 AM, Rik Theys wrote:

...

On 08/30/2016 04:47 PM, Nir Soffer wrote:

...

On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

While rebooting one of the hosts in an oVirt cluster, I noticed that thin_check is run on the thin pool devices of one of the VM's on which the disk is assigned to.

That seems strange to me. I would expect the host to stay clear of any VM disks.

We expect the same thing, but unfortunately systemd and lvm try to auto activate stuff. This may be good idea for desktop system, but probably bad idea for a server and in particular a hypervisor.

We don't have a solution yet, but you can try these:

1. disable lvmetad service

systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket

Edit /etc/lvm/lvm.conf:

use_lvmetad = 0

2. disable lvm auto activation

Edit /etc/lvm/lvm.conf:

auto_activation_volume_list = []

3. both 1 and 2

I've now applied both of the above and regenerated the initramfs and rebooted and the host no longer lists the LV's of the VM. Since I rebooted the host before without this issue, I'm not sure a single reboot is enough to conclude it has fully fixed the issue.

You mention that there's no solution yet. Does that mean the above settings are not 100% certain to avoid this behaviour?

I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only include the PV's for the hypervisor disks (on which the OS is installed) so the system lvm commands only touches those. Since vdsm is using its own lvm.conf this should be OK for vdsm?

This does not seem to work. The host can not be activated as it can't find his volume group(s). To be able to use the global_filter in /etc/lvm/lvm.conf, I believe it should be overridden in vdsm's lvm.conf to revert back to the default.

I've moved my filter from global_filter to filter and that seems to work. When lvmetad is disabled I believe this should have the same effect as global_filter? The comments in /etc/lvm/lvm.conf indicate also udev might ignore the filter setting?

Right, global_filter exist so you can override filter used from the command line.

For example, hiding certain devices from vdsm. This is why we are using filter in vdsm, leaving global_filter for the administrator.

Can you explain why do you need global_filter or filter for the hypervisor disks?

Based on the comment in /etc/lvm/lvm.conf regarding global_filter I concluded that not only lvmetad but also udev might perform action on the devices and I wanted to prevent that.

I've now set the following settings in /etc/lvm/lvm.conf:

use_lvmetad = 0 auto_activation_volume_list = [] filter = ["a|/dev/sda5|", "r|.*|" ]

Better use /dev/disk/by-uuid/ to select the specific device, without depending on device order.

On other systems I have kept the default filter.

...

Do you have any issue with the current settings, disabling auto activation and lvmetad?

Keeping those two disabled also seems to work. The ovirt LV's do show up in 'lvs' output but are not activated.

Good

I wanted to be absolutely sure the VM LV's were not touched, I added the filter on some of our hosts.

The only problem with this filter is it may break if you change the host in some way, like boot from another disk. It would be nice if you file a bug for this, and mention the configuration that fixes this issue, we certainly need to improve the way we configure lvm. Nir

On Wed, Aug 31, 2016 at 3:15 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

Hi,

On 08/31/2016 02:04 PM, Nir Soffer wrote:

...

On Wed, Aug 31, 2016 at 2:30 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

Hi,

On 08/31/2016 11:51 AM, Nir Soffer wrote:

...

On Wed, Aug 31, 2016 at 11:07 AM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote:

...

On 08/31/2016 09:43 AM, Rik Theys wrote:

...

On 08/30/2016 04:47 PM, Nir Soffer wrote: > On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys@esat.kuleuven.be> wrote: >> While rebooting one of the hosts in an oVirt cluster, I noticed that >> thin_check is run on the thin pool devices of one of the VM's on which >> the disk is assigned to. >> >> That seems strange to me. I would expect the host to stay clear of any >> VM disks. > > We expect the same thing, but unfortunately systemd and lvm try to > auto activate stuff. This may be good idea for desktop system, but > probably bad idea for a server and in particular a hypervisor. > > We don't have a solution yet, but you can try these: > > 1. disable lvmetad service > > systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket > systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket > > Edit /etc/lvm/lvm.conf: > > use_lvmetad = 0 > > 2. disable lvm auto activation > > Edit /etc/lvm/lvm.conf: > > auto_activation_volume_list = [] > > 3. both 1 and 2 >

I've now applied both of the above and regenerated the initramfs and rebooted and the host no longer lists the LV's of the VM. Since I rebooted the host before without this issue, I'm not sure a single reboot is enough to conclude it has fully fixed the issue.

You mention that there's no solution yet. Does that mean the above settings are not 100% certain to avoid this behaviour?

I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only include the PV's for the hypervisor disks (on which the OS is installed) so the system lvm commands only touches those. Since vdsm is using its own lvm.conf this should be OK for vdsm?

This does not seem to work. The host can not be activated as it can't find his volume group(s). To be able to use the global_filter in /etc/lvm/lvm.conf, I believe it should be overridden in vdsm's lvm.conf to revert back to the default.

I've moved my filter from global_filter to filter and that seems to work. When lvmetad is disabled I believe this should have the same effect as global_filter? The comments in /etc/lvm/lvm.conf indicate also udev might ignore the filter setting?

Right, global_filter exist so you can override filter used from the command line.

For example, hiding certain devices from vdsm. This is why we are using filter in vdsm, leaving global_filter for the administrator.

Can you explain why do you need global_filter or filter for the hypervisor disks?

Based on the comment in /etc/lvm/lvm.conf regarding global_filter I concluded that not only lvmetad but also udev might perform action on the devices and I wanted to prevent that.

I've now set the following settings in /etc/lvm/lvm.conf:

use_lvmetad = 0 auto_activation_volume_list = [] filter = ["a|/dev/sda5|", "r|.*|" ]

Better use /dev/disk/by-uuid/ to select the specific device, without depending on device order.

...

On other systems I have kept the default filter.

...

Do you have any issue with the current settings, disabling auto activation and lvmetad?

Keeping those two disabled also seems to work. The ovirt LV's do show up in 'lvs' output but are not activated.

Good

When vdsm runs the lvchange command to activate the LV of a VM (so it can boot it), will LVM still try to scan the new LV for PV's (and thin pools, etc)? Is this also prevented by the auto_activation_volume_list parameter in this case?

I don't know, but I don't see why lvm would scan the lv for pv and thin pools, this should happen on on the guest. We never had reports about such issue.

...

...

I wanted to be absolutely sure the VM LV's were not touched, I added the filter on some of our hosts.

The only problem with this filter is it may break if you change the host in some way, like boot from another disk.

I am aware of this issue, but in this case I would rather mess with a hypervisor that no longer boots because it needs an updated lvm.conf than try to fix corrupted VM's because the host was accessing the disks of the VM while it was running on another node.

...

It would be nice if you file a bug for this, and mention the configuration that fixes this issue, we certainly need to improve the way we configure lvm.

Against which component of oVirt should I file this bug?

vdsm, Core, Storage team Thanks, Nir