[ovirt-users] thin_check run on VM disk by host on startup ?!

[Nir Soffer]

On Wed, Aug 31, 2016 at 3:15 PM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
> Hi,
>
> On 08/31/2016 02:04 PM, Nir Soffer wrote:
>> On Wed, Aug 31, 2016 at 2:30 PM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>> Hi,
>>>
>>> On 08/31/2016 11:51 AM, Nir Soffer wrote:
>>>> On Wed, Aug 31, 2016 at 11:07 AM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>>>> On 08/31/2016 09:43 AM, Rik Theys wrote:
>>>>>> On 08/30/2016 04:47 PM, Nir Soffer wrote:
>>>>>>> On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>>>>>>> While rebooting one of the hosts in an oVirt cluster, I noticed that
>>>>>>>> thin_check is run on the thin pool devices of one of the VM's on which
>>>>>>>> the disk is assigned to.
>>>>>>>>
>>>>>>>> That seems strange to me. I would expect the host to stay clear of any
>>>>>>>> VM disks.
>>>>>>>
>>>>>>> We expect the same thing, but unfortunately systemd and lvm try to
>>>>>>> auto activate stuff. This may be good idea for desktop system, but
>>>>>>> probably bad idea for a server and in particular a hypervisor.
>>>>>>>
>>>>>>> We don't have a solution yet, but you can try these:
>>>>>>>
>>>>>>> 1. disable lvmetad service
>>>>>>>
>>>>>>>     systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket
>>>>>>>     systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket
>>>>>>>
>>>>>>> Edit /etc/lvm/lvm.conf:
>>>>>>>
>>>>>>>     use_lvmetad = 0
>>>>>>>
>>>>>>> 2. disable lvm auto activation
>>>>>>>
>>>>>>> Edit /etc/lvm/lvm.conf:
>>>>>>>
>>>>>>>     auto_activation_volume_list = []
>>>>>>>
>>>>>>> 3. both 1 and 2
>>>>>>>
>>>>>>
>>>>>> I've now applied both of the above and regenerated the initramfs and
>>>>>> rebooted and the host no longer lists the LV's of the VM. Since I
>>>>>> rebooted the host before without this issue, I'm not sure a single
>>>>>> reboot is enough to conclude it has fully fixed the issue.
>>>>>>
>>>>>> You mention that there's no solution yet. Does that mean the above
>>>>>> settings are not 100% certain to avoid this behaviour?
>>>>>>
>>>>>> I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only
>>>>>> include the PV's for the hypervisor disks (on which the OS is installed)
>>>>>> so the system lvm commands only touches those. Since vdsm is using its
>>>>>> own lvm.conf this should be OK for vdsm?
>>>>>
>>>>> This does not seem to work. The host can not be activated as it can't
>>>>> find his volume group(s). To be able to use the global_filter in
>>>>> /etc/lvm/lvm.conf, I believe it should be overridden in vdsm's lvm.conf
>>>>> to revert back to the default.
>>>>>
>>>>> I've moved my filter from global_filter to filter and that seems to
>>>>> work. When lvmetad is disabled I believe this should have the same
>>>>> effect as global_filter? The comments in /etc/lvm/lvm.conf indicate also
>>>>> udev might ignore the filter setting?
>>>>
>>>> Right, global_filter exist so you can override filter used from the command
>>>> line.
>>>>
>>>> For example, hiding certain devices from vdsm. This is why we are using
>>>> filter in vdsm, leaving global_filter for the administrator.
>>>>
>>>> Can you explain why do you need global_filter or filter for the
>>>> hypervisor disks?
>>>
>>> Based on the comment in /etc/lvm/lvm.conf regarding global_filter I
>>> concluded that not only lvmetad but also udev might perform action on
>>> the devices and I wanted to prevent that.
>>>
>>> I've now set the following settings in /etc/lvm/lvm.conf:
>>>
>>> use_lvmetad = 0
>>> auto_activation_volume_list = []
>>> filter = ["a|/dev/sda5|", "r|.*|" ]
>>
>> Better use /dev/disk/by-uuid/ to select the specific device, without
>> depending on device order.
>>
>>>
>>> On other systems I have kept the default filter.
>>>
>>>> Do you have any issue with the current settings, disabling auto activation and
>>>> lvmetad?
>>>
>>> Keeping those two disabled also seems to work. The ovirt LV's do show up
>>> in 'lvs' output but are not activated.
>>
>> Good
>
> When vdsm runs the lvchange command to activate the LV of a VM (so it
> can boot it), will LVM still try to scan the new LV for PV's (and thin
> pools, etc)? Is this also prevented by the auto_activation_volume_list
> parameter in this case?

I don't know, but I don't see why lvm would scan the lv for pv and thin pools,
this should happen on on the guest.

We never had reports about such issue.

>>> I wanted to be absolutely sure the VM LV's were not touched, I added the
>>> filter on some of our hosts.
>>
>> The only problem with this filter is it may break if you change the host in some
>> way, like boot from another disk.
>
> I am aware of this issue, but in this case I would rather mess with a
> hypervisor that no longer boots because it needs an updated lvm.conf
> than try to fix corrupted VM's because the host was accessing the disks
> of the VM while it was running on another node.
>> It would be nice if you file a bug for this, and mention the configuration that
>> fixes this issue, we certainly need to improve the way we configure lvm.
>
> Against which component of oVirt should I file this bug?

vdsm, Core, Storage team

Thanks,
Nir