[ovirt-users] thin_check run on VM disk by host on startup ?!

[Rik Theys]

Hi,

On 08/31/2016 02:04 PM, Nir Soffer wrote:
> On Wed, Aug 31, 2016 at 2:30 PM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>> Hi,
>>
>> On 08/31/2016 11:51 AM, Nir Soffer wrote:
>>> On Wed, Aug 31, 2016 at 11:07 AM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>>> On 08/31/2016 09:43 AM, Rik Theys wrote:
>>>>> On 08/30/2016 04:47 PM, Nir Soffer wrote:
>>>>>> On Tue, Aug 30, 2016 at 3:51 PM, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>>>>>> While rebooting one of the hosts in an oVirt cluster, I noticed that
>>>>>>> thin_check is run on the thin pool devices of one of the VM's on which
>>>>>>> the disk is assigned to.
>>>>>>>
>>>>>>> That seems strange to me. I would expect the host to stay clear of any
>>>>>>> VM disks.
>>>>>>
>>>>>> We expect the same thing, but unfortunately systemd and lvm try to
>>>>>> auto activate stuff. This may be good idea for desktop system, but
>>>>>> probably bad idea for a server and in particular a hypervisor.
>>>>>>
>>>>>> We don't have a solution yet, but you can try these:
>>>>>>
>>>>>> 1. disable lvmetad service
>>>>>>
>>>>>>     systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket
>>>>>>     systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket
>>>>>>
>>>>>> Edit /etc/lvm/lvm.conf:
>>>>>>
>>>>>>     use_lvmetad = 0
>>>>>>
>>>>>> 2. disable lvm auto activation
>>>>>>
>>>>>> Edit /etc/lvm/lvm.conf:
>>>>>>
>>>>>>     auto_activation_volume_list = []
>>>>>>
>>>>>> 3. both 1 and 2
>>>>>>
>>>>>
>>>>> I've now applied both of the above and regenerated the initramfs and
>>>>> rebooted and the host no longer lists the LV's of the VM. Since I
>>>>> rebooted the host before without this issue, I'm not sure a single
>>>>> reboot is enough to conclude it has fully fixed the issue.
>>>>>
>>>>> You mention that there's no solution yet. Does that mean the above
>>>>> settings are not 100% certain to avoid this behaviour?
>>>>>
>>>>> I was thinking of setting a global_filter in /etc/lvm/lvm.conf to only
>>>>> include the PV's for the hypervisor disks (on which the OS is installed)
>>>>> so the system lvm commands only touches those. Since vdsm is using its
>>>>> own lvm.conf this should be OK for vdsm?
>>>>
>>>> This does not seem to work. The host can not be activated as it can't
>>>> find his volume group(s). To be able to use the global_filter in
>>>> /etc/lvm/lvm.conf, I believe it should be overridden in vdsm's lvm.conf
>>>> to revert back to the default.
>>>>
>>>> I've moved my filter from global_filter to filter and that seems to
>>>> work. When lvmetad is disabled I believe this should have the same
>>>> effect as global_filter? The comments in /etc/lvm/lvm.conf indicate also
>>>> udev might ignore the filter setting?
>>>
>>> Right, global_filter exist so you can override filter used from the command
>>> line.
>>>
>>> For example, hiding certain devices from vdsm. This is why we are using
>>> filter in vdsm, leaving global_filter for the administrator.
>>>
>>> Can you explain why do you need global_filter or filter for the
>>> hypervisor disks?
>>
>> Based on the comment in /etc/lvm/lvm.conf regarding global_filter I
>> concluded that not only lvmetad but also udev might perform action on
>> the devices and I wanted to prevent that.
>>
>> I've now set the following settings in /etc/lvm/lvm.conf:
>>
>> use_lvmetad = 0
>> auto_activation_volume_list = []
>> filter = ["a|/dev/sda5|", "r|.*|" ]
> 
> Better use /dev/disk/by-uuid/ to select the specific device, without
> depending on device order.
> 
>>
>> On other systems I have kept the default filter.
>>
>>> Do you have any issue with the current settings, disabling auto activation and
>>> lvmetad?
>>
>> Keeping those two disabled also seems to work. The ovirt LV's do show up
>> in 'lvs' output but are not activated.
> 
> Good

When vdsm runs the lvchange command to activate the LV of a VM (so it
can boot it), will LVM still try to scan the new LV for PV's (and thin
pools, etc)? Is this also prevented by the auto_activation_volume_list
parameter in this case?

>> I wanted to be absolutely sure the VM LV's were not touched, I added the
>> filter on some of our hosts.
> 
> The only problem with this filter is it may break if you change the host in some
> way, like boot from another disk.

I am aware of this issue, but in this case I would rather mess with a
hypervisor that no longer boots because it needs an updated lvm.conf
than try to fix corrupted VM's because the host was accessing the disks
of the VM while it was running on another node.

> It would be nice if you file a bug for this, and mention the configuration that
> fixes this issue, we certainly need to improve the way we configure lvm.

Against which component of oVirt should I file this bug?

Regards,

Rik

-- 
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440  - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>