[ovirt-devel] vdsm disabling logical volumes

Jiri Moskovcak jmoskovc at redhat.com
Mon May 5 17:16:18 UTC 2014 

On 05/05/2014 03:19 PM, Nir Soffer wrote:
> ----- Original Message -----
>> From: "Jiri Moskovcak" <jmoskovc at redhat.com>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: devel at ovirt.org, "Federico Simoncelli" <fsimonce at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Greg
>> Padgett" <gpadgett at redhat.com>
>> Sent: Monday, May 5, 2014 3:44:21 PM
>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes
>>
>> On 05/05/2014 02:37 PM, Nir Soffer wrote:
>>> ----- Original Message -----
>>>> From: "Jiri Moskovcak" <jmoskovc at redhat.com>
>>>> To: "Nir Soffer" <nsoffer at redhat.com>
>>>> Cc: devel at ovirt.org, "Federico Simoncelli" <fsimonce at redhat.com>, "Allon
>>>> Mureinik" <amureini at redhat.com>, "Greg
>>>> Padgett" <gpadgett at redhat.com>
>>>> Sent: Monday, May 5, 2014 3:16:37 PM
>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes
>>>>
>>>> On 05/05/2014 12:01 AM, Nir Soffer wrote:
>>>>> ----- Original Message -----
>>>>>> From: "Jiri Moskovcak" <jmoskovc at redhat.com>
>>>>>> To: "Nir Soffer" <nsoffer at redhat.com>
>>>>>> Cc: devel at ovirt.org
>>>>>> Sent: Sunday, May 4, 2014 9:23:49 PM
>>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes
>>>>>>
>>>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote:
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Jiri Moskovcak" <jmoskovc at redhat.com>
>>>>>>>> To: devel at ovirt.org
>>>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM
>>>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes
>>>>>>>>
>>>>>>>> Greetings vdsm developers!
>>>>>>>>
>>>>>>>> While working on adding ISCSI support to the hosted engine tools, I
>>>>>>>> ran
>>>>>>>> into a problem with vdms. It seems that when stopped vdsm deactivates
>>>>>>>> ALL logical volumes in it's volume group and when it starts it
>>>>>>>> reactivates only specific logical volumes. This is a problem for
>>>>>>>> hosted
>>>>>>>> engine tools as they create logical volumes in the same volume group
>>>>>>>> and
>>>>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a way
>>>>>>>> to reactivate it, because the services drop the root permissions and
>>>>>>>> are
>>>>>>>> running as vdsm and apparently only root can activate LVs.
>>>>>>>
>>>>>>> Can you describe what volumes are you creating, and why?
>>>>>>
>>>>>> We create hosted-engine.lockspace (for sanlock) and
>>>>>> hosted-engine.metadata (keeps data about hosted engine hosts)
>>>>>
>>>>> Do you create these lvs in every vdsm vg?
>>>>
>>>> - only in the first vg created by vdsm while deploying hosted-engine
>
> It seems that the hosted engine has single point of failure - the random
> vg that contains hosted engine data.

- yes, seems like it, but that's for another discussion

>
>>>>
>>>>> Is this part of the domain structure
>>>>> used by hosted engine, or it has nothing to do with the storage domain?
>>>>
>>>> - sorry, I don't understand this question. How can I tell if it has
>>>> something to do with the storage domain? It's for storing data about
>>>> hosts set up to run the hosted-engine and data about state of engine and
>>>> the state of VM running the engine.
>>>
>>> Can you tell us exactly what lvs you are creating, and on which vg?
>>>
>>> And how are you creating those lvs - I guess through vdsm?
>>>
>>
>> - no hosted-engine tools do that by calling:
>>
>> lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE,
>>                       stderr=subprocess.PIPE,
>>                       args=["lvm", "lvcreate", "-L", str(size_bytes)+"B",
>>                             "-n", lv_name, vg_uuid])
>> ..
>
> How do you ensure that another host is not modifying the same vg in the
> same time?
>
> If you are not ensuring this, you will corrupt this vg sooner or later.
>
> When a storage domain is detached from a host, for example when the host
> is in maintenance mode, lvs on the shared storage may be deleted, invalidating
> the devices mapper maps for these devices. If you write to an lv with wrong
> maps, you may be writing to an extent belonging to another lv, corrupting that
> lv data, or even worse corrupting the engine vg data.
>

- when the vdsm wants to move some host to maintenance it contacts the 
ha-agent which stops writing data to the storage, but it might be a 
problem if the VG is changed while in maintenance, I don't think we 
handle such situation

> How do you ensure that the lvs are not deleted while you are using them?

- obviously we're not handling this, otherwise the vdsm wouldn't be able 
to disable those LVs

- I'm no expert on lvm, so I could use some advice on how to do it, so 
how does the vdsm ensures that it's VG is not modified?

>
>>
>>> The output of lvs command on a host with hosted engine installed will
>>> help us to understand what you are doing, and then we can think more
>>> clearly
>>> what would be the best way to support this in vdsm.
>>
>> The output of lvs: http://fpaste.org/99196/93619139/
>>
>> HE created these two LVs:
>> ha_agent-hosted-engine.lockspace
>> ha_agent-hosted-engine.metadata
>
> Why do you create these lvs on a vg owned by vdsm?
>

- I actually don't know this decision has been taken before I started 
working on the code

> If you want total control of these lvs, I suggest that you create your own
> vg and put what ever lvs you like there.
>

Yes, that would solve the problem with vdsm, I hope Sandro or Martin 
could explain why we use the vdsm's VG instead of creating our own...

--Jirka


>>
>> --Jirka
>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> So far the
>>>>>>>> only suitable solution seems to be to change vdsm to only
>>>>>>>> deactivate/activate it's own LVs.
>>>>>>>
>>>>>>> This sounds reasonable. You can add a list of hosted engine lv names
>>>>>>> and skip these volumes when deactivating vdsm volumes.
>>>>>>
>>>>>> - this sounds a bit suboptimal, vdsm already has list of it's LVs, so it
>>>>>> can just disable only LVs known to it, otherwise we would have to change
>>>>>> the list everytime we add some LV to the group
>>>>>
>>>>> vdsm has a list of special lvs, that needs special treatment. Otherwise,
>>>>> it
>>>>> consider any other lv as owned by vdsm, and will deactivate them when
>>>>> they
>>>>> are
>>>>> not used.
>>>>>
>>>>> I agree that this will create a dependency, but this can also be solved.
>>>>> For example, vdsm can load the list from a file installed by hosted
>>>>> engine,
>>>>> like the typical conf.d directories.
>>>>>
>>>>
>>>> - ok, this is something I actually don't have strong opinion about, for
>>>> me adding a file with a list of LVs or tagging the logical volumes is
>>>> almost the same, I just need a way to tell vdsm which LVs to ignore..
>>>>
>>>>>>
>>>>>>>
>>>>>>> Another solution is to tag hosted engine lvs, and have vdsm ignore
>>>>>>> lvs that contains this tag.
>>>>>>
>>>>>> - this sounds good, because if we teach vdsm to ignore LVs with some tag
>>>>>> we can add new LVs in future without changing vdsm. This however applies
>>>>>> also to the solution where vdsm only disables it's own LVs,
>>>>>
>>>>> vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like
>>>>> this
>>>>> using some historic tags we keep (e.g. RHAT_*), but I'd rather add new
>>>>> tag
>>>>> with
>>>>> clear semantic than use some random historic value we have.
>>>>>
>>>>>> so it
>>>>>> depends on vdsm devels which solution they find better. I think the
>>>>>> solution without tags is better, because is simpler and others (like
>>>>>> hosted-engine) can just createlv and don't bother with tags..
>>>>>
>>>>> I think that a generic tag like OVIRT_IGNORE is an easy solution for
>>>>> everyone.
>>>>>
>>>>> Federico, what do you think?
>>>>>
>>>>> Nir
>>>>>
>>>>
>>>>
>>
>>

More information about the Devel mailing list