[ovirt-devel] vdsm disabling logical volumes

Wed May 7 09:37:25 UTC 2014

On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
> On 05/07/2014 09:28 AM, Nir Soffer wrote:
> >----- Original Message -----
> >>From: "Jiri Moskovcak" <jmoskovc at redhat.com>
> >>To: "Nir Soffer" <nsoffer at redhat.com>
> >>Cc: devel at ovirt.org, "Federico Simoncelli" <fsimonce at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Greg
> >>Padgett" <gpadgett at redhat.com>, "Doron Fediuck" <dfediuck at redhat.com>
> >>Sent: Wednesday, May 7, 2014 10:21:28 AM
> >>Subject: Re: [ovirt-devel] vdsm disabling logical volumes
> >>
> >>On 05/05/2014 03:19 PM, Nir Soffer wrote:
> >>>----- Original Message -----
> >>>>From: "Jiri Moskovcak" <jmoskovc at redhat.com>
> >>>>To: "Nir Soffer" <nsoffer at redhat.com>
> >>>>Cc: devel at ovirt.org, "Federico Simoncelli" <fsimonce at redhat.com>, "Allon
> >>>>Mureinik" <amureini at redhat.com>, "Greg
> >>>>Padgett" <gpadgett at redhat.com>
> >>>>Sent: Monday, May 5, 2014 3:44:21 PM
> >>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes
> >>>>
> >>>>On 05/05/2014 02:37 PM, Nir Soffer wrote:
> >>>>>----- Original Message -----
> >>>>>>From: "Jiri Moskovcak" <jmoskovc at redhat.com>
> >>>>>>To: "Nir Soffer" <nsoffer at redhat.com>
> >>>>>>Cc: devel at ovirt.org, "Federico Simoncelli" <fsimonce at redhat.com>, "Allon
> >>>>>>Mureinik" <amureini at redhat.com>, "Greg
> >>>>>>Padgett" <gpadgett at redhat.com>
> >>>>>>Sent: Monday, May 5, 2014 3:16:37 PM
> >>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes
> >>>>>>
> >>>>>>On 05/05/2014 12:01 AM, Nir Soffer wrote:
> >>>>>>>----- Original Message -----
> >>>>>>>>From: "Jiri Moskovcak" <jmoskovc at redhat.com>
> >>>>>>>>To: "Nir Soffer" <nsoffer at redhat.com>
> >>>>>>>>Cc: devel at ovirt.org
> >>>>>>>>Sent: Sunday, May 4, 2014 9:23:49 PM
> >>>>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes
> >>>>>>>>
> >>>>>>>>On 05/04/2014 07:57 PM, Nir Soffer wrote:
> >>>>>>>>>----- Original Message -----
> >>>>>>>>>>From: "Jiri Moskovcak" <jmoskovc at redhat.com>
> >>>>>>>>>>To: devel at ovirt.org
> >>>>>>>>>>Sent: Sunday, May 4, 2014 8:08:33 PM
> >>>>>>>>>>Subject: [ovirt-devel] vdsm disabling logical volumes
> >>>>>>>>>>
> >>>>>>>>>>Greetings vdsm developers!
> >>>>>>>>>>
> >>>>>>>>>>While working on adding ISCSI support to the hosted engine tools, I
> >>>>>>>>>>ran
> >>>>>>>>>>into a problem with vdms. It seems that when stopped vdsm
> >>>>>>>>>>deactivates
> >>>>>>>>>>ALL logical volumes in it's volume group and when it starts it
> >>>>>>>>>>reactivates only specific logical volumes. This is a problem for
> >>>>>>>>>>hosted
> >>>>>>>>>>engine tools as they create logical volumes in the same volume group
> >>>>>>>>>>and
> >>>>>>>>>>when vdsm deactivates the LVs the hosted engine tools don't have a
> >>>>>>>>>>way
> >>>>>>>>>>to reactivate it, because the services drop the root permissions and
> >>>>>>>>>>are
> >>>>>>>>>>running as vdsm and apparently only root can activate LVs.
> >>>>>>>>>
> >>>>>>>>>Can you describe what volumes are you creating, and why?
> >>>>>>>>
> >>>>>>>>We create hosted-engine.lockspace (for sanlock) and
> >>>>>>>>hosted-engine.metadata (keeps data about hosted engine hosts)
> >>>>>>>
> >>>>>>>Do you create these lvs in every vdsm vg?
> >>>>>>
> >>>>>>- only in the first vg created by vdsm while deploying hosted-engine
> >>>
> >>>It seems that the hosted engine has single point of failure - the random
> >>>vg that contains hosted engine data.
> >>>
> >>>>>>
> >>>>>>>Is this part of the domain structure
> >>>>>>>used by hosted engine, or it has nothing to do with the storage domain?
> >>>>>>
> >>>>>>- sorry, I don't understand this question. How can I tell if it has
> >>>>>>something to do with the storage domain? It's for storing data about
> >>>>>>hosts set up to run the hosted-engine and data about state of engine and
> >>>>>>the state of VM running the engine.
> >>>>>
> >>>>>Can you tell us exactly what lvs you are creating, and on which vg?
> >>>>>
> >>>>>And how are you creating those lvs - I guess through vdsm?
> >>>>>
> >>>>
> >>>>- no hosted-engine tools do that by calling:
> >>>>
> >>>>lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE,
> >>>>                       stderr=subprocess.PIPE,
> >>>>                       args=["lvm", "lvcreate", "-L", str(size_bytes)+"B",
> >>>>                             "-n", lv_name, vg_uuid])
> >>>>..
> >>>
> >>>How do you ensure that another host is not modifying the same vg in the
> >>>same time?
> >>>
> >>>If you are not ensuring this, you will corrupt this vg sooner or later.
> >>>
> >>>When a storage domain is detached from a host, for example when the host
> >>>is in maintenance mode, lvs on the shared storage may be deleted,
> >>>invalidating
> >>>the devices mapper maps for these devices. If you write to an lv with wrong
> >>>maps, you may be writing to an extent belonging to another lv, corrupting
> >>>that
> >>>lv data, or even worse corrupting the engine vg data.
> >>>
> >>>How do you ensure that the lvs are not deleted while you are using them?
> >>>
> >>>>
> >>>>>The output of lvs command on a host with hosted engine installed will
> >>>>>help us to understand what you are doing, and then we can think more
> >>>>>clearly
> >>>>>what would be the best way to support this in vdsm.
> >>>>
> >>>>The output of lvs: http://fpaste.org/99196/93619139/
> >>>>
> >>>>HE created these two LVs:
> >>>>ha_agent-hosted-engine.lockspace
> >>>>ha_agent-hosted-engine.metadata
> >>>
> >>>Why do you create these lvs on a vg owned by vdsm?
> >>>
> >>>If you want total control of these lvs, I suggest that you create your own
> >>>vg and put what ever lvs you like there.
> >>>
> >>
> >>I would rather not go this way (at least not for 3.5) as it's too much
> >>code changes in hosted-engine. On the other hand the logic in vdsm seems
> >>wrong because it's not complementary (disabling all LVs and then
> >>enabling just some of them) and should be fixed anyway. This problem is
> >>blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
> >
> >Can you elaborate on this? How should vdsm behave better, and why?
> 
> Sure. So far I didn't hear any reason why it behaves like this and
> it seems not logical to disable *all* and then enable just *some*.
> 
> How: Disabling and enabling operations should be complementary.
> Why: To be less surprising.

There is an asymmetry between activation and deactivation of an LV. A
mistakenly-active LV can cause data corruption. Making sure that this
does not happen is more important than a new feature.

We do not want to deactivate and then re-activating the same set of LVs.
That would be illogical. We intentionally deactivate LVs that are no
longer used on the specific host - that's important if a qemu died while
Vdsm was down, leaving a stale LV behind.

Design-wise, Vdsm would very much like to keep its ownership of
Vdsm-created storage domain. Let us discuss how your feature can be
implemented without this breach of ownership.