Re: [ovirt-devel] vdsm disabling logical volumes

On 05/05/2014 03:19 PM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >> Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon Mureinik" <amureini(a)redhat.com&gt;, "Greg >> Padgett" <gpadgett(a)redhat.com&gt; >> Sent: Monday, May 5, 2014 3:44:21 PM >> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >> >> On 05/05/2014 02:37 PM, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>> Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon >>>> Mureinik" <amureini(a)redhat.com&gt;, "Greg >>>> Padgett" <gpadgett(a)redhat.com&gt; >>>> Sent: Monday, May 5, 2014 3:16:37 PM >>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>> >>>> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>> Cc: devel(a)ovirt.org >>>>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>> >>>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>> To: devel(a)ovirt.org >>>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>> >>>>>>>> Greetings vdsm developers! >>>>>>>> >>>>>>>> While working on adding ISCSI support to the hosted engine tools, I >>>>>>>> ran >>>>>>>> into a problem with vdms. It seems that when stopped vdsm deactivates >>>>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>>>> reactivates only specific logical volumes. This is a problem for >>>>>>>> hosted >>>>>>>> engine tools as they create logical volumes in the same volume group >>>>>>>> and >>>>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a way >>>>>>>> to reactivate it, because the services drop the root permissions and >>>>>>>> are >>>>>>>> running as vdsm and apparently only root can activate LVs. >>>>>>> >>>>>>> Can you describe what volumes are you creating, and why? >>>>>> >>>>>> We create hosted-engine.lockspace (for sanlock) and >>>>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>>>> >>>>> Do you create these lvs in every vdsm vg? >>>> >>>> - only in the first vg created by vdsm while deploying hosted-engine > > It seems that the hosted engine has single point of failure - the random > vg that contains hosted engine data. - yes, seems like it, but that's for another discussion > >>>> >>>>> Is this part of the domain structure >>>>> used by hosted engine, or it has nothing to do with the storage domain? >>>> >>>> - sorry, I don't understand this question. How can I tell if it has >>>> something to do with the storage domain? It's for storing data about >>>> hosts set up to run the hosted-engine and data about state of engine and >>>> the state of VM running the engine. >>> >>> Can you tell us exactly what lvs you are creating, and on which vg? >>> >>> And how are you creating those lvs - I guess through vdsm? >>> >> >> - no hosted-engine tools do that by calling: >> >> lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >> stderr=subprocess.PIPE, >> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >> "-n", lv_name, vg_uuid]) >> .. > > How do you ensure that another host is not modifying the same vg in the > same time? > > If you are not ensuring this, you will corrupt this vg sooner or later. > > When a storage domain is detached from a host, for example when the host > is in maintenance mode, lvs on the shared storage may be deleted, invalidating > the devices mapper maps for these devices. If you write to an lv with wrong > maps, you may be writing to an extent belonging to another lv, corrupting that > lv data, or even worse corrupting the engine vg data. > - when the vdsm wants to move some host to maintenance it contacts the ha-agent which stops writing data to the storage, but it might be a problem if the VG is changed while in maintenance, I don't think we handle such situation > How do you ensure that the lvs are not deleted while you are using them? - obviously we're not handling this, otherwise the vdsm wouldn't be able to disable those LVs - I'm no expert on lvm, so I could use some advice on how to do it, so how does the vdsm ensures that it's VG is not modified? > >> >>> The output of lvs command on a host with hosted engine installed will >>> help us to understand what you are doing, and then we can think more >>> clearly >>> what would be the best way to support this in vdsm. >> >> The output of lvs: http://fpaste.org/99196/93619139/ >> >> HE created these two LVs: >> ha_agent-hosted-engine.lockspace >> ha_agent-hosted-engine.metadata > > Why do you create these lvs on a vg owned by vdsm? > - I actually don't know this decision has been taken before I started working on the code > If you want total control of these lvs, I suggest that you create your own > vg and put what ever lvs you like there. > Yes, that would solve the problem with vdsm, I hope Sandro or Martin could explain why we use the vdsm's VG instead of creating our own...

--Jirka >> >> --Jirka >>> >>>> >>>>> >>>>>> >>>>>>> >>>>>>>> So far the >>>>>>>> only suitable solution seems to be to change vdsm to only >>>>>>>> deactivate/activate it's own LVs. >>>>>>> >>>>>>> This sounds reasonable. You can add a list of hosted engine lv names >>>>>>> and skip these volumes when deactivating vdsm volumes. >>>>>> >>>>>> - this sounds a bit suboptimal, vdsm already has list of it's LVs, so it >>>>>> can just disable only LVs known to it, otherwise we would have to change >>>>>> the list everytime we add some LV to the group >>>>> >>>>> vdsm has a list of special lvs, that needs special treatment. Otherwise, >>>>> it >>>>> consider any other lv as owned by vdsm, and will deactivate them when >>>>> they >>>>> are >>>>> not used. >>>>> >>>>> I agree that this will create a dependency, but this can also be solved. >>>>> For example, vdsm can load the list from a file installed by hosted >>>>> engine, >>>>> like the typical conf.d directories. >>>>> >>>> >>>> - ok, this is something I actually don't have strong opinion about, for >>>> me adding a file with a list of LVs or tagging the logical volumes is >>>> almost the same, I just need a way to tell vdsm which LVs to ignore.. >>>> >>>>>> >>>>>>> >>>>>>> Another solution is to tag hosted engine lvs, and have vdsm ignore >>>>>>> lvs that contains this tag. >>>>>> >>>>>> - this sounds good, because if we teach vdsm to ignore LVs with some tag >>>>>> we can add new LVs in future without changing vdsm. This however applies >>>>>> also to the solution where vdsm only disables it's own LVs, >>>>> >>>>> vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like >>>>> this >>>>> using some historic tags we keep (e.g. RHAT_*), but I'd rather add new >>>>> tag >>>>> with >>>>> clear semantic than use some random historic value we have. >>>>> >>>>>> so it >>>>>> depends on vdsm devels which solution they find better. I think the >>>>>> solution without tags is better, because is simpler and others (like >>>>>> hosted-engine) can just createlv and don't bother with tags.. >>>>> >>>>> I think that a generic tag like OVIRT_IGNORE is an easy solution for >>>>> everyone. >>>>> >>>>> Federico, what do you think? >>>>> >>>>> Nir >>>>> >>>> >>>> >> >>