Re: [ovirt-devel] vdsm disabling logical volumes

----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; > To: "Nir Soffer" <nsoffer(a)redhat.com&gt; > Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon Mureinik" <amureini(a)redhat.com&gt;, "Greg > Padgett" <gpadgett(a)redhat.com&gt;, "Doron Fediuck" <dfediuck(a)redhat.com&gt; > Sent: Wednesday, May 7, 2014 10:21:28 AM > Subject: Re: [ovirt-devel] vdsm disabling logical volumes > > On 05/05/2014 03:19 PM, Nir Soffer wrote: >> ----- Original Message ----- >>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>> Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon >>> Mureinik" <amureini(a)redhat.com&gt;, "Greg >>> Padgett" <gpadgett(a)redhat.com&gt; >>> Sent: Monday, May 5, 2014 3:44:21 PM >>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>> >>> On 05/05/2014 02:37 PM, Nir Soffer wrote: >>>> ----- Original Message ----- >>>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>> Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon >>>>> Mureinik" <amureini(a)redhat.com&gt;, "Greg >>>>> Padgett" <gpadgett(a)redhat.com&gt; >>>>> Sent: Monday, May 5, 2014 3:16:37 PM >>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>> >>>>> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>>> ----- Original Message ----- >>>>>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>>> Cc: devel(a)ovirt.org >>>>>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>> >>>>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>>> To: devel(a)ovirt.org >>>>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>> >>>>>>>>> Greetings vdsm developers! >>>>>>>>> >>>>>>>>> While working on adding ISCSI support to the hosted engine tools, I >>>>>>>>> ran >>>>>>>>> into a problem with vdms. It seems that when stopped vdsm >>>>>>>>> deactivates >>>>>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>>>>> reactivates only specific logical volumes. This is a problem for >>>>>>>>> hosted >>>>>>>>> engine tools as they create logical volumes in the same volume group >>>>>>>>> and >>>>>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>>> way >>>>>>>>> to reactivate it, because the services drop the root permissions and >>>>>>>>> are >>>>>>>>> running as vdsm and apparently only root can activate LVs. >>>>>>>> >>>>>>>> Can you describe what volumes are you creating, and why? >>>>>>> >>>>>>> We create hosted-engine.lockspace (for sanlock) and >>>>>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>>>>> >>>>>> Do you create these lvs in every vdsm vg? >>>>> >>>>> - only in the first vg created by vdsm while deploying hosted-engine >> >> It seems that the hosted engine has single point of failure - the random >> vg that contains hosted engine data. >> >>>>> >>>>>> Is this part of the domain structure >>>>>> used by hosted engine, or it has nothing to do with the storage domain? >>>>> >>>>> - sorry, I don't understand this question. How can I tell if it has >>>>> something to do with the storage domain? It's for storing data about >>>>> hosts set up to run the hosted-engine and data about state of engine and >>>>> the state of VM running the engine. >>>> >>>> Can you tell us exactly what lvs you are creating, and on which vg? >>>> >>>> And how are you creating those lvs - I guess through vdsm? >>>> >>> >>> - no hosted-engine tools do that by calling: >>> >>> lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >>> stderr=subprocess.PIPE, >>> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >>> "-n", lv_name, vg_uuid]) >>> .. >> >> How do you ensure that another host is not modifying the same vg in the >> same time? >> >> If you are not ensuring this, you will corrupt this vg sooner or later. >> >> When a storage domain is detached from a host, for example when the host >> is in maintenance mode, lvs on the shared storage may be deleted, >> invalidating >> the devices mapper maps for these devices. If you write to an lv with wrong >> maps, you may be writing to an extent belonging to another lv, corrupting >> that >> lv data, or even worse corrupting the engine vg data. >> >> How do you ensure that the lvs are not deleted while you are using them? >> >>> >>>> The output of lvs command on a host with hosted engine installed will >>>> help us to understand what you are doing, and then we can think more >>>> clearly >>>> what would be the best way to support this in vdsm. >>> >>> The output of lvs: http://fpaste.org/99196/93619139/ >>> >>> HE created these two LVs: >>> ha_agent-hosted-engine.lockspace >>> ha_agent-hosted-engine.metadata >> >> Why do you create these lvs on a vg owned by vdsm? >> >> If you want total control of these lvs, I suggest that you create your own >> vg and put what ever lvs you like there. >> > > I would rather not go this way (at least not for 3.5) as it's too much > code changes in hosted-engine. On the other hand the logic in vdsm seems > wrong because it's not complementary (disabling all LVs and then > enabling just some of them) and should be fixed anyway. This problem is > blocking one of our 3.5 features so I've created rhbz#1094657 to track it. Can you elaborate on this? How should vdsm behave better, and why?

> > --Jirka > >>> >>> --Jirka >>>> >>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>>> So far the >>>>>>>>> only suitable solution seems to be to change vdsm to only >>>>>>>>> deactivate/activate it's own LVs. >>>>>>>> >>>>>>>> This sounds reasonable. You can add a list of hosted engine lv names >>>>>>>> and skip these volumes when deactivating vdsm volumes. >>>>>>> >>>>>>> - this sounds a bit suboptimal, vdsm already has list of it's LVs, so >>>>>>> it >>>>>>> can just disable only LVs known to it, otherwise we would have to >>>>>>> change >>>>>>> the list everytime we add some LV to the group >>>>>> >>>>>> vdsm has a list of special lvs, that needs special treatment. >>>>>> Otherwise, >>>>>> it >>>>>> consider any other lv as owned by vdsm, and will deactivate them when >>>>>> they >>>>>> are >>>>>> not used. >>>>>> >>>>>> I agree that this will create a dependency, but this can also be >>>>>> solved. >>>>>> For example, vdsm can load the list from a file installed by hosted >>>>>> engine, >>>>>> like the typical conf.d directories. >>>>>> >>>>> >>>>> - ok, this is something I actually don't have strong opinion about, for >>>>> me adding a file with a list of LVs or tagging the logical volumes is >>>>> almost the same, I just need a way to tell vdsm which LVs to ignore.. >>>>> >>>>>>> >>>>>>>> >>>>>>>> Another solution is to tag hosted engine lvs, and have vdsm ignore >>>>>>>> lvs that contains this tag. >>>>>>> >>>>>>> - this sounds good, because if we teach vdsm to ignore LVs with some >>>>>>> tag >>>>>>> we can add new LVs in future without changing vdsm. This however >>>>>>> applies >>>>>>> also to the solution where vdsm only disables it's own LVs, >>>>>> >>>>>> vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like >>>>>> this >>>>>> using some historic tags we keep (e.g. RHAT_*), but I'd rather add new >>>>>> tag >>>>>> with >>>>>> clear semantic than use some random historic value we have. >>>>>> >>>>>>> so it >>>>>>> depends on vdsm devels which solution they find better. I think the >>>>>>> solution without tags is better, because is simpler and others (like >>>>>>> hosted-engine) can just createlv and don't bother with tags.. >>>>>> >>>>>> I think that a generic tag like OVIRT_IGNORE is an easy solution for >>>>>> everyone. >>>>>> >>>>>> Federico, what do you think? >>>>>> >>>>>> Nir >>>>>> >>>>> >>>>> >>> >>> > >