Re: [ovirt-devel] vdsm disabling logical volumes

On 05/12/2014 11:22 AM, Dan Kenigsberg wrote: >On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote: >>On 05/07/2014 11:37 AM, Dan Kenigsberg wrote: >>>On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote: >>>>On 05/07/2014 09:28 AM, Nir Soffer wrote: >>>>>----- Original Message ----- >>>>>>From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>>Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon Mureinik" <amureini(a)redhat.com&gt;, "Greg >>>>>>Padgett" <gpadgett(a)redhat.com&gt;, "Doron Fediuck" <dfediuck(a)redhat.com&gt; >>>>>>Sent: Wednesday, May 7, 2014 10:21:28 AM >>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>> >>>>>>On 05/05/2014 03:19 PM, Nir Soffer wrote: >>>>>>>----- Original Message ----- >>>>>>>>From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>>To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>>>>Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon >>>>>>>>Mureinik" <amureini(a)redhat.com&gt;, "Greg >>>>>>>>Padgett" <gpadgett(a)redhat.com&gt; >>>>>>>>Sent: Monday, May 5, 2014 3:44:21 PM >>>>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>>> >>>>>>>>On 05/05/2014 02:37 PM, Nir Soffer wrote: >>>>>>>>>----- Original Message ----- >>>>>>>>>>From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>>>>To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>>>>>>Cc: devel(a)ovirt.org, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, "Allon >>>>>>>>>>Mureinik" <amureini(a)redhat.com&gt;, "Greg >>>>>>>>>>Padgett" <gpadgett(a)redhat.com&gt; >>>>>>>>>>Sent: Monday, May 5, 2014 3:16:37 PM >>>>>>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>>> >>>>>>>>>>On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>>>>>>>>----- Original Message ----- >>>>>>>>>>>>From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>>>>>>To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>>>>>>>>Cc: devel(a)ovirt.org >>>>>>>>>>>>Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>>>>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>>>>> >>>>>>>>>>>>On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>>>>>>>>----- Original Message ----- >>>>>>>>>>>>>>From: "Jiri Moskovcak" <jmoskovc(a)redhat.com&gt; >>>>>>>>>>>>>>To: devel(a)ovirt.org >>>>>>>>>>>>>>Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>>>>>>>>Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>>>>>>> >>>>>>>>>>>>>>Greetings vdsm developers! >>>>>>>>>>>>>> >>>>>>>>>>>>>>While working on adding ISCSI support to the hosted engine tools, I >>>>>>>>>>>>>>ran >>>>>>>>>>>>>>into a problem with vdms. It seems that when stopped vdsm >>>>>>>>>>>>>>deactivates >>>>>>>>>>>>>>ALL logical volumes in it's volume group and when it starts it >>>>>>>>>>>>>>reactivates only specific logical volumes. This is a problem for >>>>>>>>>>>>>>hosted >>>>>>>>>>>>>>engine tools as they create logical volumes in the same volume group >>>>>>>>>>>>>>and >>>>>>>>>>>>>>when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>>>>>>>>way >>>>>>>>>>>>>>to reactivate it, because the services drop the root permissions and >>>>>>>>>>>>>>are >>>>>>>>>>>>>>running as vdsm and apparently only root can activate LVs. >>>>>>>>>>>>> >>>>>>>>>>>>>Can you describe what volumes are you creating, and why? >>>>>>>>>>>> >>>>>>>>>>>>We create hosted-engine.lockspace (for sanlock) and >>>>>>>>>>>>hosted-engine.metadata (keeps data about hosted engine hosts) >>>>>>>>>>> >>>>>>>>>>>Do you create these lvs in every vdsm vg? >>>>>>>>>> >>>>>>>>>>- only in the first vg created by vdsm while deploying hosted-engine >>>>>>> >>>>>>>It seems that the hosted engine has single point of failure - the random >>>>>>>vg that contains hosted engine data. >>>>>>> >>>>>>>>>> >>>>>>>>>>>Is this part of the domain structure >>>>>>>>>>>used by hosted engine, or it has nothing to do with the storage domain? >>>>>>>>>> >>>>>>>>>>- sorry, I don't understand this question. How can I tell if it has >>>>>>>>>>something to do with the storage domain? It's for storing data about >>>>>>>>>>hosts set up to run the hosted-engine and data about state of engine and >>>>>>>>>>the state of VM running the engine. >>>>>>>>> >>>>>>>>>Can you tell us exactly what lvs you are creating, and on which vg? >>>>>>>>> >>>>>>>>>And how are you creating those lvs - I guess through vdsm? >>>>>>>>> >>>>>>>> >>>>>>>>- no hosted-engine tools do that by calling: >>>>>>>> >>>>>>>>lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >>>>>>>> stderr=subprocess.PIPE, >>>>>>>> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >>>>>>>> "-n", lv_name, vg_uuid]) >>>>>>>>.. >>>>>>> >>>>>>>How do you ensure that another host is not modifying the same vg in the >>>>>>>same time? >>>>>>> >>>>>>>If you are not ensuring this, you will corrupt this vg sooner or later. >>>>>>> >>>>>>>When a storage domain is detached from a host, for example when the host >>>>>>>is in maintenance mode, lvs on the shared storage may be deleted, >>>>>>>invalidating >>>>>>>the devices mapper maps for these devices. If you write to an lv with wrong >>>>>>>maps, you may be writing to an extent belonging to another lv, corrupting >>>>>>>that >>>>>>>lv data, or even worse corrupting the engine vg data. >>>>>>> >>>>>>>How do you ensure that the lvs are not deleted while you are using them? >>>>>>> >>>>>>>> >>>>>>>>>The output of lvs command on a host with hosted engine installed will >>>>>>>>>help us to understand what you are doing, and then we can think more >>>>>>>>>clearly >>>>>>>>>what would be the best way to support this in vdsm. >>>>>>>> >>>>>>>>The output of lvs: http://fpaste.org/99196/93619139/ >>>>>>>> >>>>>>>>HE created these two LVs: >>>>>>>>ha_agent-hosted-engine.lockspace >>>>>>>>ha_agent-hosted-engine.metadata >>>>>>> >>>>>>>Why do you create these lvs on a vg owned by vdsm? >>>>>>> >>>>>>>If you want total control of these lvs, I suggest that you create your own >>>>>>>vg and put what ever lvs you like there. >>>>>>> >>>>>> >>>>>>I would rather not go this way (at least not for 3.5) as it's too much >>>>>>code changes in hosted-engine. On the other hand the logic in vdsm seems >>>>>>wrong because it's not complementary (disabling all LVs and then >>>>>>enabling just some of them) and should be fixed anyway. This problem is >>>>>>blocking one of our 3.5 features so I've created rhbz#1094657 to track it. >>>>> >>>>>Can you elaborate on this? How should vdsm behave better, and why? >>>> >>>>Sure. So far I didn't hear any reason why it behaves like this and >>>>it seems not logical to disable *all* and then enable just *some*. >>>> >>>>How: Disabling and enabling operations should be complementary. >>>>Why: To be less surprising. >>> >>>There is an asymmetry between activation and deactivation of an LV. A >>>mistakenly-active LV can cause data corruption. Making sure that this >>>does not happen is more important than a new feature. >> >>- just out of a curiosity, how can mistakenly-active LV cause data >>corruption? something like a stalled LV which refers to a volume >>which doesn't exists anymore? >> >>> >>>We do not want to deactivate and then re-activating the same set of LVs. >>>That would be illogical. We intentionally deactivate LVs that are no >>>longer used on the specific host - that's important if a qemu died while >>>Vdsm was down, leaving a stale LV behind. >>> >>>Design-wise, Vdsm would very much like to keep its ownership of >>>Vdsm-created storage domain. Let us discuss how your feature can be >>>implemented without this breach of ownership. >>> >> >>Ok, I agree that this should have been discussed with the storage >>team at the design phase, so let's start from the beginning and try >>to come up with a better solution. >> My problem is that I need a storage for the hosted-engine data >>which is accessible from all hosts. It seems logical to use the same >>physical storage as we use for "the storage". For NFS it's just a >>file in >>/rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how >>do you suggest to store such data in case of using lvm (iscsi in >>this case). Can we use vdsm to set it up or do we have to duplicate >>the lvm code and handle it our self? > >I think that for this to happen, we need to define a Vdsm verb that >creates a volume on a storage domain that is unrelated to any pool. Such >a verb is in planning; Federico, can its implementation be hasten in >favor of hosted engine? > >On its own, this would not solve the problem of Vdsm deactivating all >unused LVs. > >Jiri, could you describe why you keep your LV active, but not open? > - the setup flow goes approximately like this: 1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs - I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted