Il 05/05/2014 19:16, Jiri Moskovcak ha scritto:
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc@redhat.com> > To: "Nir Soffer" <nsoffer@redhat.com> > Cc: devel@ovirt.org > Sent: Sunday, May 4, 2014 9:23:49 PM > Subject: Re: [ovirt-devel] vdsm disabling logical volumes > > On 05/04/2014 07:57 PM, Nir Soffer wrote: >> ----- Original Message ----- >>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>> To: devel@ovirt.org >>> Sent: Sunday, May 4, 2014 8:08:33 PM >>> Subject: [ovirt-devel] vdsm disabling logical volumes >>> >>> Greetings vdsm developers! >>> >>> While working on adding ISCSI support to the hosted engine tools, I >>> ran >>> into a problem with vdms. It seems that when stopped vdsm deactivates >>> ALL logical volumes in it's volume group and when it starts it >>> reactivates only specific logical volumes. This is a problem for >>> hosted >>> engine tools as they create logical volumes in the same volume group >>> and >>> when vdsm deactivates the LVs the hosted engine tools don't have a way >>> to reactivate it, because the services drop the root permissions and >>> are >>> running as vdsm and apparently only root can activate LVs. >> >> Can you describe what volumes are you creating, and why? > > We create hosted-engine.lockspace (for sanlock) and > hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
- yes, seems like it, but that's for another discussion
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
- when the vdsm wants to move some host to maintenance it contacts the ha-agent which stops writing data to the storage, but it might be a problem if the VG is changed while in maintenance, I don't think we handle such situation
How do you ensure that the lvs are not deleted while you are using them?
- obviously we're not handling this, otherwise the vdsm wouldn't be able to disable those LVs
- I'm no expert on lvm, so I could use some advice on how to do it, so how does the vdsm ensures that it's VG is not modified?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
- I actually don't know this decision has been taken before I started working on the code
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
Yes, that would solve the problem with vdsm, I hope Sandro or Martin could explain why we use the vdsm's VG instead of creating our own...
I don't know why hosted engine metadata are in a different lv, I just create the storage domain using vdsm and then call Martin's library for creating the metadata. On NFS they're inside the storage domain, not on a different storage.
--Jirka
--Jirka
> >> >>> So far the >>> only suitable solution seems to be to change vdsm to only >>> deactivate/activate it's own LVs. >> >> This sounds reasonable. You can add a list of hosted engine lv names >> and skip these volumes when deactivating vdsm volumes. > > - this sounds a bit suboptimal, vdsm already has list of it's LVs, so it > can just disable only LVs known to it, otherwise we would have to change > the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
> >> >> Another solution is to tag hosted engine lvs, and have vdsm ignore >> lvs that contains this tag. > > - this sounds good, because if we teach vdsm to ignore LVs with some tag > we can add new LVs in future without changing vdsm. This however applies > also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
> so it > depends on vdsm devels which solution they find better. I think the > solution without tags is better, because is simpler and others (like > hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com