12:08 p.m.
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran
into a problem with vdms. It seems that when stopped vdsm deactivates
ALL logical volumes in it's volume group and when it starts it
reactivates only specific logical volumes. This is a problem for hosted
engine tools as they create logical volumes in the same volume group and
when vdsm deactivates the LVs the hosted engine tools don't have a way
to reactivate it, because the services drop the root permissions and are
running as vdsm and apparently only root can activate LVs. So far the
only suitable solution seems to be to change vdsm to only
deactivate/activate it's own LVs. I would be grateful for any hints.
Thank you,
Jirka
12:57 p.m.
----- Original Message -----
Can you describe what volumes are you creating, and why?
So far the
only suitable solution seems to be to change vdsm to only
deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names
and skip these volumes when deactivating vdsm volumes.
Another solution is to tag hosted engine lvs, and have vdsm ignore
lvs that contains this tag.
Nir
1:23 p.m.
On 05/04/2014 07:57 PM, Nir Soffer wrote:
We create hosted-engine.lockspace (for sanlock) and
hosted-engine.metadata (keeps data about hosted engine hosts)
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it
can just disable only LVs known to it, otherwise we would have to change
the list everytime we add some LV to the group
Another solution is to tag hosted engine lvs, and have vdsm ignore
lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag
we can add new LVs in future without changing vdsm. This however applies
also to the solution where vdsm only disables it's own LVs, so it
depends on vdsm devels which solution they find better. I think the
solution without tags is better, because is simpler and others (like
hosted-engine) can just createlv and don't bother with tags..
--Jirka
Nir
5:01 p.m.
----- Original Message -----
Do you create these lvs in every vdsm vg? Is this part of the domain structure
used by hosted engine, or it has nothing to do with the storage domain?
vdsm has a list of special lvs, that needs special treatment. Otherwise, it
consider any other lv as owned by vdsm, and will deactivate them when they are
not used.
I agree that this will create a dependency, but this can also be solved.
For example, vdsm can load the list from a file installed by hosted engine,
like the typical conf.d directories.
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this
using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with
clear semantic than use some random historic value we have.
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
7:16 a.m.
On 05/05/2014 12:01 AM, Nir Soffer wrote:
- only in the first vg created by vdsm while deploying hosted-engine
Is this part of the domain structure
used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has
something to do with the storage domain? It's for storing data about
hosts set up to run the hosted-engine and data about state of engine and
the state of VM running the engine.
- ok, this is something I actually don't have strong opinion about, for
me adding a file with a list of LVs or tagging the logical volumes is
almost the same, I just need a way to tell vdsm which LVs to ignore..
7:37 a.m.
----- Original Message -----
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
The output of lvs command on a host with hosted engine installed will
help us to understand what you are doing, and then we can think more clearly
what would be the best way to support this in vdsm.
7:44 a.m.
On 05/05/2014 02:37 PM, Nir Soffer wrote:
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
args=["lvm", "lvcreate", "-L",
str(size_bytes)+"B",
"-n", lv_name, vg_uuid])
..
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs:
ha_agent-hosted-engine.lockspace
ha_agent-hosted-engine.metadata
--Jirka
8:19 a.m.
----- Original Message -----
It seems that the hosted engine has single point of failure - the random
vg that contains hosted engine data.
How do you ensure that another host is not modifying the same vg in the
same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host
is in maintenance mode, lvs on the shared storage may be deleted, invalidating
the devices mapper maps for these devices. If you write to an lv with wrong
maps, you may be writing to an extent belonging to another lv, corrupting that
lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own
vg and put what ever lvs you like there.
12:16 p.m.
On 05/05/2014 03:19 PM, Nir Soffer wrote:
- yes, seems like it, but that's for another discussion
- when the vdsm wants to move some host to maintenance it contacts the
ha-agent which stops writing data to the storage, but it might be a
problem if the VG is changed while in maintenance, I don't think we
handle such situation
How do you ensure that the lvs are not deleted while you are using
them?
- obviously we're not handling this, otherwise the vdsm wouldn't be able
to disable those LVs
- I'm no expert on lvm, so I could use some advice on how to do it, so
how does the vdsm ensures that it's VG is not modified?
- I actually don't know this decision has been taken before I started
working on the code
Yes, that would solve the problem with vdsm, I hope Sandro or Martin
could explain why we use the vdsm's VG instead of creating our own...
--Jirka
1:03 a.m.
Il 05/05/2014 19:16, Jiri Moskovcak ha scritto:
I don't know why hosted engine metadata are in a different lv, I just create the
storage domain using vdsm and then call Martin's library for creating
the metadata. On NFS they're inside the storage domain, not on a different storage.
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
2:21 a.m.
On 05/05/2014 03:19 PM, Nir Soffer wrote:
I would rather not go this way (at least not for 3.5) as it's too much
code changes in hosted-engine. On the other hand the logic in vdsm seems
wrong because it's not complementary (disabling all LVs and then
enabling just some of them) and should be fixed anyway. This problem is
blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
--Jirka
4:37 a.m.
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
There is an asymmetry between activation and deactivation of an LV. A
mistakenly-active LV can cause data corruption. Making sure that this
does not happen is more important than a new feature.
We do not want to deactivate and then re-activating the same set of LVs.
That would be illogical. We intentionally deactivate LVs that are no
longer used on the specific host - that's important if a qemu died while
Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of
Vdsm-created storage domain. Let us discuss how your feature can be
implemented without this breach of ownership.
7:16 a.m.
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
- just out of a curiosity, how can mistakenly-active LV cause data
corruption? something like a stalled LV which refers to a volume which
doesn't exists anymore?
Ok, I agree that this should have been discussed with the storage team
at the design phase, so let's start from the beginning and try to come
up with a better solution.
My problem is that I need a storage for the hosted-engine data which
is accessible from all hosts. It seems logical to use the same physical
storage as we use for "the storage". For NFS it's just a file in
/rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do
you suggest to store such data in case of using lvm (iscsi in this
case). Can we use vdsm to set it up or do we have to duplicate the lvm
code and handle it our self?
Thanks,
Jirka
4:22 a.m.
On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote:
I think that for this to happen, we need to define a Vdsm verb that
creates a volume on a storage domain that is unrelated to any pool. Such
a verb is in planning; Federico, can its implementation be hasten in
favor of hosted engine?
On its own, this would not solve the problem of Vdsm deactivating all
unused LVs.
Jiri, could you describe why you keep your LV active, but not open?
Dan.
6:26 a.m.
On 05/12/2014 11:22 AM, Dan Kenigsberg wrote:
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine
2. install the engine into the VM
3. add the host to the engine
- this causes re-deploy of vdsm and deactivating the LVs
4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is
re-deployed, just before the HE services are started, but it won't fix
the problem if the vdsm is restarted
--Jirka
Dan.
7:04 a.m.
On 05/12/2014 01:45 PM, Dan Kenigsberg wrote:
- no, it runs as vdsm user and as it seems only root can activate LVs
and during the startup when it still has the root privs, it doesn't know
anything about the storage, the storage is connected on a client request
when it already doesn't have root privs
- we could use vdsm to activate the LV if it provides API which allows that
--Jirka
Dan.
10:27 a.m.
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
Vdsm uses "sudo" for these purposes.
- we could use vdsm to activate the LV if it provides API which
allows that
There's the undocumented prepareImage(), but please do not use it before
an ack by the storage team.
Dan.
2:16 a.m.
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
- that would solve the problem with privs, but I would really like to
avoid the situation where the vdsm disables the LVs as there is no
reason for disabling it as there can be situation where vdsm is down and
the VM with the engine is still running ok
- sure, I'm working on it with Nir now, I won't do anything without the
storage team approval ;)
Dan.
3944
days inactive
3954
days old
26 comments
6 participants
participants (6)
-
Dan Kenigsberg -
Itamar Heim -
Jiri Moskovcak -
Nir Soffer -
Saggi Mizrahi -
Sandro Bonazzola