vdsm disabling logical volumes
Greetings vdsm developers! While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs. So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs. I would be grateful for any hints. Thank you, Jirka
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes. Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag. Nir
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs, so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags.. --Jirka
Nir
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg? Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used. I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone. Federico, what do you think? Nir
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Jiri Moskovcak" <jmoskovc@redhat.com> Cc: devel@ovirt.org Sent: Monday, May 5, 2014 1:01:12 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg? Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
I personally don't like people piggy backing on domains. This kind of practice could cause other potential problems. IMHO there needs to be a way to mark extra volumes as such so that VDSM officially supports that kind of stuff.
Nir _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg? And how are you creating those lvs - I guess through vdsm? The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes
Greetings vdsm developers!
While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling: lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/ HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata --Jirka
So far the only suitable solution seems to be to change vdsm to only deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote:
----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc@redhat.com> > To: devel@ovirt.org > Sent: Sunday, May 4, 2014 8:08:33 PM > Subject: [ovirt-devel] vdsm disabling logical volumes > > Greetings vdsm developers! > > While working on adding ISCSI support to the hosted engine tools, I > ran > into a problem with vdms. It seems that when stopped vdsm deactivates > ALL logical volumes in it's volume group and when it starts it > reactivates only specific logical volumes. This is a problem for > hosted > engine tools as they create logical volumes in the same volume group > and > when vdsm deactivates the LVs the hosted engine tools don't have a way > to reactivate it, because the services drop the root permissions and > are > running as vdsm and apparently only root can activate LVs.
Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time? If you are not ensuring this, you will corrupt this vg sooner or later. When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data. How do you ensure that the lvs are not deleted while you are using them?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm? If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
--Jirka
> So far the > only suitable solution seems to be to change vdsm to only > deactivate/activate it's own LVs.
This sounds reasonable. You can add a list of hosted engine lv names and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
Another solution is to tag hosted engine lvs, and have vdsm ignore lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: devel@ovirt.org >> Sent: Sunday, May 4, 2014 8:08:33 PM >> Subject: [ovirt-devel] vdsm disabling logical volumes >> >> Greetings vdsm developers! >> >> While working on adding ISCSI support to the hosted engine tools, I >> ran >> into a problem with vdms. It seems that when stopped vdsm deactivates >> ALL logical volumes in it's volume group and when it starts it >> reactivates only specific logical volumes. This is a problem for >> hosted >> engine tools as they create logical volumes in the same volume group >> and >> when vdsm deactivates the LVs the hosted engine tools don't have a way >> to reactivate it, because the services drop the root permissions and >> are >> running as vdsm and apparently only root can activate LVs. > > Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
- yes, seems like it, but that's for another discussion
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
- when the vdsm wants to move some host to maintenance it contacts the ha-agent which stops writing data to the storage, but it might be a problem if the VG is changed while in maintenance, I don't think we handle such situation
How do you ensure that the lvs are not deleted while you are using them?
- obviously we're not handling this, otherwise the vdsm wouldn't be able to disable those LVs - I'm no expert on lvm, so I could use some advice on how to do it, so how does the vdsm ensures that it's VG is not modified?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
- I actually don't know this decision has been taken before I started working on the code
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
Yes, that would solve the problem with vdsm, I hope Sandro or Martin could explain why we use the vdsm's VG instead of creating our own... --Jirka
--Jirka
> >> So far the >> only suitable solution seems to be to change vdsm to only >> deactivate/activate it's own LVs. > > This sounds reasonable. You can add a list of hosted engine lv names > and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
> > Another solution is to tag hosted engine lvs, and have vdsm ignore > lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
Il 05/05/2014 19:16, Jiri Moskovcak ha scritto:
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc@redhat.com> > To: "Nir Soffer" <nsoffer@redhat.com> > Cc: devel@ovirt.org > Sent: Sunday, May 4, 2014 9:23:49 PM > Subject: Re: [ovirt-devel] vdsm disabling logical volumes > > On 05/04/2014 07:57 PM, Nir Soffer wrote: >> ----- Original Message ----- >>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>> To: devel@ovirt.org >>> Sent: Sunday, May 4, 2014 8:08:33 PM >>> Subject: [ovirt-devel] vdsm disabling logical volumes >>> >>> Greetings vdsm developers! >>> >>> While working on adding ISCSI support to the hosted engine tools, I >>> ran >>> into a problem with vdms. It seems that when stopped vdsm deactivates >>> ALL logical volumes in it's volume group and when it starts it >>> reactivates only specific logical volumes. This is a problem for >>> hosted >>> engine tools as they create logical volumes in the same volume group >>> and >>> when vdsm deactivates the LVs the hosted engine tools don't have a way >>> to reactivate it, because the services drop the root permissions and >>> are >>> running as vdsm and apparently only root can activate LVs. >> >> Can you describe what volumes are you creating, and why? > > We create hosted-engine.lockspace (for sanlock) and > hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
- yes, seems like it, but that's for another discussion
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
- when the vdsm wants to move some host to maintenance it contacts the ha-agent which stops writing data to the storage, but it might be a problem if the VG is changed while in maintenance, I don't think we handle such situation
How do you ensure that the lvs are not deleted while you are using them?
- obviously we're not handling this, otherwise the vdsm wouldn't be able to disable those LVs
- I'm no expert on lvm, so I could use some advice on how to do it, so how does the vdsm ensures that it's VG is not modified?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
- I actually don't know this decision has been taken before I started working on the code
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
Yes, that would solve the problem with vdsm, I hope Sandro or Martin could explain why we use the vdsm's VG instead of creating our own...
I don't know why hosted engine metadata are in a different lv, I just create the storage domain using vdsm and then call Martin's library for creating the metadata. On NFS they're inside the storage domain, not on a different storage.
--Jirka
--Jirka
> >> >>> So far the >>> only suitable solution seems to be to change vdsm to only >>> deactivate/activate it's own LVs. >> >> This sounds reasonable. You can add a list of hosted engine lv names >> and skip these volumes when deactivating vdsm volumes. > > - this sounds a bit suboptimal, vdsm already has list of it's LVs, so it > can just disable only LVs known to it, otherwise we would have to change > the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
> >> >> Another solution is to tag hosted engine lvs, and have vdsm ignore >> lvs that contains this tag. > > - this sounds good, because if we teach vdsm to ignore LVs with some tag > we can add new LVs in future without changing vdsm. This however applies > also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
> so it > depends on vdsm devels which solution they find better. I think the > solution without tags is better, because is simpler and others (like > hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/04/2014 07:57 PM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: devel@ovirt.org >> Sent: Sunday, May 4, 2014 8:08:33 PM >> Subject: [ovirt-devel] vdsm disabling logical volumes >> >> Greetings vdsm developers! >> >> While working on adding ISCSI support to the hosted engine tools, I >> ran >> into a problem with vdms. It seems that when stopped vdsm deactivates >> ALL logical volumes in it's volume group and when it starts it >> reactivates only specific logical volumes. This is a problem for >> hosted >> engine tools as they create logical volumes in the same volume group >> and >> when vdsm deactivates the LVs the hosted engine tools don't have a way >> to reactivate it, because the services drop the root permissions and >> are >> running as vdsm and apparently only root can activate LVs. > > Can you describe what volumes are you creating, and why?
We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it. --Jirka
--Jirka
> >> So far the >> only suitable solution seems to be to change vdsm to only >> deactivate/activate it's own LVs. > > This sounds reasonable. You can add a list of hosted engine lv names > and skip these volumes when deactivating vdsm volumes.
- this sounds a bit suboptimal, vdsm already has list of it's LVs, so it can just disable only LVs known to it, otherwise we would have to change the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
> > Another solution is to tag hosted engine lvs, and have vdsm ignore > lvs that contains this tag.
- this sounds good, because if we teach vdsm to ignore LVs with some tag we can add new LVs in future without changing vdsm. This however applies also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
so it depends on vdsm devels which solution they find better. I think the solution without tags is better, because is simpler and others (like hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote:
----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc@redhat.com> > To: "Nir Soffer" <nsoffer@redhat.com> > Cc: devel@ovirt.org > Sent: Sunday, May 4, 2014 9:23:49 PM > Subject: Re: [ovirt-devel] vdsm disabling logical volumes > > On 05/04/2014 07:57 PM, Nir Soffer wrote: >> ----- Original Message ----- >>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>> To: devel@ovirt.org >>> Sent: Sunday, May 4, 2014 8:08:33 PM >>> Subject: [ovirt-devel] vdsm disabling logical volumes >>> >>> Greetings vdsm developers! >>> >>> While working on adding ISCSI support to the hosted engine tools, I >>> ran >>> into a problem with vdms. It seems that when stopped vdsm >>> deactivates >>> ALL logical volumes in it's volume group and when it starts it >>> reactivates only specific logical volumes. This is a problem for >>> hosted >>> engine tools as they create logical volumes in the same volume group >>> and >>> when vdsm deactivates the LVs the hosted engine tools don't have a >>> way >>> to reactivate it, because the services drop the root permissions and >>> are >>> running as vdsm and apparently only root can activate LVs. >> >> Can you describe what volumes are you creating, and why? > > We create hosted-engine.lockspace (for sanlock) and > hosted-engine.metadata (keeps data about hosted engine hosts)
Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
--Jirka
--Jirka
> >> >>> So far the >>> only suitable solution seems to be to change vdsm to only >>> deactivate/activate it's own LVs. >> >> This sounds reasonable. You can add a list of hosted engine lv names >> and skip these volumes when deactivating vdsm volumes. > > - this sounds a bit suboptimal, vdsm already has list of it's LVs, so > it > can just disable only LVs known to it, otherwise we would have to > change > the list everytime we add some LV to the group
vdsm has a list of special lvs, that needs special treatment. Otherwise, it consider any other lv as owned by vdsm, and will deactivate them when they are not used.
I agree that this will create a dependency, but this can also be solved. For example, vdsm can load the list from a file installed by hosted engine, like the typical conf.d directories.
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
> >> >> Another solution is to tag hosted engine lvs, and have vdsm ignore >> lvs that contains this tag. > > - this sounds good, because if we teach vdsm to ignore LVs with some > tag > we can add new LVs in future without changing vdsm. This however > applies > also to the solution where vdsm only disables it's own LVs,
vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like this using some historic tags we keep (e.g. RHAT_*), but I'd rather add new tag with clear semantic than use some random historic value we have.
> so it > depends on vdsm devels which solution they find better. I think the > solution without tags is better, because is simpler and others (like > hosted-engine) can just createlv and don't bother with tags..
I think that a generic tag like OVIRT_IGNORE is an easy solution for everyone.
Federico, what do you think?
Nir
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 12:01 AM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: "Nir Soffer" <nsoffer@redhat.com> >> Cc: devel@ovirt.org >> Sent: Sunday, May 4, 2014 9:23:49 PM >> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >> >> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>> To: devel@ovirt.org >>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>> >>>> Greetings vdsm developers! >>>> >>>> While working on adding ISCSI support to the hosted engine tools, I >>>> ran >>>> into a problem with vdms. It seems that when stopped vdsm >>>> deactivates >>>> ALL logical volumes in it's volume group and when it starts it >>>> reactivates only specific logical volumes. This is a problem for >>>> hosted >>>> engine tools as they create logical volumes in the same volume group >>>> and >>>> when vdsm deactivates the LVs the hosted engine tools don't have a >>>> way >>>> to reactivate it, because the services drop the root permissions and >>>> are >>>> running as vdsm and apparently only root can activate LVs. >>> >>> Can you describe what volumes are you creating, and why? >> >> We create hosted-engine.lockspace (for sanlock) and >> hosted-engine.metadata (keeps data about hosted engine hosts) > > Do you create these lvs in every vdsm vg?
- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
> Is this part of the domain structure > used by hosted engine, or it has nothing to do with the storage domain?
- sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*. How: Disabling and enabling operations should be complementary. Why: To be less surprising. --Jirka
--Jirka
--Jirka
> >> >>> >>>> So far the >>>> only suitable solution seems to be to change vdsm to only >>>> deactivate/activate it's own LVs. >>> >>> This sounds reasonable. You can add a list of hosted engine lv names >>> and skip these volumes when deactivating vdsm volumes. >> >> - this sounds a bit suboptimal, vdsm already has list of it's LVs, so >> it >> can just disable only LVs known to it, otherwise we would have to >> change >> the list everytime we add some LV to the group > > vdsm has a list of special lvs, that needs special treatment. > Otherwise, > it > consider any other lv as owned by vdsm, and will deactivate them when > they > are > not used. > > I agree that this will create a dependency, but this can also be > solved. > For example, vdsm can load the list from a file installed by hosted > engine, > like the typical conf.d directories. >
- ok, this is something I actually don't have strong opinion about, for me adding a file with a list of LVs or tagging the logical volumes is almost the same, I just need a way to tell vdsm which LVs to ignore..
>> >>> >>> Another solution is to tag hosted engine lvs, and have vdsm ignore >>> lvs that contains this tag. >> >> - this sounds good, because if we teach vdsm to ignore LVs with some >> tag >> we can add new LVs in future without changing vdsm. This however >> applies >> also to the solution where vdsm only disables it's own LVs, > > vdsm own lvs are *all* lvs in vdsm vgs. We can implement something like > this > using some historic tags we keep (e.g. RHAT_*), but I'd rather add new > tag > with > clear semantic than use some random historic value we have. > >> so it >> depends on vdsm devels which solution they find better. I think the >> solution without tags is better, because is simpler and others (like >> hosted-engine) can just createlv and don't bother with tags.. > > I think that a generic tag like OVIRT_IGNORE is an easy solution for > everyone. > > Federico, what do you think? > > Nir >
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote:
----- Original Message ----- >From: "Jiri Moskovcak" <jmoskovc@redhat.com> >To: "Nir Soffer" <nsoffer@redhat.com> >Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >Mureinik" <amureini@redhat.com>, "Greg >Padgett" <gpadgett@redhat.com> >Sent: Monday, May 5, 2014 3:16:37 PM >Subject: Re: [ovirt-devel] vdsm disabling logical volumes > >On 05/05/2014 12:01 AM, Nir Soffer wrote: >>----- Original Message ----- >>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>To: "Nir Soffer" <nsoffer@redhat.com> >>>Cc: devel@ovirt.org >>>Sent: Sunday, May 4, 2014 9:23:49 PM >>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>> >>>On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>----- Original Message ----- >>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>To: devel@ovirt.org >>>>>Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>Subject: [ovirt-devel] vdsm disabling logical volumes >>>>> >>>>>Greetings vdsm developers! >>>>> >>>>>While working on adding ISCSI support to the hosted engine tools, I >>>>>ran >>>>>into a problem with vdms. It seems that when stopped vdsm >>>>>deactivates >>>>>ALL logical volumes in it's volume group and when it starts it >>>>>reactivates only specific logical volumes. This is a problem for >>>>>hosted >>>>>engine tools as they create logical volumes in the same volume group >>>>>and >>>>>when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>way >>>>>to reactivate it, because the services drop the root permissions and >>>>>are >>>>>running as vdsm and apparently only root can activate LVs. >>>> >>>>Can you describe what volumes are you creating, and why? >>> >>>We create hosted-engine.lockspace (for sanlock) and >>>hosted-engine.metadata (keeps data about hosted engine hosts) >> >>Do you create these lvs in every vdsm vg? > >- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
> >>Is this part of the domain structure >>used by hosted engine, or it has nothing to do with the storage domain? > >- sorry, I don't understand this question. How can I tell if it has >something to do with the storage domain? It's for storing data about >hosts set up to run the hosted-engine and data about state of engine and >the state of VM running the engine.
Can you tell us exactly what lvs you are creating, and on which vg?
And how are you creating those lvs - I guess through vdsm?
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature. We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind. Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com> Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 02:37 PM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: "Nir Soffer" <nsoffer@redhat.com> >> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >> Mureinik" <amureini@redhat.com>, "Greg >> Padgett" <gpadgett@redhat.com> >> Sent: Monday, May 5, 2014 3:16:37 PM >> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >> >> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>> Cc: devel@ovirt.org >>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>> >>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>> To: devel@ovirt.org >>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>> >>>>>> Greetings vdsm developers! >>>>>> >>>>>> While working on adding ISCSI support to the hosted engine tools, I >>>>>> ran >>>>>> into a problem with vdms. It seems that when stopped vdsm >>>>>> deactivates >>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>> reactivates only specific logical volumes. This is a problem for >>>>>> hosted >>>>>> engine tools as they create logical volumes in the same volume group >>>>>> and >>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>> way >>>>>> to reactivate it, because the services drop the root permissions and >>>>>> are >>>>>> running as vdsm and apparently only root can activate LVs. >>>>> >>>>> Can you describe what volumes are you creating, and why? >>>> >>>> We create hosted-engine.lockspace (for sanlock) and >>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>> >>> Do you create these lvs in every vdsm vg? >> >> - only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
>> >>> Is this part of the domain structure >>> used by hosted engine, or it has nothing to do with the storage domain? >> >> - sorry, I don't understand this question. How can I tell if it has >> something to do with the storage domain? It's for storing data about >> hosts set up to run the hosted-engine and data about state of engine and >> the state of VM running the engine. > > Can you tell us exactly what lvs you are creating, and on which vg? > > And how are you creating those lvs - I guess through vdsm? >
- no hosted-engine tools do that by calling:
lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", "-n", lv_name, vg_uuid]) ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
> The output of lvs command on a host with hosted engine installed will > help us to understand what you are doing, and then we can think more > clearly > what would be the best way to support this in vdsm.
The output of lvs: http://fpaste.org/99196/93619139/
HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self? Thanks, Jirka
On 05/07/2014 08:16 AM, Jiri Moskovcak wrote:
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message ----- > From: "Jiri Moskovcak" <jmoskovc@redhat.com> > To: "Nir Soffer" <nsoffer@redhat.com> > Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, > "Allon > Mureinik" <amureini@redhat.com>, "Greg > Padgett" <gpadgett@redhat.com> > Sent: Monday, May 5, 2014 3:44:21 PM > Subject: Re: [ovirt-devel] vdsm disabling logical volumes > > On 05/05/2014 02:37 PM, Nir Soffer wrote: >> ----- Original Message ----- >>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>> To: "Nir Soffer" <nsoffer@redhat.com> >>> Cc: devel@ovirt.org, "Federico Simoncelli" >>> <fsimonce@redhat.com>, "Allon >>> Mureinik" <amureini@redhat.com>, "Greg >>> Padgett" <gpadgett@redhat.com> >>> Sent: Monday, May 5, 2014 3:16:37 PM >>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>> >>> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>> ----- Original Message ----- >>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>>> Cc: devel@ovirt.org >>>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>> >>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>> ----- Original Message ----- >>>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>> To: devel@ovirt.org >>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>> >>>>>>> Greetings vdsm developers! >>>>>>> >>>>>>> While working on adding ISCSI support to the hosted engine >>>>>>> tools, I >>>>>>> ran >>>>>>> into a problem with vdms. It seems that when stopped vdsm >>>>>>> deactivates >>>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>>> reactivates only specific logical volumes. This is a >>>>>>> problem for >>>>>>> hosted >>>>>>> engine tools as they create logical volumes in the same >>>>>>> volume group >>>>>>> and >>>>>>> when vdsm deactivates the LVs the hosted engine tools don't >>>>>>> have a >>>>>>> way >>>>>>> to reactivate it, because the services drop the root >>>>>>> permissions and >>>>>>> are >>>>>>> running as vdsm and apparently only root can activate LVs. >>>>>> >>>>>> Can you describe what volumes are you creating, and why? >>>>> >>>>> We create hosted-engine.lockspace (for sanlock) and >>>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>>> >>>> Do you create these lvs in every vdsm vg? >>> >>> - only in the first vg created by vdsm while deploying >>> hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
>>> >>>> Is this part of the domain structure >>>> used by hosted engine, or it has nothing to do with the >>>> storage domain? >>> >>> - sorry, I don't understand this question. How can I tell if it >>> has >>> something to do with the storage domain? It's for storing data >>> about >>> hosts set up to run the hosted-engine and data about state of >>> engine and >>> the state of VM running the engine. >> >> Can you tell us exactly what lvs you are creating, and on which vg? >> >> And how are you creating those lvs - I guess through vdsm? >> > > - no hosted-engine tools do that by calling: > > lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > args=["lvm", "lvcreate", "-L", > str(size_bytes)+"B", > "-n", lv_name, vg_uuid]) > ..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
> >> The output of lvs command on a host with hosted engine installed >> will >> help us to understand what you are doing, and then we can think >> more >> clearly >> what would be the best way to support this in vdsm. > > The output of lvs: http://fpaste.org/99196/93619139/ > > HE created these two LVs: > ha_agent-hosted-engine.lockspace > ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self?
please note during your discussions that hosted engine VM should reside on a regular vdsm storage domain. we want it to be managed by the engine later on...
On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote:
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote:
----- Original Message ----- >From: "Jiri Moskovcak" <jmoskovc@redhat.com> >To: "Nir Soffer" <nsoffer@redhat.com> >Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >Mureinik" <amureini@redhat.com>, "Greg >Padgett" <gpadgett@redhat.com> >Sent: Monday, May 5, 2014 3:44:21 PM >Subject: Re: [ovirt-devel] vdsm disabling logical volumes > >On 05/05/2014 02:37 PM, Nir Soffer wrote: >>----- Original Message ----- >>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>To: "Nir Soffer" <nsoffer@redhat.com> >>>Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>Mureinik" <amureini@redhat.com>, "Greg >>>Padgett" <gpadgett@redhat.com> >>>Sent: Monday, May 5, 2014 3:16:37 PM >>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>> >>>On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>----- Original Message ----- >>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>To: "Nir Soffer" <nsoffer@redhat.com> >>>>>Cc: devel@ovirt.org >>>>>Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>> >>>>>On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>----- Original Message ----- >>>>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>To: devel@ovirt.org >>>>>>>Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>> >>>>>>>Greetings vdsm developers! >>>>>>> >>>>>>>While working on adding ISCSI support to the hosted engine tools, I >>>>>>>ran >>>>>>>into a problem with vdms. It seems that when stopped vdsm >>>>>>>deactivates >>>>>>>ALL logical volumes in it's volume group and when it starts it >>>>>>>reactivates only specific logical volumes. This is a problem for >>>>>>>hosted >>>>>>>engine tools as they create logical volumes in the same volume group >>>>>>>and >>>>>>>when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>way >>>>>>>to reactivate it, because the services drop the root permissions and >>>>>>>are >>>>>>>running as vdsm and apparently only root can activate LVs. >>>>>> >>>>>>Can you describe what volumes are you creating, and why? >>>>> >>>>>We create hosted-engine.lockspace (for sanlock) and >>>>>hosted-engine.metadata (keeps data about hosted engine hosts) >>>> >>>>Do you create these lvs in every vdsm vg? >>> >>>- only in the first vg created by vdsm while deploying hosted-engine
It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data.
>>> >>>>Is this part of the domain structure >>>>used by hosted engine, or it has nothing to do with the storage domain? >>> >>>- sorry, I don't understand this question. How can I tell if it has >>>something to do with the storage domain? It's for storing data about >>>hosts set up to run the hosted-engine and data about state of engine and >>>the state of VM running the engine. >> >>Can you tell us exactly what lvs you are creating, and on which vg? >> >>And how are you creating those lvs - I guess through vdsm? >> > >- no hosted-engine tools do that by calling: > >lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", > "-n", lv_name, vg_uuid]) >..
How do you ensure that another host is not modifying the same vg in the same time?
If you are not ensuring this, you will corrupt this vg sooner or later.
When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data.
How do you ensure that the lvs are not deleted while you are using them?
> >>The output of lvs command on a host with hosted engine installed will >>help us to understand what you are doing, and then we can think more >>clearly >>what would be the best way to support this in vdsm. > >The output of lvs: http://fpaste.org/99196/93619139/ > >HE created these two LVs: >ha_agent-hosted-engine.lockspace >ha_agent-hosted-engine.metadata
Why do you create these lvs on a vg owned by vdsm?
If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there.
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self?
I think that for this to happen, we need to define a Vdsm verb that creates a volume on a storage domain that is unrelated to any pool. Such a verb is in planning; Federico, can its implementation be hasten in favor of hosted engine? On its own, this would not solve the problem of Vdsm deactivating all unused LVs. Jiri, could you describe why you keep your LV active, but not open? Dan.
On 05/12/2014 11:22 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote:
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message -----
From: "Jiri Moskovcak" <jmoskovc@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes
On 05/05/2014 03:19 PM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: "Nir Soffer" <nsoffer@redhat.com> >> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >> Mureinik" <amureini@redhat.com>, "Greg >> Padgett" <gpadgett@redhat.com> >> Sent: Monday, May 5, 2014 3:44:21 PM >> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >> >> On 05/05/2014 02:37 PM, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>> Mureinik" <amureini@redhat.com>, "Greg >>>> Padgett" <gpadgett@redhat.com> >>>> Sent: Monday, May 5, 2014 3:16:37 PM >>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>> >>>> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>>>> Cc: devel@ovirt.org >>>>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>> >>>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>> To: devel@ovirt.org >>>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>> >>>>>>>> Greetings vdsm developers! >>>>>>>> >>>>>>>> While working on adding ISCSI support to the hosted engine tools, I >>>>>>>> ran >>>>>>>> into a problem with vdms. It seems that when stopped vdsm >>>>>>>> deactivates >>>>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>>>> reactivates only specific logical volumes. This is a problem for >>>>>>>> hosted >>>>>>>> engine tools as they create logical volumes in the same volume group >>>>>>>> and >>>>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>> way >>>>>>>> to reactivate it, because the services drop the root permissions and >>>>>>>> are >>>>>>>> running as vdsm and apparently only root can activate LVs. >>>>>>> >>>>>>> Can you describe what volumes are you creating, and why? >>>>>> >>>>>> We create hosted-engine.lockspace (for sanlock) and >>>>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>>>> >>>>> Do you create these lvs in every vdsm vg? >>>> >>>> - only in the first vg created by vdsm while deploying hosted-engine > > It seems that the hosted engine has single point of failure - the random > vg that contains hosted engine data. > >>>> >>>>> Is this part of the domain structure >>>>> used by hosted engine, or it has nothing to do with the storage domain? >>>> >>>> - sorry, I don't understand this question. How can I tell if it has >>>> something to do with the storage domain? It's for storing data about >>>> hosts set up to run the hosted-engine and data about state of engine and >>>> the state of VM running the engine. >>> >>> Can you tell us exactly what lvs you are creating, and on which vg? >>> >>> And how are you creating those lvs - I guess through vdsm? >>> >> >> - no hosted-engine tools do that by calling: >> >> lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >> stderr=subprocess.PIPE, >> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >> "-n", lv_name, vg_uuid]) >> .. > > How do you ensure that another host is not modifying the same vg in the > same time? > > If you are not ensuring this, you will corrupt this vg sooner or later. > > When a storage domain is detached from a host, for example when the host > is in maintenance mode, lvs on the shared storage may be deleted, > invalidating > the devices mapper maps for these devices. If you write to an lv with wrong > maps, you may be writing to an extent belonging to another lv, corrupting > that > lv data, or even worse corrupting the engine vg data. > > How do you ensure that the lvs are not deleted while you are using them? > >> >>> The output of lvs command on a host with hosted engine installed will >>> help us to understand what you are doing, and then we can think more >>> clearly >>> what would be the best way to support this in vdsm. >> >> The output of lvs: http://fpaste.org/99196/93619139/ >> >> HE created these two LVs: >> ha_agent-hosted-engine.lockspace >> ha_agent-hosted-engine.metadata > > Why do you create these lvs on a vg owned by vdsm? > > If you want total control of these lvs, I suggest that you create your own > vg and put what ever lvs you like there. >
I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self?
I think that for this to happen, we need to define a Vdsm verb that creates a volume on a storage domain that is unrelated to any pool. Such a verb is in planning; Federico, can its implementation be hasten in favor of hosted engine?
On its own, this would not solve the problem of Vdsm deactivating all unused LVs.
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this: 1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs - I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted --Jirka
Dan.
On Mon, May 12, 2014 at 01:26:33PM +0200, Jiri Moskovcak wrote:
On 05/12/2014 11:22 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote:
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote:
----- Original Message ----- >From: "Jiri Moskovcak" <jmoskovc@redhat.com> >To: "Nir Soffer" <nsoffer@redhat.com> >Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg >Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> >Sent: Wednesday, May 7, 2014 10:21:28 AM >Subject: Re: [ovirt-devel] vdsm disabling logical volumes > >On 05/05/2014 03:19 PM, Nir Soffer wrote: >>----- Original Message ----- >>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>To: "Nir Soffer" <nsoffer@redhat.com> >>>Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>Mureinik" <amureini@redhat.com>, "Greg >>>Padgett" <gpadgett@redhat.com> >>>Sent: Monday, May 5, 2014 3:44:21 PM >>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>> >>>On 05/05/2014 02:37 PM, Nir Soffer wrote: >>>>----- Original Message ----- >>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>To: "Nir Soffer" <nsoffer@redhat.com> >>>>>Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>>>Mureinik" <amureini@redhat.com>, "Greg >>>>>Padgett" <gpadgett@redhat.com> >>>>>Sent: Monday, May 5, 2014 3:16:37 PM >>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>> >>>>>On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>>>----- Original Message ----- >>>>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>To: "Nir Soffer" <nsoffer@redhat.com> >>>>>>>Cc: devel@ovirt.org >>>>>>>Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>>>Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>> >>>>>>>On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>>>----- Original Message ----- >>>>>>>>>From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>>>To: devel@ovirt.org >>>>>>>>>Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>>>Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>> >>>>>>>>>Greetings vdsm developers! >>>>>>>>> >>>>>>>>>While working on adding ISCSI support to the hosted engine tools, I >>>>>>>>>ran >>>>>>>>>into a problem with vdms. It seems that when stopped vdsm >>>>>>>>>deactivates >>>>>>>>>ALL logical volumes in it's volume group and when it starts it >>>>>>>>>reactivates only specific logical volumes. This is a problem for >>>>>>>>>hosted >>>>>>>>>engine tools as they create logical volumes in the same volume group >>>>>>>>>and >>>>>>>>>when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>>>way >>>>>>>>>to reactivate it, because the services drop the root permissions and >>>>>>>>>are >>>>>>>>>running as vdsm and apparently only root can activate LVs. >>>>>>>> >>>>>>>>Can you describe what volumes are you creating, and why? >>>>>>> >>>>>>>We create hosted-engine.lockspace (for sanlock) and >>>>>>>hosted-engine.metadata (keeps data about hosted engine hosts) >>>>>> >>>>>>Do you create these lvs in every vdsm vg? >>>>> >>>>>- only in the first vg created by vdsm while deploying hosted-engine >> >>It seems that the hosted engine has single point of failure - the random >>vg that contains hosted engine data. >> >>>>> >>>>>>Is this part of the domain structure >>>>>>used by hosted engine, or it has nothing to do with the storage domain? >>>>> >>>>>- sorry, I don't understand this question. How can I tell if it has >>>>>something to do with the storage domain? It's for storing data about >>>>>hosts set up to run the hosted-engine and data about state of engine and >>>>>the state of VM running the engine. >>>> >>>>Can you tell us exactly what lvs you are creating, and on which vg? >>>> >>>>And how are you creating those lvs - I guess through vdsm? >>>> >>> >>>- no hosted-engine tools do that by calling: >>> >>>lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >>> stderr=subprocess.PIPE, >>> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >>> "-n", lv_name, vg_uuid]) >>>.. >> >>How do you ensure that another host is not modifying the same vg in the >>same time? >> >>If you are not ensuring this, you will corrupt this vg sooner or later. >> >>When a storage domain is detached from a host, for example when the host >>is in maintenance mode, lvs on the shared storage may be deleted, >>invalidating >>the devices mapper maps for these devices. If you write to an lv with wrong >>maps, you may be writing to an extent belonging to another lv, corrupting >>that >>lv data, or even worse corrupting the engine vg data. >> >>How do you ensure that the lvs are not deleted while you are using them? >> >>> >>>>The output of lvs command on a host with hosted engine installed will >>>>help us to understand what you are doing, and then we can think more >>>>clearly >>>>what would be the best way to support this in vdsm. >>> >>>The output of lvs: http://fpaste.org/99196/93619139/ >>> >>>HE created these two LVs: >>>ha_agent-hosted-engine.lockspace >>>ha_agent-hosted-engine.metadata >> >>Why do you create these lvs on a vg owned by vdsm? >> >>If you want total control of these lvs, I suggest that you create your own >>vg and put what ever lvs you like there. >> > >I would rather not go this way (at least not for 3.5) as it's too much >code changes in hosted-engine. On the other hand the logic in vdsm seems >wrong because it's not complementary (disabling all LVs and then >enabling just some of them) and should be fixed anyway. This problem is >blocking one of our 3.5 features so I've created rhbz#1094657 to track it.
Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self?
I think that for this to happen, we need to define a Vdsm verb that creates a volume on a storage domain that is unrelated to any pool. Such a verb is in planning; Federico, can its implementation be hasten in favor of hosted engine?
On its own, this would not solve the problem of Vdsm deactivating all unused LVs.
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV. Dan.
On 05/12/2014 01:45 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 01:26:33PM +0200, Jiri Moskovcak wrote:
On 05/12/2014 11:22 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote:
On 05/07/2014 11:37 AM, Dan Kenigsberg wrote:
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote:
On 05/07/2014 09:28 AM, Nir Soffer wrote: > ----- Original Message ----- >> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >> To: "Nir Soffer" <nsoffer@redhat.com> >> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Greg >> Padgett" <gpadgett@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com> >> Sent: Wednesday, May 7, 2014 10:21:28 AM >> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >> >> On 05/05/2014 03:19 PM, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>> Mureinik" <amureini@redhat.com>, "Greg >>>> Padgett" <gpadgett@redhat.com> >>>> Sent: Monday, May 5, 2014 3:44:21 PM >>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>> >>>> On 05/05/2014 02:37 PM, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>>>> Cc: devel@ovirt.org, "Federico Simoncelli" <fsimonce@redhat.com>, "Allon >>>>>> Mureinik" <amureini@redhat.com>, "Greg >>>>>> Padgett" <gpadgett@redhat.com> >>>>>> Sent: Monday, May 5, 2014 3:16:37 PM >>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>> >>>>>> On 05/05/2014 12:01 AM, Nir Soffer wrote: >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>> To: "Nir Soffer" <nsoffer@redhat.com> >>>>>>>> Cc: devel@ovirt.org >>>>>>>> Sent: Sunday, May 4, 2014 9:23:49 PM >>>>>>>> Subject: Re: [ovirt-devel] vdsm disabling logical volumes >>>>>>>> >>>>>>>> On 05/04/2014 07:57 PM, Nir Soffer wrote: >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Jiri Moskovcak" <jmoskovc@redhat.com> >>>>>>>>>> To: devel@ovirt.org >>>>>>>>>> Sent: Sunday, May 4, 2014 8:08:33 PM >>>>>>>>>> Subject: [ovirt-devel] vdsm disabling logical volumes >>>>>>>>>> >>>>>>>>>> Greetings vdsm developers! >>>>>>>>>> >>>>>>>>>> While working on adding ISCSI support to the hosted engine tools, I >>>>>>>>>> ran >>>>>>>>>> into a problem with vdms. It seems that when stopped vdsm >>>>>>>>>> deactivates >>>>>>>>>> ALL logical volumes in it's volume group and when it starts it >>>>>>>>>> reactivates only specific logical volumes. This is a problem for >>>>>>>>>> hosted >>>>>>>>>> engine tools as they create logical volumes in the same volume group >>>>>>>>>> and >>>>>>>>>> when vdsm deactivates the LVs the hosted engine tools don't have a >>>>>>>>>> way >>>>>>>>>> to reactivate it, because the services drop the root permissions and >>>>>>>>>> are >>>>>>>>>> running as vdsm and apparently only root can activate LVs. >>>>>>>>> >>>>>>>>> Can you describe what volumes are you creating, and why? >>>>>>>> >>>>>>>> We create hosted-engine.lockspace (for sanlock) and >>>>>>>> hosted-engine.metadata (keeps data about hosted engine hosts) >>>>>>> >>>>>>> Do you create these lvs in every vdsm vg? >>>>>> >>>>>> - only in the first vg created by vdsm while deploying hosted-engine >>> >>> It seems that the hosted engine has single point of failure - the random >>> vg that contains hosted engine data. >>> >>>>>> >>>>>>> Is this part of the domain structure >>>>>>> used by hosted engine, or it has nothing to do with the storage domain? >>>>>> >>>>>> - sorry, I don't understand this question. How can I tell if it has >>>>>> something to do with the storage domain? It's for storing data about >>>>>> hosts set up to run the hosted-engine and data about state of engine and >>>>>> the state of VM running the engine. >>>>> >>>>> Can you tell us exactly what lvs you are creating, and on which vg? >>>>> >>>>> And how are you creating those lvs - I guess through vdsm? >>>>> >>>> >>>> - no hosted-engine tools do that by calling: >>>> >>>> lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, >>>> stderr=subprocess.PIPE, >>>> args=["lvm", "lvcreate", "-L", str(size_bytes)+"B", >>>> "-n", lv_name, vg_uuid]) >>>> .. >>> >>> How do you ensure that another host is not modifying the same vg in the >>> same time? >>> >>> If you are not ensuring this, you will corrupt this vg sooner or later. >>> >>> When a storage domain is detached from a host, for example when the host >>> is in maintenance mode, lvs on the shared storage may be deleted, >>> invalidating >>> the devices mapper maps for these devices. If you write to an lv with wrong >>> maps, you may be writing to an extent belonging to another lv, corrupting >>> that >>> lv data, or even worse corrupting the engine vg data. >>> >>> How do you ensure that the lvs are not deleted while you are using them? >>> >>>> >>>>> The output of lvs command on a host with hosted engine installed will >>>>> help us to understand what you are doing, and then we can think more >>>>> clearly >>>>> what would be the best way to support this in vdsm. >>>> >>>> The output of lvs: http://fpaste.org/99196/93619139/ >>>> >>>> HE created these two LVs: >>>> ha_agent-hosted-engine.lockspace >>>> ha_agent-hosted-engine.metadata >>> >>> Why do you create these lvs on a vg owned by vdsm? >>> >>> If you want total control of these lvs, I suggest that you create your own >>> vg and put what ever lvs you like there. >>> >> >> I would rather not go this way (at least not for 3.5) as it's too much >> code changes in hosted-engine. On the other hand the logic in vdsm seems >> wrong because it's not complementary (disabling all LVs and then >> enabling just some of them) and should be fixed anyway. This problem is >> blocking one of our 3.5 features so I've created rhbz#1094657 to track it. > > Can you elaborate on this? How should vdsm behave better, and why?
Sure. So far I didn't hear any reason why it behaves like this and it seems not logical to disable *all* and then enable just *some*.
How: Disabling and enabling operations should be complementary. Why: To be less surprising.
There is an asymmetry between activation and deactivation of an LV. A mistakenly-active LV can cause data corruption. Making sure that this does not happen is more important than a new feature.
- just out of a curiosity, how can mistakenly-active LV cause data corruption? something like a stalled LV which refers to a volume which doesn't exists anymore?
We do not want to deactivate and then re-activating the same set of LVs. That would be illogical. We intentionally deactivate LVs that are no longer used on the specific host - that's important if a qemu died while Vdsm was down, leaving a stale LV behind.
Design-wise, Vdsm would very much like to keep its ownership of Vdsm-created storage domain. Let us discuss how your feature can be implemented without this breach of ownership.
Ok, I agree that this should have been discussed with the storage team at the design phase, so let's start from the beginning and try to come up with a better solution. My problem is that I need a storage for the hosted-engine data which is accessible from all hosts. It seems logical to use the same physical storage as we use for "the storage". For NFS it's just a file in /rhev/data-center/mnt/<IP>\mountpoint/<UUID>/ha_agent/. So where/how do you suggest to store such data in case of using lvm (iscsi in this case). Can we use vdsm to set it up or do we have to duplicate the lvm code and handle it our self?
I think that for this to happen, we need to define a Vdsm verb that creates a volume on a storage domain that is unrelated to any pool. Such a verb is in planning; Federico, can its implementation be hasten in favor of hosted engine?
On its own, this would not solve the problem of Vdsm deactivating all unused LVs.
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs - we could use vdsm to activate the LV if it provides API which allows that --Jirka
Dan.
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team. Dan.
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- that would solve the problem with privs, but I would really like to avoid the situation where the vdsm disables the LVs as there is no reason for disabling it as there can be situation where vdsm is down and the VM with the engine is still running ok
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team.
- sure, I'm working on it with Nir now, I won't do anything without the storage team approval ;)
Dan.
On 05/13/2014 03:16 AM, Jiri Moskovcak wrote:
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
Jiri, could you describe why you keep your LV active, but not open?
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- that would solve the problem with privs, but I would really like to avoid the situation where the vdsm disables the LVs as there is no reason for disabling it as there can be situation where vdsm is down and the VM with the engine is still running ok
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team.
- sure, I'm working on it with Nir now, I won't do anything without the storage team approval ;)
Dan.
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
I'm missing something here - why doesn't VDSM knows about this LV? its a VM started by VDSM?
On 05/14/2014 02:57 AM, Itamar Heim wrote:
On 05/13/2014 03:16 AM, Jiri Moskovcak wrote:
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
> Jiri, could you describe why you keep your LV active, but not open? >
- the setup flow goes approximately like this:
1. create the LVs for the hosted-engine 2. install the engine into the VM 3. add the host to the engine - this causes re-deploy of vdsm and deactivating the LVs 4. start the ha-agent and ha-broker services which uses the LVs
- I guess we could move the creation of the LVs after the vdsm is re-deployed, just before the HE services are started, but it won't fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- that would solve the problem with privs, but I would really like to avoid the situation where the vdsm disables the LVs as there is no reason for disabling it as there can be situation where vdsm is down and the VM with the engine is still running ok
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team.
- sure, I'm working on it with Nir now, I won't do anything without the storage team approval ;)
Dan.
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
I'm missing something here - why doesn't VDSM knows about this LV? its a VM started by VDSM?
- these LVs are created by hosted-engine-setup and are not connected with the VM --Jirka
On 05/14/2014 03:02 AM, Jiri Moskovcak wrote:
On 05/14/2014 02:57 AM, Itamar Heim wrote:
On 05/13/2014 03:16 AM, Jiri Moskovcak wrote:
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
>> Jiri, could you describe why you keep your LV active, but not open? >> > > - the setup flow goes approximately like this: > > 1. create the LVs for the hosted-engine > 2. install the engine into the VM > 3. add the host to the engine > - this causes re-deploy of vdsm and deactivating the LVs > 4. start the ha-agent and ha-broker services which uses the LVs > > - I guess we could move the creation of the LVs after the vdsm is > re-deployed, just before the HE services are started, but it won't > fix the problem if the vdsm is restarted
It's not going to solve our design problem, but can the users of that LV activate it just before they open it? This way there's only a small window where vdsm can crash, restart, and deactivate the LV.
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- that would solve the problem with privs, but I would really like to avoid the situation where the vdsm disables the LVs as there is no reason for disabling it as there can be situation where vdsm is down and the VM with the engine is still running ok
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team.
- sure, I'm working on it with Nir now, I won't do anything without the storage team approval ;)
Dan.
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
I'm missing something here - why doesn't VDSM knows about this LV? its a VM started by VDSM?
- these LVs are created by hosted-engine-setup and are not connected with the VM
for the metadata/config/etc.?
On 05/14/2014 12:28 PM, Itamar Heim wrote:
On 05/14/2014 03:02 AM, Jiri Moskovcak wrote:
On 05/14/2014 02:57 AM, Itamar Heim wrote:
On 05/13/2014 03:16 AM, Jiri Moskovcak wrote:
On 05/12/2014 05:27 PM, Dan Kenigsberg wrote:
On Mon, May 12, 2014 at 02:04:05PM +0200, Jiri Moskovcak wrote:
>>> Jiri, could you describe why you keep your LV active, but not >>> open? >>> >> >> - the setup flow goes approximately like this: >> >> 1. create the LVs for the hosted-engine >> 2. install the engine into the VM >> 3. add the host to the engine >> - this causes re-deploy of vdsm and deactivating the LVs >> 4. start the ha-agent and ha-broker services which uses the LVs >> >> - I guess we could move the creation of the LVs after the vdsm is >> re-deployed, just before the HE services are started, but it won't >> fix the problem if the vdsm is restarted > > It's not going to solve our design problem, but can the users of > that LV > activate it just before they open it? This way there's only a small > window where vdsm can crash, restart, and deactivate the LV. >
- no, it runs as vdsm user and as it seems only root can activate LVs and during the startup when it still has the root privs, it doesn't know anything about the storage, the storage is connected on a client request when it already doesn't have root privs
Vdsm uses "sudo" for these purposes.
- that would solve the problem with privs, but I would really like to avoid the situation where the vdsm disables the LVs as there is no reason for disabling it as there can be situation where vdsm is down and the VM with the engine is still running ok
- we could use vdsm to activate the LV if it provides API which allows that
There's the undocumented prepareImage(), but please do not use it before an ack by the storage team.
- sure, I'm working on it with Nir now, I won't do anything without the storage team approval ;)
Dan.
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
I'm missing something here - why doesn't VDSM knows about this LV? its a VM started by VDSM?
- these LVs are created by hosted-engine-setup and are not connected with the VM
for the metadata/config/etc.?
- yes
participants (6)
-
Dan Kenigsberg -
Itamar Heim -
Jiri Moskovcak -
Nir Soffer -
Saggi Mizrahi -
Sandro Bonazzola