2:21 p.m.
Hi all,
Recently support for Ceph network disk landed in master. It its possible
now to start a vm using Ceph network disk or hot-plug/unplug such disk
using Cephx authentication.
However, to make it work, you must add the relevant Ceph secret to
libvirt manually, in the same way it is done in OpenStack deployment.
Our goal is to manage secrets automatically and use ephemeral (safer)
secrets.
The next patches in the Ceph topic [1], implement secret management in
the same way we manage storage domains or server connections:
The concept is - all hosts can use all secrets, so you can migrate a vm
using Ceph disk to any host in the cluster.
1. When host becomes up, we register the secrets associated with all the
current active domains with libvirt
2. When activating a domain, we register the secrets associated with the
new domain with libvirt
3. When deactivating a domain, we unregister the secrets associated with
the domain from libvirt
4. When moving host to maintenance, we clear all secrets
5. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep
stale or unneeded secrets on a host
This system seems to work, but Federico pointed few issues and suggested
a new (simpler?) approach.
In future libvirt version, libvirt will support the concept of transient
secrets so you can start a transient vm using secret without registering
the secret with libvirt before starting the vm. The secret will be
specified in the vm XML (for starting a vm) or disk XML (for hot-plug).
This will make our secret management system and APIs useless.
Managing state on multiple hosts is hard; we will probably have to deal
with nasty edge cases (e.g. lost messages, network errors), which may
lead to host with missing secret, which cannot run some vms. We probably
do this right for storage domains (after 8 years?), and we should not
assume that we are smarter and secret management will work in the first
try.
The new approach is to *not* manage state or multiple hosts. Instead,
send the required secrets only to the host that starting a vm or
hot-plugging a disk that need a libvirt secret:
1. When starting a vm, add the required secrets to the vm description.
On the host, vdsm will register these secrets with libvirt before
starting the vm.
2. When migrating a vm, add the required secrets to the vm description.
On the host, vdsm will send these secrets to the destination host,
and on the destination host, vdsm will register the secrets with libvirt
before starting the vm.
3. When hot-plugging a disk, send the secret if needed in the disk
description. On the host, vdsm will register the secrets with libvirt.
4. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep
stale or unneeded secrets on a host
5. We never unregister secrets, since they are ephemeral anyway.
6. Alternatively, we can implement secrets reference counting so when a vm
stops or disk is hot-unplugged we decrease the reference count on the
secrets associated with this vm/disk, and if no other vms need the
secret, we can unregister the secret from libvirt.
The new approach is simpler, if we avoid the fancy secret reference
counting. I believe we can get it merged in couple of weeks with help
from the virt team.
Please share your thoughts on these alternative solutions.
Thanks,
Nir
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
3:10 p.m.
Here are more details on the new approach.
A Ceph key is required only when starting a vm or hot-plugging a disk.
Once the operation is done, libvirt does not need the Ceph key any more.
A vm operation requiring a secret, will register a Ceph key using new
random UUID, and remove the libvirt secret as soon as the operation was
finished or failed.
This scheme does not require secret reference counting. If multiple vms
need the same Ceph key, we register it multiple times with libvirt,
using unique UUIDs.
This also avoid possible races when removing a libvirt secret in the
same time another vm is trying to add it, or updating secret usage id,
which is currently racy (you must remove the existing secret and
register a new one).
Positive flow:
- Engine adds required Ceph keys to vm description
- Vdsm register keys with libvirt, using new random UUID
- When the vm operation is done (vm started, disk hot-plugged), remove
the temporary secret (e.g. using try-finally)
Negative flows:
- Vm operation fails - temporary secret is unregistered in the finally
block
- Vdsm crash during the operation - temporary secret is removed when
vdsm starts again.
- Libvirt crash during the operation - secret removed since we use
ephemeral secrets
- Host crash during operation - same
- Libvirt fail to remove secret - we cannot handle this :-)
Flows
=====
Start vm
--------
- Engine add required secrets to vm description
- Vdsm register temporary secrets with libvirt
- When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm
----------
- Engine add required secrets to vm description
- Vdsm add secrets to the vm description sent to the destination
- On the destination, Vdsm register temporary secrets with libvirt
- On the destination, when vm is up or if operation failed, Vdsm remove
the temporary secret
Hot-plug disk
-------------
- Engine add secret to disk description
- Vdsm register temporary secret with libvirt
- When disk is successfully plugged, or if operation failed, Vdsm remove
the temporary secret
I think this is the correct direction, assuming that we can get it works
for migration - I have no idea on that part.
Nir
----- Original Message -----
From: "Nir Soffer" <nsoffer(a)redhat.com>
To: "devel" <devel(a)ovirt.org>
Cc: "Francesco Romani" <fromani(a)redhat.com>, "Federico
Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Adam Litke" <alitke(a)redhat.com>, "Allon
Mureinik" <amureini(a)redhat.com>, "Daniel Erez"
<derez(a)redhat.com>, "Michal Skrivanek" <mskrivan(a)redhat.com>,
"Eric Blake" <eblake(a)redhat.com>
Sent: Friday, June 12, 2015 2:21:46 PM
Subject: Libvirt secrets management - take 2
Hi all,
Recently support for Ceph network disk landed in master. It its possible
now to start a vm using Ceph network disk or hot-plug/unplug such disk
using Cephx authentication.
However, to make it work, you must add the relevant Ceph secret to
libvirt manually, in the same way it is done in OpenStack deployment.
Our goal is to manage secrets automatically and use ephemeral (safer)
secrets.
The next patches in the Ceph topic [1], implement secret management in
the same way we manage storage domains or server connections:
The concept is - all hosts can use all secrets, so you can migrate a vm
using Ceph disk to any host in the cluster.
1. When host becomes up, we register the secrets associated with all the
current active domains with libvirt
2. When activating a domain, we register the secrets associated with the
new domain with libvirt
3. When deactivating a domain, we unregister the secrets associated with
the domain from libvirt
4. When moving host to maintenance, we clear all secrets
5. When vdsm shutdown or starts, clear all secrets to ensure that we don't
keep
stale or unneeded secrets on a host
This system seems to work, but Federico pointed few issues and suggested
a new (simpler?) approach.
In future libvirt version, libvirt will support the concept of transient
secrets so you can start a transient vm using secret without registering
the secret with libvirt before starting the vm. The secret will be
specified in the vm XML (for starting a vm) or disk XML (for hot-plug).
This will make our secret management system and APIs useless.
Managing state on multiple hosts is hard; we will probably have to deal
with nasty edge cases (e.g. lost messages, network errors), which may
lead to host with missing secret, which cannot run some vms. We probably
do this right for storage domains (after 8 years?), and we should not
assume that we are smarter and secret management will work in the first
try.
The new approach is to *not* manage state or multiple hosts. Instead,
send the required secrets only to the host that starting a vm or
hot-plugging a disk that need a libvirt secret:
1. When starting a vm, add the required secrets to the vm description.
On the host, vdsm will register these secrets with libvirt before
starting the vm.
2. When migrating a vm, add the required secrets to the vm description.
On the host, vdsm will send these secrets to the destination host,
and on the destination host, vdsm will register the secrets with libvirt
before starting the vm.
3. When hot-plugging a disk, send the secret if needed in the disk
description. On the host, vdsm will register the secrets with libvirt.
4. When vdsm shutdown or starts, clear all secrets to ensure that we don't
keep
stale or unneeded secrets on a host
5. We never unregister secrets, since they are ephemeral anyway.
6. Alternatively, we can implement secrets reference counting so when a vm
stops or disk is hot-unplugged we decrease the reference count on the
secrets associated with this vm/disk, and if no other vms need the
secret, we can unregister the secret from libvirt.
The new approach is simpler, if we avoid the fancy secret reference
counting. I believe we can get it merged in couple of weeks with help
from the virt team.
Please share your thoughts on these alternative solutions.
Thanks,
Nir
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
4:52 p.m.
On 12/06/15 08:10 -0400, Nir Soffer wrote:
Here are more details on the new approach.
A Ceph key is required only when starting a vm or hot-plugging a disk.
Once the operation is done, libvirt does not need the Ceph key any more.
A vm operation requiring a secret, will register a Ceph key using new
random UUID, and remove the libvirt secret as soon as the operation was
finished or failed.
This scheme does not require secret reference counting. If multiple vms
need the same Ceph key, we register it multiple times with libvirt,
using unique UUIDs.
This also avoid possible races when removing a libvirt secret in the
same time another vm is trying to add it, or updating secret usage id,
which is currently racy (you must remove the existing secret and
register a new one).
I really like this new design. As long as you remove the secrets when
you're done with them then I am not concerned with having an
accumulation of untracked secrets. I am happy to get rid of the
complexity of secret reference counting.
Positive flow:
- Engine adds required Ceph keys to vm description
- Vdsm register keys with libvirt, using new random UUID
- When the vm operation is done (vm started, disk hot-plugged), remove
the temporary secret (e.g. using try-finally)
Negative flows:
- Vm operation fails - temporary secret is unregistered in the finally
block
- Vdsm crash during the operation - temporary secret is removed when
vdsm starts again.
- Libvirt crash during the operation - secret removed since we use
ephemeral secrets
- Host crash during operation - same
- Libvirt fail to remove secret - we cannot handle this :-)
What about the case where a storage domain becomes unavailable
(causing the VM to pause with -EIO). When the domainMonitor
reestablishes a connection to the domain would the secrets need to be
renewed?
Flows
=====
Start vm
--------
- Engine add required secrets to vm description
- Vdsm register temporary secrets with libvirt
- When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm
----------
- Engine add required secrets to vm description
- Vdsm add secrets to the vm description sent to the destination
- On the destination, Vdsm register temporary secrets with libvirt
- On the destination, when vm is up or if operation failed, Vdsm remove
the temporary secret
Hot-plug disk
-------------
- Engine add secret to disk description
- Vdsm register temporary secret with libvirt
- When disk is successfully plugged, or if operation failed, Vdsm remove
the temporary secret
I think this is the correct direction, assuming that we can get it works
for migration - I have no idea on that part.
+1 - This is a much improved design in my opinion.
----- Original Message -----
> From: "Nir Soffer" <nsoffer(a)redhat.com>
> To: "devel" <devel(a)ovirt.org>
> Cc: "Francesco Romani" <fromani(a)redhat.com>, "Federico
Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
> <danken(a)redhat.com>, "Adam Litke" <alitke(a)redhat.com>,
"Allon Mureinik" <amureini(a)redhat.com>, "Daniel Erez"
> <derez(a)redhat.com>, "Michal Skrivanek" <mskrivan(a)redhat.com>,
"Eric Blake" <eblake(a)redhat.com>
> Sent: Friday, June 12, 2015 2:21:46 PM
> Subject: Libvirt secrets management - take 2
>
> Hi all,
>
> Recently support for Ceph network disk landed in master. It its possible
> now to start a vm using Ceph network disk or hot-plug/unplug such disk
> using Cephx authentication.
>
> However, to make it work, you must add the relevant Ceph secret to
> libvirt manually, in the same way it is done in OpenStack deployment.
> Our goal is to manage secrets automatically and use ephemeral (safer)
> secrets.
>
> The next patches in the Ceph topic [1], implement secret management in
> the same way we manage storage domains or server connections:
>
> The concept is - all hosts can use all secrets, so you can migrate a vm
> using Ceph disk to any host in the cluster.
>
> 1. When host becomes up, we register the secrets associated with all the
> current active domains with libvirt
>
> 2. When activating a domain, we register the secrets associated with the
> new domain with libvirt
>
> 3. When deactivating a domain, we unregister the secrets associated with
> the domain from libvirt
>
> 4. When moving host to maintenance, we clear all secrets
>
> 5. When vdsm shutdown or starts, clear all secrets to ensure that we don't
> keep
> stale or unneeded secrets on a host
>
> This system seems to work, but Federico pointed few issues and suggested
> a new (simpler?) approach.
>
> In future libvirt version, libvirt will support the concept of transient
> secrets so you can start a transient vm using secret without registering
> the secret with libvirt before starting the vm. The secret will be
> specified in the vm XML (for starting a vm) or disk XML (for hot-plug).
> This will make our secret management system and APIs useless.
>
> Managing state on multiple hosts is hard; we will probably have to deal
> with nasty edge cases (e.g. lost messages, network errors), which may
> lead to host with missing secret, which cannot run some vms. We probably
> do this right for storage domains (after 8 years?), and we should not
> assume that we are smarter and secret management will work in the first
> try.
>
> The new approach is to *not* manage state or multiple hosts. Instead,
> send the required secrets only to the host that starting a vm or
> hot-plugging a disk that need a libvirt secret:
>
> 1. When starting a vm, add the required secrets to the vm description.
> On the host, vdsm will register these secrets with libvirt before
> starting the vm.
>
> 2. When migrating a vm, add the required secrets to the vm description.
> On the host, vdsm will send these secrets to the destination host,
> and on the destination host, vdsm will register the secrets with libvirt
> before starting the vm.
>
> 3. When hot-plugging a disk, send the secret if needed in the disk
> description. On the host, vdsm will register the secrets with libvirt.
>
> 4. When vdsm shutdown or starts, clear all secrets to ensure that we don't
> keep
> stale or unneeded secrets on a host
>
> 5. We never unregister secrets, since they are ephemeral anyway.
>
> 6. Alternatively, we can implement secrets reference counting so when a vm
> stops or disk is hot-unplugged we decrease the reference count on the
> secrets associated with this vm/disk, and if no other vms need the
> secret, we can unregister the secret from libvirt.
>
> The new approach is simpler, if we avoid the fancy secret reference
> counting. I believe we can get it merged in couple of weeks with help
> from the virt team.
>
> Please share your thoughts on these alternative solutions.
>
> Thanks,
> Nir
>
> [1]
> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
--
Adam Litke
12:14 a.m.
----- Original Message -----
From: "Adam Litke" <alitke(a)redhat.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "devel" <devel(a)ovirt.org>, "Francesco Romani"
<fromani(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
"Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
Sent: Saturday, June 13, 2015 4:52:19 PM
Subject: Re: Libvirt secrets management - take 2
On 12/06/15 08:10 -0400, Nir Soffer wrote:
>Here are more details on the new approach.
>
>A Ceph key is required only when starting a vm or hot-plugging a disk.
>Once the operation is done, libvirt does not need the Ceph key any more.
>A vm operation requiring a secret, will register a Ceph key using new
>random UUID, and remove the libvirt secret as soon as the operation was
>finished or failed.
>
>This scheme does not require secret reference counting. If multiple vms
>need the same Ceph key, we register it multiple times with libvirt,
>using unique UUIDs.
>
>This also avoid possible races when removing a libvirt secret in the
>same time another vm is trying to add it, or updating secret usage id,
>which is currently racy (you must remove the existing secret and
>register a new one).
I really like this new design. As long as you remove the secrets when
you're done with them then I am not concerned with having an
accumulation of untracked secrets. I am happy to get rid of the
complexity of secret reference counting.
>Positive flow:
>
>- Engine adds required Ceph keys to vm description
>- Vdsm register keys with libvirt, using new random UUID
>- When the vm operation is done (vm started, disk hot-plugged), remove
> the temporary secret (e.g. using try-finally)
>
>Negative flows:
>
>- Vm operation fails - temporary secret is unregistered in the finally
> block
>- Vdsm crash during the operation - temporary secret is removed when
> vdsm starts again.
>- Libvirt crash during the operation - secret removed since we use
> ephemeral secrets
>- Host crash during operation - same
>- Libvirt fail to remove secret - we cannot handle this :-)
What about the case where a storage domain becomes unavailable
(causing the VM to pause with -EIO). When the domainMonitor
reestablishes a connection to the domain would the secrets need to be
renewed?
We don't have domain monitoring for ceph disks. I don't think we tried
yet what happen when you block access to all ceph monitors, but I guess
that qemu can renew the connection if needed since it is holding the
authentication key.
>
>Flows
>=====
>
>Start vm
>--------
>- Engine add required secrets to vm description
>- Vdsm register temporary secrets with libvirt
>- When vm is up or if operation failed, Vdsm remove the temporary secret
>
>Migrate vm
>----------
>- Engine add required secrets to vm description
>- Vdsm add secrets to the vm description sent to the destination
>- On the destination, Vdsm register temporary secrets with libvirt
>- On the destination, when vm is up or if operation failed, Vdsm remove
> the temporary secret
On issue with migration - do we have to keep the same auth information as in the
original vm xml, or we can create the vm on the destination using different auth
xml?
In the original, xml, each disk may have auth element with secret uuid. This
secret must be defined in the destination libvirt so qemu can connect to the
disk on the destination machine.
Francesco? Eric?
>
>Hot-plug disk
>-------------
>- Engine add secret to disk description
>- Vdsm register temporary secret with libvirt
>- When disk is successfully plugged, or if operation failed, Vdsm remove
> the temporary secret
>
>I think this is the correct direction, assuming that we can get it works
>for migration - I have no idea on that part.
+1 - This is a much improved design in my opinion.
>
>----- Original Message -----
>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>> To: "devel" <devel(a)ovirt.org>
>> Cc: "Francesco Romani" <fromani(a)redhat.com>, "Federico
Simoncelli"
>> <fsimonce(a)redhat.com>, "Dan Kenigsberg"
>> <danken(a)redhat.com>, "Adam Litke" <alitke(a)redhat.com>,
"Allon Mureinik"
>> <amureini(a)redhat.com>, "Daniel Erez"
>> <derez(a)redhat.com>, "Michal Skrivanek"
<mskrivan(a)redhat.com>, "Eric Blake"
>> <eblake(a)redhat.com>
>> Sent: Friday, June 12, 2015 2:21:46 PM
>> Subject: Libvirt secrets management - take 2
>>
>> Hi all,
>>
>> Recently support for Ceph network disk landed in master. It its possible
>> now to start a vm using Ceph network disk or hot-plug/unplug such disk
>> using Cephx authentication.
>>
>> However, to make it work, you must add the relevant Ceph secret to
>> libvirt manually, in the same way it is done in OpenStack deployment.
>> Our goal is to manage secrets automatically and use ephemeral (safer)
>> secrets.
>>
>> The next patches in the Ceph topic [1], implement secret management in
>> the same way we manage storage domains or server connections:
>>
>> The concept is - all hosts can use all secrets, so you can migrate a vm
>> using Ceph disk to any host in the cluster.
>>
>> 1. When host becomes up, we register the secrets associated with all the
>> current active domains with libvirt
>>
>> 2. When activating a domain, we register the secrets associated with the
>> new domain with libvirt
>>
>> 3. When deactivating a domain, we unregister the secrets associated with
>> the domain from libvirt
>>
>> 4. When moving host to maintenance, we clear all secrets
>>
>> 5. When vdsm shutdown or starts, clear all secrets to ensure that we don't
>> keep
>> stale or unneeded secrets on a host
>>
>> This system seems to work, but Federico pointed few issues and suggested
>> a new (simpler?) approach.
>>
>> In future libvirt version, libvirt will support the concept of transient
>> secrets so you can start a transient vm using secret without registering
>> the secret with libvirt before starting the vm. The secret will be
>> specified in the vm XML (for starting a vm) or disk XML (for hot-plug).
>> This will make our secret management system and APIs useless.
>>
>> Managing state on multiple hosts is hard; we will probably have to deal
>> with nasty edge cases (e.g. lost messages, network errors), which may
>> lead to host with missing secret, which cannot run some vms. We probably
>> do this right for storage domains (after 8 years?), and we should not
>> assume that we are smarter and secret management will work in the first
>> try.
>>
>> The new approach is to *not* manage state or multiple hosts. Instead,
>> send the required secrets only to the host that starting a vm or
>> hot-plugging a disk that need a libvirt secret:
>>
>> 1. When starting a vm, add the required secrets to the vm description.
>> On the host, vdsm will register these secrets with libvirt before
>> starting the vm.
>>
>> 2. When migrating a vm, add the required secrets to the vm description.
>> On the host, vdsm will send these secrets to the destination host,
>> and on the destination host, vdsm will register the secrets with
>> libvirt
>> before starting the vm.
>>
>> 3. When hot-plugging a disk, send the secret if needed in the disk
>> description. On the host, vdsm will register the secrets with libvirt.
>>
>> 4. When vdsm shutdown or starts, clear all secrets to ensure that we don't
>> keep
>> stale or unneeded secrets on a host
>>
>> 5. We never unregister secrets, since they are ephemeral anyway.
>>
>> 6. Alternatively, we can implement secrets reference counting so when a vm
>> stops or disk is hot-unplugged we decrease the reference count on the
>> secrets associated with this vm/disk, and if no other vms need the
>> secret, we can unregister the secret from libvirt.
>>
>> The new approach is simpler, if we avoid the fancy secret reference
>> counting. I believe we can get it merged in couple of weeks with help
>> from the virt team.
>>
>> Please share your thoughts on these alternative solutions.
>>
>> Thanks,
>> Nir
>>
>> [1]
>> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
--
Adam Litke
4:08 p.m.
----- Original Message -----
From: "Nir Soffer" <nsoffer(a)redhat.com>
To: "Adam Litke" <alitke(a)redhat.com>
Cc: "devel" <devel(a)ovirt.org>, "Francesco Romani"
<fromani(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
"Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
Sent: Saturday, June 13, 2015 11:14:03 PM
Subject: Re: Libvirt secrets management - take 2
[...]
> >Flows
> >=====
> >
> >Start vm
> >--------
> >- Engine add required secrets to vm description
> >- Vdsm register temporary secrets with libvirt
> >- When vm is up or if operation failed, Vdsm remove the temporary secret
> >
> >Migrate vm
> >----------
> >- Engine add required secrets to vm description
> >- Vdsm add secrets to the vm description sent to the destination
> >- On the destination, Vdsm register temporary secrets with libvirt
> >- On the destination, when vm is up or if operation failed, Vdsm remove
> > the temporary secret
On issue with migration - do we have to keep the same auth information as in
the original vm xml, or we can create the vm on the destination using different
auth xml?
I don't have an answer for this. Need to check and play with this a bit.
> >Hot-plug disk
> >-------------
> >- Engine add secret to disk description
> >- Vdsm register temporary secret with libvirt
> >- When disk is successfully plugged, or if operation failed, Vdsm remove
> > the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret
to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has
forgorgotten the secret, then we need get it back again from Engine and pass it through
the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for
updates,
and consider the VM migrated only after source exited correctly.
I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri)
- source VDSM sends migrationCreate to destination VDSM
- once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start
the migration
- source VDSM monitors the migration, report progress, updates the downtime
- migration fails: source VDSM automatically send VM.destroy to destination VDSM
- migration succeeds: source VDSM destroy source VM, execution now continues on
destination VM
so far, Engine talks with destination VM only if migration completed succesfully (unless
I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate verb
This is only the very first thought of mine, we can probably think something cleaner.
Bests,
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
6:06 p.m.
----- Original Message -----
From: "Francesco Romani" <fromani(a)redhat.com>
To: "devel" <devel(a)ovirt.org>
Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico Simoncelli"
<fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
"Daniel Erez" <derez(a)redhat.com>, "Michal Skrivanek"
<mskrivan(a)redhat.com>, "Eric Blake" <eblake(a)redhat.com>, "Nir
Soffer" <nsoffer(a)redhat.com>
Sent: Monday, June 15, 2015 4:08:09 PM
Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
> From: "Nir Soffer" <nsoffer(a)redhat.com>
> To: "Adam Litke" <alitke(a)redhat.com>
> Cc: "devel" <devel(a)ovirt.org>, "Francesco Romani"
<fromani(a)redhat.com>,
> "Federico Simoncelli" <fsimonce(a)redhat.com>,
> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
> <amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
> "Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
> Sent: Saturday, June 13, 2015 11:14:03 PM
> Subject: Re: Libvirt secrets management - take 2
[...]
> > >Flows
> > >=====
> > >
> > >Start vm
> > >--------
> > >- Engine add required secrets to vm description
> > >- Vdsm register temporary secrets with libvirt
> > >- When vm is up or if operation failed, Vdsm remove the temporary secret
> > >
> > >Migrate vm
> > >----------
> > >- Engine add required secrets to vm description
> > >- Vdsm add secrets to the vm description sent to the destination
> > >- On the destination, Vdsm register temporary secrets with libvirt
> > >- On the destination, when vm is up or if operation failed, Vdsm remove
> > > the temporary secret
>
> On issue with migration - do we have to keep the same auth information as
> in
> the original vm xml, or we can create the vm on the destination using
> different
> auth xml?
I don't have an answer for this. Need to check and play with this a bit.
> > >Hot-plug disk
> > >-------------
> > >- Engine add secret to disk description
> > >- Vdsm register temporary secret with libvirt
> > >- When disk is successfully plugged, or if operation failed, Vdsm remove
> > > the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register
this secret
to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and
if it has
forgorgotten the secret, then we need get it back again from Engine and pass
it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls
source VM for updates,
and consider the VM migrated only after source exited correctly.
I don't recall the fine details engine side (need to check), but on VDSM
side:
- source VDSM receives VM.migrate(destination_uri)
- source VDSM sends migrationCreate to destination VDSM
- once destination VDSM created special-purpose VM, source VDSM instructs
libvirt to start the migration
- source VDSM monitors the migration, report progress, updates the downtime
- migration fails: source VDSM automatically send VM.destroy to destination
VDSM
- migration succeeds: source VDSM destroy source VM, execution now continues
on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
If this special vm (waiting for migration) is connected to all the disks, then
we only need to register the secrets sent from the source in migrationCreate,
start the special vm, and one it is running, delete the secrets.
so far, Engine talks with destination VM only if migration completed
succesfully (unless I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate
verb
This is only the very first thought of mine, we can probably think something
cleaner.
Bests,
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
6:16 p.m.
----- Original Message -----
From: "Nir Soffer" <nsoffer(a)redhat.com>
To: "Francesco Romani" <fromani(a)redhat.com>
Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan
Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
"Michal
Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
Sent: Monday, June 15, 2015 5:06:09 PM
Subject: Re: Libvirt secrets management - take 2
> - source VDSM receives VM.migrate(destination_uri)
> - source VDSM sends migrationCreate to destination VDSM
> - once destination VDSM created special-purpose VM, source VDSM instructs
> libvirt to start the migration
> - source VDSM monitors the migration, report progress, updates the downtime
> - migration fails: source VDSM automatically send VM.destroy to destination
> VDSM
> - migration succeeds: source VDSM destroy source VM, execution now
> continues
> on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only
"special" thing here
(to partially amend my own wording) is the reported status of the VM itself.
If this special vm (waiting for migration) is connected to all the
disks,
then we only need to register the secrets sent from the source in migrationCreate,
start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner
approach.
To have source VM act like a passthrough feels a little funny, but probably this is
the only thing we can do give the architecture.
So, changes needed on virt flows:
- in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets
must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards
- in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most
of the infrastructure we have to create VM. It is little different from just create
a VM from Engine.
- in 'migration destination' flow, it doesn't look different from simple VM
creation.
we should just drop all the secrets once the disks are attached (or failed to attach).
At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :)
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
6:30 p.m.
On 15 Jun 2015, at 17:16, Francesco Romani wrote:
----- Original Message -----
> From: "Nir Soffer" <nsoffer(a)redhat.com>
> To: "Francesco Romani" <fromani(a)redhat.com>
> Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan
> Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
"Michal
> Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
> Sent: Monday, June 15, 2015 5:06:09 PM
> Subject: Re: Libvirt secrets management - take 2
>> - source VDSM receives VM.migrate(destination_uri)
>> - source VDSM sends migrationCreate to destination VDSM
>> - once destination VDSM created special-purpose VM, source VDSM instructs
>> libvirt to start the migration
>> - source VDSM monitors the migration, report progress, updates the downtime
>> - migration fails: source VDSM automatically send VM.destroy to destination
>> VDSM
>> - migration succeeds: source VDSM destroy source VM, execution now
>> continues
>> on destination VM
>
> Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only
"special" thing here
(to partially amend my own wording) is the reported status of the VM itself.
> If this special vm (waiting for migration) is connected to all the disks,
> then we only need to register the secrets sent from the source in migrationCreate,
> start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner
approach.
To have source VM act like a passthrough feels a little funny, but probably this is
the only thing we can do give the architecture.
So, changes needed on virt flows:
- in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets
must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards
- in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most
of the infrastructure we have to create VM. It is little different from just create
a VM from Engine.
- in 'migration destination' flow, it doesn't look different from simple VM
creation.
we should just drop all the secrets once the disks are attached (or failed to attach).
At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :)
Could it not be handled by lower layer? I suppose it has some kind of token or something?
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
6:33 p.m.
----- Original Message -----
From: "Michal Skrivanek" <mskrivan(a)redhat.com>
To: "Francesco Romani" <fromani(a)redhat.com>, "Nir Soffer"
<nsoffer(a)redhat.com>
Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan
Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>, "Eric
Blake" <eblake(a)redhat.com>
Sent: Monday, June 15, 2015 6:30:00 PM
Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:16, Francesco Romani wrote:
> ----- Original Message -----
>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>> To: "Francesco Romani" <fromani(a)redhat.com>
>> Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico
>> Simoncelli" <fsimonce(a)redhat.com>, "Dan
>> Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>,
>> "Daniel Erez" <derez(a)redhat.com>, "Michal
>> Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
>> Sent: Monday, June 15, 2015 5:06:09 PM
>> Subject: Re: Libvirt secrets management - take 2
>
>>> - source VDSM receives VM.migrate(destination_uri)
>>> - source VDSM sends migrationCreate to destination VDSM
>>> - once destination VDSM created special-purpose VM, source VDSM instructs
>>> libvirt to start the migration
>>> - source VDSM monitors the migration, report progress, updates the
>>> downtime
>>> - migration fails: source VDSM automatically send VM.destroy to
>>> destination
>>> VDSM
>>> - migration succeeds: source VDSM destroy source VM, execution now
>>> continues
>>> on destination VM
>>
>> Ok, when we create the special purpose vm, do we include all the devices?
>
> Sure. It has to be (actually: is) a clone of he source VM. Actually the
> only "special" thing here
> (to partially amend my own wording) is the reported status of the VM
> itself.
>
>> If this special vm (waiting for migration) is connected to all the disks,
>> then we only need to register the secrets sent from the source in
>> migrationCreate,
>> start the special vm, and one it is running, delete the secrets.
>
> Indeed. I just wonder, since we're still talking about design, if there is
> a cleaner
> approach.
>
> To have source VM act like a passthrough feels a little funny, but probably
> this is
> the only thing we can do give the architecture.
>
> So, changes needed on virt flows:
>
> - in VM.migrate (public API, Engine->VDSM) support secrets passing. These
> secrets
> must not be stored, but passed verbatim to VM.migrationCreate and forgotten
> afterwards
> - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse
> most
> of the infrastructure we have to create VM. It is little different from
> just create
> a VM from Engine.
> - in 'migration destination' flow, it doesn't look different from simple
VM
> creation.
> we should just drop all the secrets once the disks are attached (or failed
> to attach).
>
> At glance looks simple, should be quick and easy, unless I'm missing some
> pitfalls :)
Could it not be handled by lower layer? I suppose it has some kind of token
or something?
What do you mean?
6:45 p.m.
On 15 Jun 2015, at 17:33, Nir Soffer wrote:
----- Original Message -----
> From: "Michal Skrivanek" <mskrivan(a)redhat.com>
> To: "Francesco Romani" <fromani(a)redhat.com>, "Nir Soffer"
<nsoffer(a)redhat.com>
> Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico Simoncelli" <fsimonce(a)redhat.com>,
"Dan
> Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>, "Eric
> Blake" <eblake(a)redhat.com>
> Sent: Monday, June 15, 2015 6:30:00 PM
> Subject: Re: Libvirt secrets management - take 2
>
>
> On 15 Jun 2015, at 17:16, Francesco Romani wrote:
>
>> ----- Original Message -----
>>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>>> To: "Francesco Romani" <fromani(a)redhat.com>
>>> Cc: "devel" <devel(a)ovirt.org>, "Adam Litke"
<alitke(a)redhat.com>, "Federico
>>> Simoncelli" <fsimonce(a)redhat.com>, "Dan
>>> Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>,
>>> "Daniel Erez" <derez(a)redhat.com>, "Michal
>>> Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
>>> Sent: Monday, June 15, 2015 5:06:09 PM
>>> Subject: Re: Libvirt secrets management - take 2
>>
>>>> - source VDSM receives VM.migrate(destination_uri)
>>>> - source VDSM sends migrationCreate to destination VDSM
>>>> - once destination VDSM created special-purpose VM, source VDSM
instructs
>>>> libvirt to start the migration
>>>> - source VDSM monitors the migration, report progress, updates the
>>>> downtime
>>>> - migration fails: source VDSM automatically send VM.destroy to
>>>> destination
>>>> VDSM
>>>> - migration succeeds: source VDSM destroy source VM, execution now
>>>> continues
>>>> on destination VM
>>>
>>> Ok, when we create the special purpose vm, do we include all the devices?
>>
>> Sure. It has to be (actually: is) a clone of he source VM. Actually the
>> only "special" thing here
>> (to partially amend my own wording) is the reported status of the VM
>> itself.
>>
>>> If this special vm (waiting for migration) is connected to all the disks,
>>> then we only need to register the secrets sent from the source in
>>> migrationCreate,
>>> start the special vm, and one it is running, delete the secrets.
>>
>> Indeed. I just wonder, since we're still talking about design, if there is
>> a cleaner
>> approach.
>>
>> To have source VM act like a passthrough feels a little funny, but probably
>> this is
>> the only thing we can do give the architecture.
>>
>> So, changes needed on virt flows:
>>
>> - in VM.migrate (public API, Engine->VDSM) support secrets passing. These
>> secrets
>> must not be stored, but passed verbatim to VM.migrationCreate and forgotten
>> afterwards
>> - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse
>> most
>> of the infrastructure we have to create VM. It is little different from
>> just create
>> a VM from Engine.
>> - in 'migration destination' flow, it doesn't look different from
simple VM
>> creation.
>> we should just drop all the secrets once the disks are attached (or failed
>> to attach).
>>
>> At glance looks simple, should be quick and easy, unless I'm missing some
>> pitfalls :)
>
> Could it not be handled by lower layer? I suppose it has some kind of token
> or something?
What do you mean?
do we have to manage the connection? If qemu lose the connection we can't
re-establish?
I understood that libvirt keeps the secret in memory - is that correct? If so then it can
send it to the destination, right?
And what do we do about libvirt restarts?
I missed the initial discussions so please ignore me if i'm completely off....
6:20 p.m.
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
> From: "Francesco Romani" <fromani(a)redhat.com>
> To: "devel" <devel(a)ovirt.org>
> Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico Simoncelli"
<fsimonce(a)redhat.com>, "Dan Kenigsberg"
> <danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
"Daniel Erez" <derez(a)redhat.com>, "Michal Skrivanek"
> <mskrivan(a)redhat.com>, "Eric Blake" <eblake(a)redhat.com>,
"Nir Soffer" <nsoffer(a)redhat.com>
> Sent: Monday, June 15, 2015 4:08:09 PM
> Subject: Re: Libvirt secrets management - take 2
>
> ----- Original Message -----
>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>> To: "Adam Litke" <alitke(a)redhat.com>
>> Cc: "devel" <devel(a)ovirt.org>, "Francesco Romani"
<fromani(a)redhat.com>,
>> "Federico Simoncelli" <fsimonce(a)redhat.com>,
>> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon Mureinik"
>> <amureini(a)redhat.com>, "Daniel Erez" <derez(a)redhat.com>,
>> "Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
>> Sent: Saturday, June 13, 2015 11:14:03 PM
>> Subject: Re: Libvirt secrets management - take 2
>
> [...]
>>>> Flows
>>>> =====
>>>>
>>>> Start vm
>>>> --------
>>>> - Engine add required secrets to vm description
>>>> - Vdsm register temporary secrets with libvirt
>>>> - When vm is up or if operation failed, Vdsm remove the temporary secret
>>>>
>>>> Migrate vm
>>>> ----------
>>>> - Engine add required secrets to vm description
>>>> - Vdsm add secrets to the vm description sent to the destination
>>>> - On the destination, Vdsm register temporary secrets with libvirt
>>>> - On the destination, when vm is up or if operation failed, Vdsm remove
>>>> the temporary secret
>>
>> On issue with migration - do we have to keep the same auth information as
>> in
>> the original vm xml, or we can create the vm on the destination using
>> different
>> auth xml?
>
> I don't have an answer for this. Need to check and play with this a bit.
>
>>>> Hot-plug disk
>>>> -------------
>>>> - Engine add secret to disk description
>>>> - Vdsm register temporary secret with libvirt
>>>> - When disk is successfully plugged, or if operation failed, Vdsm remove
>>>> the temporary secret
>
> OK, what if this VM is migrate afterwards? destination VDSM needs to register
> this secret
> to libvirt.
>
> The fact here is that is the source VDSM that orchestrates the migration, and
> if it has
> forgorgotten the secret, then we need get it back again from Engine and pass
> it through the destination VM.
>
> On migration, it is the source VM which orchestrates the flow. Engine polls
> source VM for updates,
> and consider the VM migrated only after source exited correctly.
> I don't recall the fine details engine side (need to check), but on VDSM
> side:
>
> - source VDSM receives VM.migrate(destination_uri)
> - source VDSM sends migrationCreate to destination VDSM
> - once destination VDSM created special-purpose VM, source VDSM instructs
> libvirt to start the migration
> - source VDSM monitors the migration, report progress, updates the downtime
> - migration fails: source VDSM automatically send VM.destroy to destination
> VDSM
> - migration succeeds: source VDSM destroy source VM, execution now continues
> on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an
explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in
migrationCreate,
start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
>
> so far, Engine talks with destination VM only if migration completed
> succesfully (unless I'm missing something :))
>
> so we need to pass back again all the needed secrets as part of VM.migrate
> verb
>
> This is only the very first thought of mine, we can probably think something
> cleaner.
>
> Bests,
>
> --
> Francesco Romani
> RedHat Engineering Virtualization R & D
> Phone: 8261328
> IRC: fromani
>
6:57 p.m.
----- Original Message -----
From: "Michal Skrivanek" <mskrivan(a)redhat.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "Francesco Romani" <fromani(a)redhat.com>, "devel"
<devel(a)ovirt.org>, "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
"Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
Sent: Monday, June 15, 2015 6:20:34 PM
Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
>
>
> ----- Original Message -----
>> From: "Francesco Romani" <fromani(a)redhat.com>
>> To: "devel" <devel(a)ovirt.org>
>> Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli"
>> <fsimonce(a)redhat.com>, "Dan Kenigsberg"
>> <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez"
>> <derez(a)redhat.com>, "Michal Skrivanek"
>> <mskrivan(a)redhat.com>, "Eric Blake" <eblake(a)redhat.com>,
"Nir Soffer"
>> <nsoffer(a)redhat.com>
>> Sent: Monday, June 15, 2015 4:08:09 PM
>> Subject: Re: Libvirt secrets management - take 2
>>
>> ----- Original Message -----
>>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>>> To: "Adam Litke" <alitke(a)redhat.com>
>>> Cc: "devel" <devel(a)ovirt.org>, "Francesco Romani"
<fromani(a)redhat.com>,
>>> "Federico Simoncelli" <fsimonce(a)redhat.com>,
>>> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon
Mureinik"
>>> <amureini(a)redhat.com>, "Daniel Erez"
<derez(a)redhat.com>,
>>> "Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric
Blake"
>>> <eblake(a)redhat.com>
>>> Sent: Saturday, June 13, 2015 11:14:03 PM
>>> Subject: Re: Libvirt secrets management - take 2
>>
>> [...]
>>>>> Flows
>>>>> =====
>>>>>
>>>>> Start vm
>>>>> --------
>>>>> - Engine add required secrets to vm description
>>>>> - Vdsm register temporary secrets with libvirt
>>>>> - When vm is up or if operation failed, Vdsm remove the temporary
>>>>> secret
>>>>>
>>>>> Migrate vm
>>>>> ----------
>>>>> - Engine add required secrets to vm description
>>>>> - Vdsm add secrets to the vm description sent to the destination
>>>>> - On the destination, Vdsm register temporary secrets with libvirt
>>>>> - On the destination, when vm is up or if operation failed, Vdsm
remove
>>>>> the temporary secret
>>>
>>> On issue with migration - do we have to keep the same auth information as
>>> in
>>> the original vm xml, or we can create the vm on the destination using
>>> different
>>> auth xml?
>>
>> I don't have an answer for this. Need to check and play with this a bit.
>>
>>>>> Hot-plug disk
>>>>> -------------
>>>>> - Engine add secret to disk description
>>>>> - Vdsm register temporary secret with libvirt
>>>>> - When disk is successfully plugged, or if operation failed, Vdsm
>>>>> remove
>>>>> the temporary secret
>>
>> OK, what if this VM is migrate afterwards? destination VDSM needs to
>> register
>> this secret
>> to libvirt.
>>
>> The fact here is that is the source VDSM that orchestrates the migration,
>> and
>> if it has
>> forgorgotten the secret, then we need get it back again from Engine and
>> pass
>> it through the destination VM.
>>
>> On migration, it is the source VM which orchestrates the flow. Engine
>> polls
>> source VM for updates,
>> and consider the VM migrated only after source exited correctly.
>> I don't recall the fine details engine side (need to check), but on VDSM
>> side:
>>
>> - source VDSM receives VM.migrate(destination_uri)
>> - source VDSM sends migrationCreate to destination VDSM
>> - once destination VDSM created special-purpose VM, source VDSM instructs
>> libvirt to start the migration
>> - source VDSM monitors the migration, report progress, updates the
>> downtime
>> - migration fails: source VDSM automatically send VM.destroy to
>> destination
>> VDSM
>> - migration succeeds: source VDSM destroy source VM, execution now
>> continues
>> on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an
explicit destroy after the migration is deemed successful in the engine too
>
> Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
>
> If this special vm (waiting for migration) is connected to all the disks,
> then
it is (it goes though preparePaths as any other VM creation)
> we only need to register the secrets sent from the source in
> migrationCreate,
> start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts.
- qemu got the secret key and connected to the ceph disks.
- libvirt knew about the secret when starting qemu/ hot-plugging a disk, but
now it does not know anything about the secret, except the secret uuid
in the <disk> <auth> element.
- vdsm knows about the auth dicts in the disk devices, but does not know the
key (the key cannot be persisted)
So engine will have to send the secrets again, so vdsm can send them to the destination.
>
>>
>> so far, Engine talks with destination VM only if migration completed
>> succesfully (unless I'm missing something :))
>>
>> so we need to pass back again all the needed secrets as part of VM.migrate
>> verb
>>
>> This is only the very first thought of mine, we can probably think
>> something
>> cleaner.
>>
>> Bests,
>>
>> --
>> Francesco Romani
>> RedHat Engineering Virtualization R & D
>> Phone: 8261328
>> IRC: fromani
>>
6:57 p.m.
On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer(a)redhat.com> wrote:
----- Original Message -----
> From: "Michal Skrivanek" <mskrivan(a)redhat.com>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: "Francesco Romani" <fromani(a)redhat.com>, "devel"
<devel(a)ovirt.org>, "Adam Litke" <alitke(a)redhat.com>, "Federico
> Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
> "Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
> Sent: Monday, June 15, 2015 6:20:34 PM
> Subject: Re: Libvirt secrets management - take 2
>
>
> On 15 Jun 2015, at 17:06, Nir Soffer wrote:
>
>>
>>
>> ----- Original Message -----
>>> From: "Francesco Romani" <fromani(a)redhat.com>
>>> To: "devel" <devel(a)ovirt.org>
>>> Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli"
>>> <fsimonce(a)redhat.com>, "Dan Kenigsberg"
>>> <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel Erez"
>>> <derez(a)redhat.com>, "Michal Skrivanek"
>>> <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>, "Nir Soffer"
>>> <nsoffer(a)redhat.com>
>>> Sent: Monday, June 15, 2015 4:08:09 PM
>>> Subject: Re: Libvirt secrets management - take 2
>>>
>>> ----- Original Message -----
>>>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>>>> To: "Adam Litke" <alitke(a)redhat.com>
>>>> Cc: "devel" <devel(a)ovirt.org>, "Francesco
Romani" <fromani(a)redhat.com>,
>>>> "Federico Simoncelli" <fsimonce(a)redhat.com>,
>>>> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon
Mureinik"
>>>> <amureini(a)redhat.com>, "Daniel Erez"
<derez(a)redhat.com>,
>>>> "Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric
Blake"
>>>> <eblake(a)redhat.com>
>>>> Sent: Saturday, June 13, 2015 11:14:03 PM
>>>> Subject: Re: Libvirt secrets management - take 2
>>>
>>> [...]
>>>>>> Flows
>>>>>> =====
>>>>>>
>>>>>> Start vm
>>>>>> --------
>>>>>> - Engine add required secrets to vm description
>>>>>> - Vdsm register temporary secrets with libvirt
>>>>>> - When vm is up or if operation failed, Vdsm remove the
temporary
>>>>>> secret
>>>>>>
>>>>>> Migrate vm
>>>>>> ----------
>>>>>> - Engine add required secrets to vm description
>>>>>> - Vdsm add secrets to the vm description sent to the destination
>>>>>> - On the destination, Vdsm register temporary secrets with
libvirt
>>>>>> - On the destination, when vm is up or if operation failed, Vdsm
remove
>>>>>> the temporary secret
>>>>
>>>> On issue with migration - do we have to keep the same auth information
as
>>>> in
>>>> the original vm xml, or we can create the vm on the destination using
>>>> different
>>>> auth xml?
>>>
>>> I don't have an answer for this. Need to check and play with this a bit.
>>>
>>>>>> Hot-plug disk
>>>>>> -------------
>>>>>> - Engine add secret to disk description
>>>>>> - Vdsm register temporary secret with libvirt
>>>>>> - When disk is successfully plugged, or if operation failed,
Vdsm
>>>>>> remove
>>>>>> the temporary secret
>>>
>>> OK, what if this VM is migrate afterwards? destination VDSM needs to
>>> register
>>> this secret
>>> to libvirt.
>>>
>>> The fact here is that is the source VDSM that orchestrates the migration,
>>> and
>>> if it has
>>> forgorgotten the secret, then we need get it back again from Engine and
>>> pass
>>> it through the destination VM.
>>>
>>> On migration, it is the source VM which orchestrates the flow. Engine
>>> polls
>>> source VM for updates,
>>> and consider the VM migrated only after source exited correctly.
>>> I don't recall the fine details engine side (need to check), but on VDSM
>>> side:
>>>
>>> - source VDSM receives VM.migrate(destination_uri)
>>> - source VDSM sends migrationCreate to destination VDSM
>>> - once destination VDSM created special-purpose VM, source VDSM instructs
>>> libvirt to start the migration
>>> - source VDSM monitors the migration, report progress, updates the
>>> downtime
>>> - migration fails: source VDSM automatically send VM.destroy to
>>> destination
>>> VDSM
>>> - migration succeeds: source VDSM destroy source VM, execution now
>>> continues
>>> on destination VM
>
> the source side is "destroyed" by qemu automatically. Engine just sends an
> explicit destroy after the migration is deemed successful in the engine too
>
>>
>> Ok, when we create the special purpose vm, do we include all the devices?
>
> it's an exact copy of the source vm. yes
>
>>
>> If this special vm (waiting for migration) is connected to all the disks,
>> then
>
> it is (it goes though preparePaths as any other VM creation)
>
>> we only need to register the secrets sent from the source in
>> migrationCreate,
>> start the special vm, and one it is running, delete the secrets.
>
> probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts.
- qemu got the secret key and connected to the ceph disks.
- libvirt knew about the secret when starting qemu/ hot-plugging a disk, but
now it does not know anything about the secret, except the secret uuid
in the <disk> <auth> element.
So when the VM loses ceph connection and needs to reconnect is the uuid enough?
- vdsm knows about the auth dicts in the disk devices, but does not
know the
key (the key cannot be persisted)
Assuming it can reconnect (as above) using the auth, can't the new destination VM use
the same?
So engine will have to send the secrets again, so vdsm can send them to the destination.
Doable. But the the vdsm can cache it as well, perhaps exluded from saveState(),but
otherwise why not?
Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common have no
ppc64le builds. When is this going to be added?
Thanks,
michal
>
>>
>>>
>>> so far, Engine talks with destination VM only if migration completed
>>> succesfully (unless I'm missing something :))
>>>
>>> so we need to pass back again all the needed secrets as part of VM.migrate
>>> verb
>>>
>>> This is only the very first thought of mine, we can probably think
>>> something
>>> cleaner.
>>>
>>> Bests,
>>>
>>> --
>>> Francesco Romani
>>> RedHat Engineering Virtualization R & D
>>> Phone: 8261328
>>> IRC: fromani
>
>
7:22 p.m.
----- Original Message -----
From: "Michal Skrivanek" <mskrivan(a)redhat.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "Francesco Romani" <fromani(a)redhat.com>, "devel"
<devel(a)ovirt.org>, "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
"Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
Sent: Tuesday, June 16, 2015 6:57:35 PM
Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer(a)redhat.com> wrote:
> ----- Original Message -----
>> From: "Michal Skrivanek" <mskrivan(a)redhat.com>
>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>> Cc: "Francesco Romani" <fromani(a)redhat.com>, "devel"
<devel(a)ovirt.org>,
>> "Adam Litke" <alitke(a)redhat.com>, "Federico
>> Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>,
>> "Allon Mureinik" <amureini(a)redhat.com>,
>> "Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
>> Sent: Monday, June 15, 2015 6:20:34 PM
>> Subject: Re: Libvirt secrets management - take 2
>>
>>
>> On 15 Jun 2015, at 17:06, Nir Soffer wrote:
>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Francesco Romani" <fromani(a)redhat.com>
>>>> To: "devel" <devel(a)ovirt.org>
>>>> Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli"
>>>> <fsimonce(a)redhat.com>, "Dan Kenigsberg"
>>>> <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel
>>>> Erez"
>>>> <derez(a)redhat.com>, "Michal Skrivanek"
>>>> <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>, "Nir Soffer"
>>>> <nsoffer(a)redhat.com>
>>>> Sent: Monday, June 15, 2015 4:08:09 PM
>>>> Subject: Re: Libvirt secrets management - take 2
>>>>
>>>> ----- Original Message -----
>>>>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>>>>> To: "Adam Litke" <alitke(a)redhat.com>
>>>>> Cc: "devel" <devel(a)ovirt.org>, "Francesco
Romani" <fromani(a)redhat.com>,
>>>>> "Federico Simoncelli" <fsimonce(a)redhat.com>,
>>>>> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon
Mureinik"
>>>>> <amureini(a)redhat.com>, "Daniel Erez"
<derez(a)redhat.com>,
>>>>> "Michal Skrivanek" <mskrivan(a)redhat.com>, "Eric
Blake"
>>>>> <eblake(a)redhat.com>
>>>>> Sent: Saturday, June 13, 2015 11:14:03 PM
>>>>> Subject: Re: Libvirt secrets management - take 2
>>>>
>>>> [...]
>>>>>>> Flows
>>>>>>> =====
>>>>>>>
>>>>>>> Start vm
>>>>>>> --------
>>>>>>> - Engine add required secrets to vm description
>>>>>>> - Vdsm register temporary secrets with libvirt
>>>>>>> - When vm is up or if operation failed, Vdsm remove the
temporary
>>>>>>> secret
>>>>>>>
>>>>>>> Migrate vm
>>>>>>> ----------
>>>>>>> - Engine add required secrets to vm description
>>>>>>> - Vdsm add secrets to the vm description sent to the
destination
>>>>>>> - On the destination, Vdsm register temporary secrets with
libvirt
>>>>>>> - On the destination, when vm is up or if operation failed,
Vdsm
>>>>>>> remove
>>>>>>> the temporary secret
>>>>>
>>>>> On issue with migration - do we have to keep the same auth
information
>>>>> as
>>>>> in
>>>>> the original vm xml, or we can create the vm on the destination
using
>>>>> different
>>>>> auth xml?
>>>>
>>>> I don't have an answer for this. Need to check and play with this a
bit.
>>>>
>>>>>>> Hot-plug disk
>>>>>>> -------------
>>>>>>> - Engine add secret to disk description
>>>>>>> - Vdsm register temporary secret with libvirt
>>>>>>> - When disk is successfully plugged, or if operation failed,
Vdsm
>>>>>>> remove
>>>>>>> the temporary secret
>>>>
>>>> OK, what if this VM is migrate afterwards? destination VDSM needs to
>>>> register
>>>> this secret
>>>> to libvirt.
>>>>
>>>> The fact here is that is the source VDSM that orchestrates the
>>>> migration,
>>>> and
>>>> if it has
>>>> forgorgotten the secret, then we need get it back again from Engine and
>>>> pass
>>>> it through the destination VM.
>>>>
>>>> On migration, it is the source VM which orchestrates the flow. Engine
>>>> polls
>>>> source VM for updates,
>>>> and consider the VM migrated only after source exited correctly.
>>>> I don't recall the fine details engine side (need to check), but on
VDSM
>>>> side:
>>>>
>>>> - source VDSM receives VM.migrate(destination_uri)
>>>> - source VDSM sends migrationCreate to destination VDSM
>>>> - once destination VDSM created special-purpose VM, source VDSM
>>>> instructs
>>>> libvirt to start the migration
>>>> - source VDSM monitors the migration, report progress, updates the
>>>> downtime
>>>> - migration fails: source VDSM automatically send VM.destroy to
>>>> destination
>>>> VDSM
>>>> - migration succeeds: source VDSM destroy source VM, execution now
>>>> continues
>>>> on destination VM
>>
>> the source side is "destroyed" by qemu automatically. Engine just
sends an
>> explicit destroy after the migration is deemed successful in the engine
>> too
>>
>>>
>>> Ok, when we create the special purpose vm, do we include all the devices?
>>
>> it's an exact copy of the source vm. yes
>>
>>>
>>> If this special vm (waiting for migration) is connected to all the disks,
>>> then
>>
>> it is (it goes though preparePaths as any other VM creation)
>>
>>> we only need to register the secrets sent from the source in
>>> migrationCreate,
>>> start the special vm, and one it is running, delete the secrets.
>>
>> probably. Does the source know those secrets?
>
> At the time of migration, the source does not know much about the secrerts.
>
> - qemu got the secret key and connected to the ceph disks.
> - libvirt knew about the secret when starting qemu/ hot-plugging a disk,
> but
> now it does not know anything about the secret, except the secret uuid
> in the <disk> <auth> element.
So when the VM loses ceph connection and needs to reconnect is the uuid
enough?
No, qemu needs the actual authentication key, which it got when the vm was
started, or when hot-plugging a disk.
I did not check yet if qemu can reconnect after breaking the connection
to ceph disk, but it cannot use libvirt secrets - we use private secret
that libvirt will not expose to anyone.
> - vdsm knows about the auth dicts in the disk devices, but does not know
> the
> key (the key cannot be persisted)
Assuming it can reconnect (as above) using the auth, can't the new
destination VM use the same?
The destination vm is created by libvirt with the auth key, if libvirt
have a secret with the specified uuid.
>
> So engine will have to send the secrets again, so vdsm can send them to the
> destination.
Doable. But the the vdsm can cache it as well, perhaps exluded from
saveState(),but otherwise why not?
Cashing the secret is not very useful, since after vdsm crash/restart
we will have no secrets anyway.
Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common
have no ppc64le builds. When is this going to be added?
I'm looking at that, got no response yet in #ceph about the missing
package.
How urgent is this issue?
Nir
7:34 p.m.
On 16 Jun 2015, at 18:22, Nir Soffer wrote:
----- Original Message -----
> From: "Michal Skrivanek" <mskrivan(a)redhat.com>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: "Francesco Romani" <fromani(a)redhat.com>, "devel"
<devel(a)ovirt.org>, "Adam Litke" <alitke(a)redhat.com>, "Federico
> Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>, "Allon Mureinik" <amureini(a)redhat.com>,
> "Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
> Sent: Tuesday, June 16, 2015 6:57:35 PM
> Subject: Re: Libvirt secrets management - take 2
>
>
>
> On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> ----- Original Message -----
>>> From: "Michal Skrivanek" <mskrivan(a)redhat.com>
>>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>>> Cc: "Francesco Romani" <fromani(a)redhat.com>,
"devel" <devel(a)ovirt.org>,
>>> "Adam Litke" <alitke(a)redhat.com>, "Federico
>>> Simoncelli" <fsimonce(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>,
>>> "Allon Mureinik" <amureini(a)redhat.com>,
>>> "Daniel Erez" <derez(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>
>>> Sent: Monday, June 15, 2015 6:20:34 PM
>>> Subject: Re: Libvirt secrets management - take 2
>>>
>>>
>>> On 15 Jun 2015, at 17:06, Nir Soffer wrote:
>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Francesco Romani" <fromani(a)redhat.com>
>>>>> To: "devel" <devel(a)ovirt.org>
>>>>> Cc: "Adam Litke" <alitke(a)redhat.com>, "Federico
Simoncelli"
>>>>> <fsimonce(a)redhat.com>, "Dan Kenigsberg"
>>>>> <danken(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Daniel
>>>>> Erez"
>>>>> <derez(a)redhat.com>, "Michal Skrivanek"
>>>>> <mskrivan(a)redhat.com>, "Eric Blake"
<eblake(a)redhat.com>, "Nir Soffer"
>>>>> <nsoffer(a)redhat.com>
>>>>> Sent: Monday, June 15, 2015 4:08:09 PM
>>>>> Subject: Re: Libvirt secrets management - take 2
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Nir Soffer" <nsoffer(a)redhat.com>
>>>>>> To: "Adam Litke" <alitke(a)redhat.com>
>>>>>> Cc: "devel" <devel(a)ovirt.org>, "Francesco
Romani" <fromani(a)redhat.com>,
>>>>>> "Federico Simoncelli" <fsimonce(a)redhat.com>,
>>>>>> "Dan Kenigsberg" <danken(a)redhat.com>, "Allon
Mureinik"
>>>>>> <amureini(a)redhat.com>, "Daniel Erez"
<derez(a)redhat.com>,
>>>>>> "Michal Skrivanek" <mskrivan(a)redhat.com>,
"Eric Blake"
>>>>>> <eblake(a)redhat.com>
>>>>>> Sent: Saturday, June 13, 2015 11:14:03 PM
>>>>>> Subject: Re: Libvirt secrets management - take 2
>>>>>
>>>>> [...]
>>>>>>>> Flows
>>>>>>>> =====
>>>>>>>>
>>>>>>>> Start vm
>>>>>>>> --------
>>>>>>>> - Engine add required secrets to vm description
>>>>>>>> - Vdsm register temporary secrets with libvirt
>>>>>>>> - When vm is up or if operation failed, Vdsm remove the
temporary
>>>>>>>> secret
>>>>>>>>
>>>>>>>> Migrate vm
>>>>>>>> ----------
>>>>>>>> - Engine add required secrets to vm description
>>>>>>>> - Vdsm add secrets to the vm description sent to the
destination
>>>>>>>> - On the destination, Vdsm register temporary secrets
with libvirt
>>>>>>>> - On the destination, when vm is up or if operation
failed, Vdsm
>>>>>>>> remove
>>>>>>>> the temporary secret
>>>>>>
>>>>>> On issue with migration - do we have to keep the same auth
information
>>>>>> as
>>>>>> in
>>>>>> the original vm xml, or we can create the vm on the destination
using
>>>>>> different
>>>>>> auth xml?
>>>>>
>>>>> I don't have an answer for this. Need to check and play with this
a bit.
>>>>>
>>>>>>>> Hot-plug disk
>>>>>>>> -------------
>>>>>>>> - Engine add secret to disk description
>>>>>>>> - Vdsm register temporary secret with libvirt
>>>>>>>> - When disk is successfully plugged, or if operation
failed, Vdsm
>>>>>>>> remove
>>>>>>>> the temporary secret
>>>>>
>>>>> OK, what if this VM is migrate afterwards? destination VDSM needs to
>>>>> register
>>>>> this secret
>>>>> to libvirt.
>>>>>
>>>>> The fact here is that is the source VDSM that orchestrates the
>>>>> migration,
>>>>> and
>>>>> if it has
>>>>> forgorgotten the secret, then we need get it back again from Engine
and
>>>>> pass
>>>>> it through the destination VM.
>>>>>
>>>>> On migration, it is the source VM which orchestrates the flow.
Engine
>>>>> polls
>>>>> source VM for updates,
>>>>> and consider the VM migrated only after source exited correctly.
>>>>> I don't recall the fine details engine side (need to check), but
on VDSM
>>>>> side:
>>>>>
>>>>> - source VDSM receives VM.migrate(destination_uri)
>>>>> - source VDSM sends migrationCreate to destination VDSM
>>>>> - once destination VDSM created special-purpose VM, source VDSM
>>>>> instructs
>>>>> libvirt to start the migration
>>>>> - source VDSM monitors the migration, report progress, updates the
>>>>> downtime
>>>>> - migration fails: source VDSM automatically send VM.destroy to
>>>>> destination
>>>>> VDSM
>>>>> - migration succeeds: source VDSM destroy source VM, execution now
>>>>> continues
>>>>> on destination VM
>>>
>>> the source side is "destroyed" by qemu automatically. Engine just
sends an
>>> explicit destroy after the migration is deemed successful in the engine
>>> too
>>>
>>>>
>>>> Ok, when we create the special purpose vm, do we include all the
devices?
>>>
>>> it's an exact copy of the source vm. yes
>>>
>>>>
>>>> If this special vm (waiting for migration) is connected to all the
disks,
>>>> then
>>>
>>> it is (it goes though preparePaths as any other VM creation)
>>>
>>>> we only need to register the secrets sent from the source in
>>>> migrationCreate,
>>>> start the special vm, and one it is running, delete the secrets.
>>>
>>> probably. Does the source know those secrets?
>>
>> At the time of migration, the source does not know much about the secrerts.
>>
>> - qemu got the secret key and connected to the ceph disks.
>> - libvirt knew about the secret when starting qemu/ hot-plugging a disk,
>> but
>> now it does not know anything about the secret, except the secret uuid
>> in the <disk> <auth> element.
>
> So when the VM loses ceph connection and needs to reconnect is the uuid
> enough?
No, qemu needs the actual authentication key, which it got when the vm was
started, or when hot-plugging a disk.
I did not check yet if qemu can reconnect after breaking the connection
to ceph disk, but it cannot use libvirt secrets - we use private secret
that libvirt will not expose to anyone.
>
>
>> - vdsm knows about the auth dicts in the disk devices, but does not know
>> the
>> key (the key cannot be persisted)
>
> Assuming it can reconnect (as above) using the auth, can't the new
> destination VM use the same?
The destination vm is created by libvirt with the auth key, if libvirt
have a secret with the specified uuid.
>
>>
>> So engine will have to send the secrets again, so vdsm can send them to the
>> destination.
>
> Doable. But the the vdsm can cache it as well, perhaps exluded from
> saveState(),but otherwise why not?
Cashing the secret is not very useful, since after vdsm crash/restart
we will have no secrets anyway.
if we want VMs to be able to reconnect after network outage we would require engine to be
reachable too
don't know if it's easier to sync/send the secret on vdsm restart, rather than
reconnects
>
> Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common
> have no ppc64le builds. When is this going to be added?
I'm looking at that, got no response yet in #ceph about the missing
package.
How urgent is this issue?
ppc64le is a supported platform in 3.6, any gap from x86 needs to be communicated and
acked by IBM
Thanks,
michal
Nir
3447
days inactive
3451
days old
14 comments
4 participants
participants (4)
-
Adam Litke -
Francesco Romani -
Michal Skrivanek -
Nir Soffer