Hi all, Recently support for Ceph network disk landed in master. It its possible now to start a vm using Ceph network disk or hot-plug/unplug such disk using Cephx authentication. However, to make it work, you must add the relevant Ceph secret to libvirt manually, in the same way it is done in OpenStack deployment. Our goal is to manage secrets automatically and use ephemeral (safer) secrets. The next patches in the Ceph topic [1], implement secret management in the same way we manage storage domains or server connections: The concept is - all hosts can use all secrets, so you can migrate a vm using Ceph disk to any host in the cluster. 1. When host becomes up, we register the secrets associated with all the current active domains with libvirt 2. When activating a domain, we register the secrets associated with the new domain with libvirt 3. When deactivating a domain, we unregister the secrets associated with the domain from libvirt 4. When moving host to maintenance, we clear all secrets 5. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host This system seems to work, but Federico pointed few issues and suggested a new (simpler?) approach. In future libvirt version, libvirt will support the concept of transient secrets so you can start a transient vm using secret without registering the secret with libvirt before starting the vm. The secret will be specified in the vm XML (for starting a vm) or disk XML (for hot-plug). This will make our secret management system and APIs useless. Managing state on multiple hosts is hard; we will probably have to deal with nasty edge cases (e.g. lost messages, network errors), which may lead to host with missing secret, which cannot run some vms. We probably do this right for storage domains (after 8 years?), and we should not assume that we are smarter and secret management will work in the first try. The new approach is to *not* manage state or multiple hosts. Instead, send the required secrets only to the host that starting a vm or hot-plugging a disk that need a libvirt secret: 1. When starting a vm, add the required secrets to the vm description. On the host, vdsm will register these secrets with libvirt before starting the vm. 2. When migrating a vm, add the required secrets to the vm description. On the host, vdsm will send these secrets to the destination host, and on the destination host, vdsm will register the secrets with libvirt before starting the vm. 3. When hot-plugging a disk, send the secret if needed in the disk description. On the host, vdsm will register the secrets with libvirt. 4. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host 5. We never unregister secrets, since they are ephemeral anyway. 6. Alternatively, we can implement secrets reference counting so when a vm stops or disk is hot-unplugged we decrease the reference count on the secrets associated with this vm/disk, and if no other vms need the secret, we can unregister the secret from libvirt. The new approach is simpler, if we avoid the fancy secret reference counting. I believe we can get it merged in couple of weeks with help from the virt team. Please share your thoughts on these alternative solutions. Thanks, Nir [1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:ce...
Here are more details on the new approach. A Ceph key is required only when starting a vm or hot-plugging a disk. Once the operation is done, libvirt does not need the Ceph key any more. A vm operation requiring a secret, will register a Ceph key using new random UUID, and remove the libvirt secret as soon as the operation was finished or failed. This scheme does not require secret reference counting. If multiple vms need the same Ceph key, we register it multiple times with libvirt, using unique UUIDs. This also avoid possible races when removing a libvirt secret in the same time another vm is trying to add it, or updating secret usage id, which is currently racy (you must remove the existing secret and register a new one). Positive flow: - Engine adds required Ceph keys to vm description - Vdsm register keys with libvirt, using new random UUID - When the vm operation is done (vm started, disk hot-plugged), remove the temporary secret (e.g. using try-finally) Negative flows: - Vm operation fails - temporary secret is unregistered in the finally block - Vdsm crash during the operation - temporary secret is removed when vdsm starts again. - Libvirt crash during the operation - secret removed since we use ephemeral secrets - Host crash during operation - same - Libvirt fail to remove secret - we cannot handle this :-) Flows ===== Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret I think this is the correct direction, assuming that we can get it works for migration - I have no idea on that part. Nir ----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Adam Litke" <alitke@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Friday, June 12, 2015 2:21:46 PM Subject: Libvirt secrets management - take 2
Hi all,
Recently support for Ceph network disk landed in master. It its possible now to start a vm using Ceph network disk or hot-plug/unplug such disk using Cephx authentication.
However, to make it work, you must add the relevant Ceph secret to libvirt manually, in the same way it is done in OpenStack deployment. Our goal is to manage secrets automatically and use ephemeral (safer) secrets.
The next patches in the Ceph topic [1], implement secret management in the same way we manage storage domains or server connections:
The concept is - all hosts can use all secrets, so you can migrate a vm using Ceph disk to any host in the cluster.
1. When host becomes up, we register the secrets associated with all the current active domains with libvirt
2. When activating a domain, we register the secrets associated with the new domain with libvirt
3. When deactivating a domain, we unregister the secrets associated with the domain from libvirt
4. When moving host to maintenance, we clear all secrets
5. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
This system seems to work, but Federico pointed few issues and suggested a new (simpler?) approach.
In future libvirt version, libvirt will support the concept of transient secrets so you can start a transient vm using secret without registering the secret with libvirt before starting the vm. The secret will be specified in the vm XML (for starting a vm) or disk XML (for hot-plug). This will make our secret management system and APIs useless.
Managing state on multiple hosts is hard; we will probably have to deal with nasty edge cases (e.g. lost messages, network errors), which may lead to host with missing secret, which cannot run some vms. We probably do this right for storage domains (after 8 years?), and we should not assume that we are smarter and secret management will work in the first try.
The new approach is to *not* manage state or multiple hosts. Instead, send the required secrets only to the host that starting a vm or hot-plugging a disk that need a libvirt secret:
1. When starting a vm, add the required secrets to the vm description. On the host, vdsm will register these secrets with libvirt before starting the vm.
2. When migrating a vm, add the required secrets to the vm description. On the host, vdsm will send these secrets to the destination host, and on the destination host, vdsm will register the secrets with libvirt before starting the vm.
3. When hot-plugging a disk, send the secret if needed in the disk description. On the host, vdsm will register the secrets with libvirt.
4. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
5. We never unregister secrets, since they are ephemeral anyway.
6. Alternatively, we can implement secrets reference counting so when a vm stops or disk is hot-unplugged we decrease the reference count on the secrets associated with this vm/disk, and if no other vms need the secret, we can unregister the secret from libvirt.
The new approach is simpler, if we avoid the fancy secret reference counting. I believe we can get it merged in couple of weeks with help from the virt team.
Please share your thoughts on these alternative solutions.
Thanks, Nir
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:ce...
On 12/06/15 08:10 -0400, Nir Soffer wrote:
Here are more details on the new approach.
A Ceph key is required only when starting a vm or hot-plugging a disk. Once the operation is done, libvirt does not need the Ceph key any more.
A vm operation requiring a secret, will register a Ceph key using new random UUID, and remove the libvirt secret as soon as the operation was finished or failed.
This scheme does not require secret reference counting. If multiple vms need the same Ceph key, we register it multiple times with libvirt, using unique UUIDs.
This also avoid possible races when removing a libvirt secret in the same time another vm is trying to add it, or updating secret usage id, which is currently racy (you must remove the existing secret and register a new one).
I really like this new design. As long as you remove the secrets when you're done with them then I am not concerned with having an accumulation of untracked secrets. I am happy to get rid of the complexity of secret reference counting.
Positive flow:
- Engine adds required Ceph keys to vm description - Vdsm register keys with libvirt, using new random UUID - When the vm operation is done (vm started, disk hot-plugged), remove the temporary secret (e.g. using try-finally)
Negative flows:
- Vm operation fails - temporary secret is unregistered in the finally block - Vdsm crash during the operation - temporary secret is removed when vdsm starts again. - Libvirt crash during the operation - secret removed since we use ephemeral secrets - Host crash during operation - same - Libvirt fail to remove secret - we cannot handle this :-)
What about the case where a storage domain becomes unavailable (causing the VM to pause with -EIO). When the domainMonitor reestablishes a connection to the domain would the secrets need to be renewed?
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
I think this is the correct direction, assuming that we can get it works for migration - I have no idea on that part.
+1 - This is a much improved design in my opinion.
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Adam Litke" <alitke@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Friday, June 12, 2015 2:21:46 PM Subject: Libvirt secrets management - take 2
Hi all,
Recently support for Ceph network disk landed in master. It its possible now to start a vm using Ceph network disk or hot-plug/unplug such disk using Cephx authentication.
However, to make it work, you must add the relevant Ceph secret to libvirt manually, in the same way it is done in OpenStack deployment. Our goal is to manage secrets automatically and use ephemeral (safer) secrets.
The next patches in the Ceph topic [1], implement secret management in the same way we manage storage domains or server connections:
The concept is - all hosts can use all secrets, so you can migrate a vm using Ceph disk to any host in the cluster.
1. When host becomes up, we register the secrets associated with all the current active domains with libvirt
2. When activating a domain, we register the secrets associated with the new domain with libvirt
3. When deactivating a domain, we unregister the secrets associated with the domain from libvirt
4. When moving host to maintenance, we clear all secrets
5. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
This system seems to work, but Federico pointed few issues and suggested a new (simpler?) approach.
In future libvirt version, libvirt will support the concept of transient secrets so you can start a transient vm using secret without registering the secret with libvirt before starting the vm. The secret will be specified in the vm XML (for starting a vm) or disk XML (for hot-plug). This will make our secret management system and APIs useless.
Managing state on multiple hosts is hard; we will probably have to deal with nasty edge cases (e.g. lost messages, network errors), which may lead to host with missing secret, which cannot run some vms. We probably do this right for storage domains (after 8 years?), and we should not assume that we are smarter and secret management will work in the first try.
The new approach is to *not* manage state or multiple hosts. Instead, send the required secrets only to the host that starting a vm or hot-plugging a disk that need a libvirt secret:
1. When starting a vm, add the required secrets to the vm description. On the host, vdsm will register these secrets with libvirt before starting the vm.
2. When migrating a vm, add the required secrets to the vm description. On the host, vdsm will send these secrets to the destination host, and on the destination host, vdsm will register the secrets with libvirt before starting the vm.
3. When hot-plugging a disk, send the secret if needed in the disk description. On the host, vdsm will register the secrets with libvirt.
4. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
5. We never unregister secrets, since they are ephemeral anyway.
6. Alternatively, we can implement secrets reference counting so when a vm stops or disk is hot-unplugged we decrease the reference count on the secrets associated with this vm/disk, and if no other vms need the secret, we can unregister the secret from libvirt.
The new approach is simpler, if we avoid the fancy secret reference counting. I believe we can get it merged in couple of weeks with help from the virt team.
Please share your thoughts on these alternative solutions.
Thanks, Nir
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:ce...
-- Adam Litke
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 4:52:19 PM Subject: Re: Libvirt secrets management - take 2
On 12/06/15 08:10 -0400, Nir Soffer wrote:
Here are more details on the new approach.
A Ceph key is required only when starting a vm or hot-plugging a disk. Once the operation is done, libvirt does not need the Ceph key any more.
A vm operation requiring a secret, will register a Ceph key using new random UUID, and remove the libvirt secret as soon as the operation was finished or failed.
This scheme does not require secret reference counting. If multiple vms need the same Ceph key, we register it multiple times with libvirt, using unique UUIDs.
This also avoid possible races when removing a libvirt secret in the same time another vm is trying to add it, or updating secret usage id, which is currently racy (you must remove the existing secret and register a new one).
I really like this new design. As long as you remove the secrets when you're done with them then I am not concerned with having an accumulation of untracked secrets. I am happy to get rid of the complexity of secret reference counting.
Positive flow:
- Engine adds required Ceph keys to vm description - Vdsm register keys with libvirt, using new random UUID - When the vm operation is done (vm started, disk hot-plugged), remove the temporary secret (e.g. using try-finally)
Negative flows:
- Vm operation fails - temporary secret is unregistered in the finally block - Vdsm crash during the operation - temporary secret is removed when vdsm starts again. - Libvirt crash during the operation - secret removed since we use ephemeral secrets - Host crash during operation - same - Libvirt fail to remove secret - we cannot handle this :-)
What about the case where a storage domain becomes unavailable (causing the VM to pause with -EIO). When the domainMonitor reestablishes a connection to the domain would the secrets need to be renewed?
We don't have domain monitoring for ceph disks. I don't think we tried yet what happen when you block access to all ceph monitors, but I guess that qemu can renew the connection if needed since it is holding the authentication key.
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml? In the original, xml, each disk may have auth element with secret uuid. This secret must be defined in the destination libvirt so qemu can connect to the disk on the destination machine. Francesco? Eric?
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
I think this is the correct direction, assuming that we can get it works for migration - I have no idea on that part.
+1 - This is a much improved design in my opinion.
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Adam Litke" <alitke@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Friday, June 12, 2015 2:21:46 PM Subject: Libvirt secrets management - take 2
Hi all,
Recently support for Ceph network disk landed in master. It its possible now to start a vm using Ceph network disk or hot-plug/unplug such disk using Cephx authentication.
However, to make it work, you must add the relevant Ceph secret to libvirt manually, in the same way it is done in OpenStack deployment. Our goal is to manage secrets automatically and use ephemeral (safer) secrets.
The next patches in the Ceph topic [1], implement secret management in the same way we manage storage domains or server connections:
The concept is - all hosts can use all secrets, so you can migrate a vm using Ceph disk to any host in the cluster.
1. When host becomes up, we register the secrets associated with all the current active domains with libvirt
2. When activating a domain, we register the secrets associated with the new domain with libvirt
3. When deactivating a domain, we unregister the secrets associated with the domain from libvirt
4. When moving host to maintenance, we clear all secrets
5. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
This system seems to work, but Federico pointed few issues and suggested a new (simpler?) approach.
In future libvirt version, libvirt will support the concept of transient secrets so you can start a transient vm using secret without registering the secret with libvirt before starting the vm. The secret will be specified in the vm XML (for starting a vm) or disk XML (for hot-plug). This will make our secret management system and APIs useless.
Managing state on multiple hosts is hard; we will probably have to deal with nasty edge cases (e.g. lost messages, network errors), which may lead to host with missing secret, which cannot run some vms. We probably do this right for storage domains (after 8 years?), and we should not assume that we are smarter and secret management will work in the first try.
The new approach is to *not* manage state or multiple hosts. Instead, send the required secrets only to the host that starting a vm or hot-plugging a disk that need a libvirt secret:
1. When starting a vm, add the required secrets to the vm description. On the host, vdsm will register these secrets with libvirt before starting the vm.
2. When migrating a vm, add the required secrets to the vm description. On the host, vdsm will send these secrets to the destination host, and on the destination host, vdsm will register the secrets with libvirt before starting the vm.
3. When hot-plugging a disk, send the secret if needed in the disk description. On the host, vdsm will register the secrets with libvirt.
4. When vdsm shutdown or starts, clear all secrets to ensure that we don't keep stale or unneeded secrets on a host
5. We never unregister secrets, since they are ephemeral anyway.
6. Alternatively, we can implement secrets reference counting so when a vm stops or disk is hot-unplugged we decrease the reference count on the secrets associated with this vm/disk, and if no other vms need the secret, we can unregister the secret from libvirt.
The new approach is simpler, if we avoid the fancy secret reference counting. I believe we can get it merged in couple of weeks with help from the virt team.
Please share your thoughts on these alternative solutions.
Thanks, Nir
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:ce...
-- Adam Litke
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt. The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM. On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side: - source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM so far, Engine talks with destination VM only if migration completed succesfully (unless I'm missing something :)) so we need to pass back again all the needed secrets as part of VM.migrate verb This is only the very first thought of mine, we can probably think something cleaner. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
Ok, when we create the special purpose vm, do we include all the devices? If this special vm (waiting for migration) is connected to all the disks, then we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
so far, Engine talks with destination VM only if migration completed succesfully (unless I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate verb
This is only the very first thought of mine, we can probably think something cleaner.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 5:06:09 PM Subject: Re: Libvirt secrets management - take 2
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only "special" thing here (to partially amend my own wording) is the reported status of the VM itself.
If this special vm (waiting for migration) is connected to all the disks, then we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner approach. To have source VM act like a passthrough feels a little funny, but probably this is the only thing we can do give the architecture. So, changes needed on virt flows: - in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most of the infrastructure we have to create VM. It is little different from just create a VM from Engine. - in 'migration destination' flow, it doesn't look different from simple VM creation. we should just drop all the secrets once the disks are attached (or failed to attach). At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :) -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
On 15 Jun 2015, at 17:16, Francesco Romani wrote:
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 5:06:09 PM Subject: Re: Libvirt secrets management - take 2
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only "special" thing here (to partially amend my own wording) is the reported status of the VM itself.
If this special vm (waiting for migration) is connected to all the disks, then we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner approach.
To have source VM act like a passthrough feels a little funny, but probably this is the only thing we can do give the architecture.
So, changes needed on virt flows:
- in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most of the infrastructure we have to create VM. It is little different from just create a VM from Engine. - in 'migration destination' flow, it doesn't look different from simple VM creation. we should just drop all the secrets once the disks are attached (or failed to attach).
At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :)
Could it not be handled by lower layer? I suppose it has some kind of token or something?
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Francesco Romani" <fromani@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:30:00 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:16, Francesco Romani wrote:
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 5:06:09 PM Subject: Re: Libvirt secrets management - take 2
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only "special" thing here (to partially amend my own wording) is the reported status of the VM itself.
If this special vm (waiting for migration) is connected to all the disks, then we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner approach.
To have source VM act like a passthrough feels a little funny, but probably this is the only thing we can do give the architecture.
So, changes needed on virt flows:
- in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most of the infrastructure we have to create VM. It is little different from just create a VM from Engine. - in 'migration destination' flow, it doesn't look different from simple VM creation. we should just drop all the secrets once the disks are attached (or failed to attach).
At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :)
Could it not be handled by lower layer? I suppose it has some kind of token or something?
What do you mean?
On 15 Jun 2015, at 17:33, Nir Soffer wrote:
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Francesco Romani" <fromani@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:30:00 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:16, Francesco Romani wrote:
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 5:06:09 PM Subject: Re: Libvirt secrets management - take 2
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
Ok, when we create the special purpose vm, do we include all the devices?
Sure. It has to be (actually: is) a clone of he source VM. Actually the only "special" thing here (to partially amend my own wording) is the reported status of the VM itself.
If this special vm (waiting for migration) is connected to all the disks, then we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
Indeed. I just wonder, since we're still talking about design, if there is a cleaner approach.
To have source VM act like a passthrough feels a little funny, but probably this is the only thing we can do give the architecture.
So, changes needed on virt flows:
- in VM.migrate (public API, Engine->VDSM) support secrets passing. These secrets must not be stored, but passed verbatim to VM.migrationCreate and forgotten afterwards - in VM.migrationCreate (private API, VDSM->VDSM): as above. here we reuse most of the infrastructure we have to create VM. It is little different from just create a VM from Engine. - in 'migration destination' flow, it doesn't look different from simple VM creation. we should just drop all the secrets once the disks are attached (or failed to attach).
At glance looks simple, should be quick and easy, unless I'm missing some pitfalls :)
Could it not be handled by lower layer? I suppose it has some kind of token or something?
What do you mean?
do we have to manage the connection? If qemu lose the connection we can't re-establish? I understood that libvirt keeps the secret in memory - is that correct? If so then it can send it to the destination, right? And what do we do about libvirt restarts? I missed the initial discussions so please ignore me if i'm completely off....
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
so far, Engine talks with destination VM only if migration completed succesfully (unless I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate verb
This is only the very first thought of mine, we can probably think something cleaner.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:20:34 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
Flows =====
Start vm -------- - Engine add required secrets to vm description - Vdsm register temporary secrets with libvirt - When vm is up or if operation failed, Vdsm remove the temporary secret
Migrate vm ---------- - Engine add required secrets to vm description - Vdsm add secrets to the vm description sent to the destination - On the destination, Vdsm register temporary secrets with libvirt - On the destination, when vm is up or if operation failed, Vdsm remove the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
Hot-plug disk ------------- - Engine add secret to disk description - Vdsm register temporary secret with libvirt - When disk is successfully plugged, or if operation failed, Vdsm remove the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts. - qemu got the secret key and connected to the ceph disks. - libvirt knew about the secret when starting qemu/ hot-plugging a disk, but now it does not know anything about the secret, except the secret uuid in the <disk> <auth> element. - vdsm knows about the auth dicts in the disk devices, but does not know the key (the key cannot be persisted) So engine will have to send the secrets again, so vdsm can send them to the destination.
so far, Engine talks with destination VM only if migration completed succesfully (unless I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate verb
This is only the very first thought of mine, we can probably think something cleaner.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer@redhat.com> wrote:
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:20:34 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
> Flows > ===== > > Start vm > -------- > - Engine add required secrets to vm description > - Vdsm register temporary secrets with libvirt > - When vm is up or if operation failed, Vdsm remove the temporary > secret > > Migrate vm > ---------- > - Engine add required secrets to vm description > - Vdsm add secrets to the vm description sent to the destination > - On the destination, Vdsm register temporary secrets with libvirt > - On the destination, when vm is up or if operation failed, Vdsm remove > the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
> Hot-plug disk > ------------- > - Engine add secret to disk description > - Vdsm register temporary secret with libvirt > - When disk is successfully plugged, or if operation failed, Vdsm > remove > the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts.
- qemu got the secret key and connected to the ceph disks. - libvirt knew about the secret when starting qemu/ hot-plugging a disk, but now it does not know anything about the secret, except the secret uuid in the <disk> <auth> element.
So when the VM loses ceph connection and needs to reconnect is the uuid enough?
- vdsm knows about the auth dicts in the disk devices, but does not know the key (the key cannot be persisted)
Assuming it can reconnect (as above) using the auth, can't the new destination VM use the same?
So engine will have to send the secrets again, so vdsm can send them to the destination.
Doable. But the the vdsm can cache it as well, perhaps exluded from saveState(),but otherwise why not? Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common have no ppc64le builds. When is this going to be added? Thanks, michal
so far, Engine talks with destination VM only if migration completed succesfully (unless I'm missing something :))
so we need to pass back again all the needed secrets as part of VM.migrate verb
This is only the very first thought of mine, we can probably think something cleaner.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Tuesday, June 16, 2015 6:57:35 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer@redhat.com> wrote:
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:20:34 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message -----
From: "Nir Soffer" <nsoffer@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Saturday, June 13, 2015 11:14:03 PM Subject: Re: Libvirt secrets management - take 2
[...]
>> Flows >> ===== >> >> Start vm >> -------- >> - Engine add required secrets to vm description >> - Vdsm register temporary secrets with libvirt >> - When vm is up or if operation failed, Vdsm remove the temporary >> secret >> >> Migrate vm >> ---------- >> - Engine add required secrets to vm description >> - Vdsm add secrets to the vm description sent to the destination >> - On the destination, Vdsm register temporary secrets with libvirt >> - On the destination, when vm is up or if operation failed, Vdsm >> remove >> the temporary secret
On issue with migration - do we have to keep the same auth information as in the original vm xml, or we can create the vm on the destination using different auth xml?
I don't have an answer for this. Need to check and play with this a bit.
>> Hot-plug disk >> ------------- >> - Engine add secret to disk description >> - Vdsm register temporary secret with libvirt >> - When disk is successfully plugged, or if operation failed, Vdsm >> remove >> the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts.
- qemu got the secret key and connected to the ceph disks. - libvirt knew about the secret when starting qemu/ hot-plugging a disk, but now it does not know anything about the secret, except the secret uuid in the <disk> <auth> element.
So when the VM loses ceph connection and needs to reconnect is the uuid enough?
No, qemu needs the actual authentication key, which it got when the vm was started, or when hot-plugging a disk. I did not check yet if qemu can reconnect after breaking the connection to ceph disk, but it cannot use libvirt secrets - we use private secret that libvirt will not expose to anyone.
- vdsm knows about the auth dicts in the disk devices, but does not know the key (the key cannot be persisted)
Assuming it can reconnect (as above) using the auth, can't the new destination VM use the same?
The destination vm is created by libvirt with the auth key, if libvirt have a secret with the specified uuid.
So engine will have to send the secrets again, so vdsm can send them to the destination.
Doable. But the the vdsm can cache it as well, perhaps exluded from saveState(),but otherwise why not?
Cashing the secret is not very useful, since after vdsm crash/restart we will have no secrets anyway.
Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common have no ppc64le builds. When is this going to be added?
I'm looking at that, got no response yet in #ceph about the missing package. How urgent is this issue? Nir
On 16 Jun 2015, at 18:22, Nir Soffer wrote:
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Tuesday, June 16, 2015 6:57:35 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer@redhat.com> wrote:
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, "devel" <devel@ovirt.org>, "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Eric Blake" <eblake@redhat.com> Sent: Monday, June 15, 2015 6:20:34 PM Subject: Re: Libvirt secrets management - take 2
On 15 Jun 2015, at 17:06, Nir Soffer wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "devel" <devel@ovirt.org> Cc: "Adam Litke" <alitke@redhat.com>, "Federico Simoncelli" <fsimonce@redhat.com>, "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" <eblake@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Monday, June 15, 2015 4:08:09 PM Subject: Re: Libvirt secrets management - take 2
----- Original Message ----- > From: "Nir Soffer" <nsoffer@redhat.com> > To: "Adam Litke" <alitke@redhat.com> > Cc: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com>, > "Federico Simoncelli" <fsimonce@redhat.com>, > "Dan Kenigsberg" <danken@redhat.com>, "Allon Mureinik" > <amureini@redhat.com>, "Daniel Erez" <derez@redhat.com>, > "Michal Skrivanek" <mskrivan@redhat.com>, "Eric Blake" > <eblake@redhat.com> > Sent: Saturday, June 13, 2015 11:14:03 PM > Subject: Re: Libvirt secrets management - take 2
[...] >>> Flows >>> ===== >>> >>> Start vm >>> -------- >>> - Engine add required secrets to vm description >>> - Vdsm register temporary secrets with libvirt >>> - When vm is up or if operation failed, Vdsm remove the temporary >>> secret >>> >>> Migrate vm >>> ---------- >>> - Engine add required secrets to vm description >>> - Vdsm add secrets to the vm description sent to the destination >>> - On the destination, Vdsm register temporary secrets with libvirt >>> - On the destination, when vm is up or if operation failed, Vdsm >>> remove >>> the temporary secret > > On issue with migration - do we have to keep the same auth information > as > in > the original vm xml, or we can create the vm on the destination using > different > auth xml?
I don't have an answer for this. Need to check and play with this a bit.
>>> Hot-plug disk >>> ------------- >>> - Engine add secret to disk description >>> - Vdsm register temporary secret with libvirt >>> - When disk is successfully plugged, or if operation failed, Vdsm >>> remove >>> the temporary secret
OK, what if this VM is migrate afterwards? destination VDSM needs to register this secret to libvirt.
The fact here is that is the source VDSM that orchestrates the migration, and if it has forgorgotten the secret, then we need get it back again from Engine and pass it through the destination VM.
On migration, it is the source VM which orchestrates the flow. Engine polls source VM for updates, and consider the VM migrated only after source exited correctly. I don't recall the fine details engine side (need to check), but on VDSM side:
- source VDSM receives VM.migrate(destination_uri) - source VDSM sends migrationCreate to destination VDSM - once destination VDSM created special-purpose VM, source VDSM instructs libvirt to start the migration - source VDSM monitors the migration, report progress, updates the downtime - migration fails: source VDSM automatically send VM.destroy to destination VDSM - migration succeeds: source VDSM destroy source VM, execution now continues on destination VM
the source side is "destroyed" by qemu automatically. Engine just sends an explicit destroy after the migration is deemed successful in the engine too
Ok, when we create the special purpose vm, do we include all the devices?
it's an exact copy of the source vm. yes
If this special vm (waiting for migration) is connected to all the disks, then
it is (it goes though preparePaths as any other VM creation)
we only need to register the secrets sent from the source in migrationCreate, start the special vm, and one it is running, delete the secrets.
probably. Does the source know those secrets?
At the time of migration, the source does not know much about the secrerts.
- qemu got the secret key and connected to the ceph disks. - libvirt knew about the secret when starting qemu/ hot-plugging a disk, but now it does not know anything about the secret, except the secret uuid in the <disk> <auth> element.
So when the VM loses ceph connection and needs to reconnect is the uuid enough?
No, qemu needs the actual authentication key, which it got when the vm was started, or when hot-plugging a disk.
I did not check yet if qemu can reconnect after breaking the connection to ceph disk, but it cannot use libvirt secrets - we use private secret that libvirt will not expose to anyone.
- vdsm knows about the auth dicts in the disk devices, but does not know the key (the key cannot be persisted)
Assuming it can reconnect (as above) using the auth, can't the new destination VM use the same?
The destination vm is created by libvirt with the auth key, if libvirt have a secret with the specified uuid.
So engine will have to send the secrets again, so vdsm can send them to the destination.
Doable. But the the vdsm can cache it as well, perhaps exluded from saveState(),but otherwise why not?
Cashing the secret is not very useful, since after vdsm crash/restart we will have no secrets anyway.
if we want VMs to be able to reconnect after network outage we would require engine to be reachable too don't know if it's easier to sync/send the secret on vdsm restart, rather than reconnects
Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common have no ppc64le builds. When is this going to be added?
I'm looking at that, got no response yet in #ceph about the missing package.
How urgent is this issue?
ppc64le is a supported platform in 3.6, any gap from x86 needs to be communicated and acked by IBM Thanks, michal
Nir
participants (4)
-
Adam Litke -
Francesco Romani -
Michal Skrivanek -
Nir Soffer