Re: [ovirt-devel] Libvirt secrets management - take 2

----- Original Message ----- > From: "Michal Skrivanek" <mskrivan(a)redhat.com&gt; > To: "Nir Soffer" <nsoffer(a)redhat.com&gt; > Cc: "Francesco Romani" <fromani(a)redhat.com&gt;, "devel" <devel(a)ovirt.org&gt;, "Adam Litke" <alitke(a)redhat.com&gt;, "Federico > Simoncelli" <fsimonce(a)redhat.com&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Allon Mureinik" <amureini(a)redhat.com&gt;, > "Daniel Erez" <derez(a)redhat.com&gt;, "Eric Blake" <eblake(a)redhat.com&gt; > Sent: Tuesday, June 16, 2015 6:57:35 PM > Subject: Re: Libvirt secrets management - take 2 > > > > On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > >> ----- Original Message ----- >>> From: "Michal Skrivanek" <mskrivan(a)redhat.com&gt; >>> To: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>> Cc: "Francesco Romani" <fromani(a)redhat.com&gt;, "devel" <devel(a)ovirt.org&gt;, >>> "Adam Litke" <alitke(a)redhat.com&gt;, "Federico >>> Simoncelli" <fsimonce(a)redhat.com&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt;, >>> "Allon Mureinik" <amureini(a)redhat.com&gt;, >>> "Daniel Erez" <derez(a)redhat.com&gt;, "Eric Blake" <eblake(a)redhat.com&gt; >>> Sent: Monday, June 15, 2015 6:20:34 PM >>> Subject: Re: Libvirt secrets management - take 2 >>> >>> >>> On 15 Jun 2015, at 17:06, Nir Soffer wrote: >>> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Francesco Romani" <fromani(a)redhat.com&gt; >>>>> To: "devel" <devel(a)ovirt.org&gt; >>>>> Cc: "Adam Litke" <alitke(a)redhat.com&gt;, "Federico Simoncelli" >>>>> <fsimonce(a)redhat.com&gt;, "Dan Kenigsberg" >>>>> <danken(a)redhat.com&gt;, "Allon Mureinik" <amureini(a)redhat.com&gt;, "Daniel >>>>> Erez" >>>>> <derez(a)redhat.com&gt;, "Michal Skrivanek" >>>>> <mskrivan(a)redhat.com&gt;, "Eric Blake" <eblake(a)redhat.com&gt;, "Nir Soffer" >>>>> <nsoffer(a)redhat.com&gt; >>>>> Sent: Monday, June 15, 2015 4:08:09 PM >>>>> Subject: Re: Libvirt secrets management - take 2 >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Nir Soffer" <nsoffer(a)redhat.com&gt; >>>>>> To: "Adam Litke" <alitke(a)redhat.com&gt; >>>>>> Cc: "devel" <devel(a)ovirt.org&gt;, "Francesco Romani" <fromani(a)redhat.com&gt;, >>>>>> "Federico Simoncelli" <fsimonce(a)redhat.com&gt;, >>>>>> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Allon Mureinik" >>>>>> <amureini(a)redhat.com&gt;, "Daniel Erez" <derez(a)redhat.com&gt;, >>>>>> "Michal Skrivanek" <mskrivan(a)redhat.com&gt;, "Eric Blake" >>>>>> <eblake(a)redhat.com&gt; >>>>>> Sent: Saturday, June 13, 2015 11:14:03 PM >>>>>> Subject: Re: Libvirt secrets management - take 2 >>>>> >>>>> [...] >>>>>>>> Flows >>>>>>>> ===== >>>>>>>> >>>>>>>> Start vm >>>>>>>> -------- >>>>>>>> - Engine add required secrets to vm description >>>>>>>> - Vdsm register temporary secrets with libvirt >>>>>>>> - When vm is up or if operation failed, Vdsm remove the temporary >>>>>>>> secret >>>>>>>> >>>>>>>> Migrate vm >>>>>>>> ---------- >>>>>>>> - Engine add required secrets to vm description >>>>>>>> - Vdsm add secrets to the vm description sent to the destination >>>>>>>> - On the destination, Vdsm register temporary secrets with libvirt >>>>>>>> - On the destination, when vm is up or if operation failed, Vdsm >>>>>>>> remove >>>>>>>> the temporary secret >>>>>> >>>>>> On issue with migration - do we have to keep the same auth information >>>>>> as >>>>>> in >>>>>> the original vm xml, or we can create the vm on the destination using >>>>>> different >>>>>> auth xml? >>>>> >>>>> I don't have an answer for this. Need to check and play with this a bit. >>>>> >>>>>>>> Hot-plug disk >>>>>>>> ------------- >>>>>>>> - Engine add secret to disk description >>>>>>>> - Vdsm register temporary secret with libvirt >>>>>>>> - When disk is successfully plugged, or if operation failed, Vdsm >>>>>>>> remove >>>>>>>> the temporary secret >>>>> >>>>> OK, what if this VM is migrate afterwards? destination VDSM needs to >>>>> register >>>>> this secret >>>>> to libvirt. >>>>> >>>>> The fact here is that is the source VDSM that orchestrates the >>>>> migration, >>>>> and >>>>> if it has >>>>> forgorgotten the secret, then we need get it back again from Engine and >>>>> pass >>>>> it through the destination VM. >>>>> >>>>> On migration, it is the source VM which orchestrates the flow. Engine >>>>> polls >>>>> source VM for updates, >>>>> and consider the VM migrated only after source exited correctly. >>>>> I don't recall the fine details engine side (need to check), but on VDSM >>>>> side: >>>>> >>>>> - source VDSM receives VM.migrate(destination_uri) >>>>> - source VDSM sends migrationCreate to destination VDSM >>>>> - once destination VDSM created special-purpose VM, source VDSM >>>>> instructs >>>>> libvirt to start the migration >>>>> - source VDSM monitors the migration, report progress, updates the >>>>> downtime >>>>> - migration fails: source VDSM automatically send VM.destroy to >>>>> destination >>>>> VDSM >>>>> - migration succeeds: source VDSM destroy source VM, execution now >>>>> continues >>>>> on destination VM >>> >>> the source side is "destroyed" by qemu automatically. Engine just sends an >>> explicit destroy after the migration is deemed successful in the engine >>> too >>> >>>> >>>> Ok, when we create the special purpose vm, do we include all the devices? >>> >>> it's an exact copy of the source vm. yes >>> >>>> >>>> If this special vm (waiting for migration) is connected to all the disks, >>>> then >>> >>> it is (it goes though preparePaths as any other VM creation) >>> >>>> we only need to register the secrets sent from the source in >>>> migrationCreate, >>>> start the special vm, and one it is running, delete the secrets. >>> >>> probably. Does the source know those secrets? >> >> At the time of migration, the source does not know much about the secrerts. >> >> - qemu got the secret key and connected to the ceph disks. >> - libvirt knew about the secret when starting qemu/ hot-plugging a disk, >> but >> now it does not know anything about the secret, except the secret uuid >> in the <disk> <auth> element. > > So when the VM loses ceph connection and needs to reconnect is the uuid > enough? No, qemu needs the actual authentication key, which it got when the vm was started, or when hot-plugging a disk. I did not check yet if qemu can reconnect after breaking the connection to ceph disk, but it cannot use libvirt secrets - we use private secret that libvirt will not expose to anyone. > > >> - vdsm knows about the auth dicts in the disk devices, but does not know >> the >> key (the key cannot be persisted) > > Assuming it can reconnect (as above) using the auth, can't the new > destination VM use the same? The destination vm is created by libvirt with the auth key, if libvirt have a secret with the specified uuid. > >> >> So engine will have to send the secrets again, so vdsm can send them to the >> destination. > > Doable. But the the vdsm can cache it as well, perhaps exluded from > saveState(),but otherwise why not? Cashing the secret is not very useful, since after vdsm crash/restart we will have no secrets anyway.

> > Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common > have no ppc64le builds. When is this going to be added? I'm looking at that, got no response yet in #ceph about the missing package. How urgent is this issue?

Nir