[ovirt-devel] Libvirt secrets management - take 2

[Nir Soffer]

----- Original Message -----
> From: "Michal Skrivanek" <mskrivan at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: "Francesco Romani" <fromani at redhat.com>, "devel" <devel at ovirt.org>, "Adam Litke" <alitke at redhat.com>, "Federico
> Simoncelli" <fsimonce at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
> "Daniel Erez" <derez at redhat.com>, "Eric Blake" <eblake at redhat.com>
> Sent: Tuesday, June 16, 2015 6:57:35 PM
> Subject: Re: Libvirt secrets management - take 2
> 
> 
> 
> On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer at redhat.com> wrote:
> 
> > ----- Original Message -----
> >> From: "Michal Skrivanek" <mskrivan at redhat.com>
> >> To: "Nir Soffer" <nsoffer at redhat.com>
> >> Cc: "Francesco Romani" <fromani at redhat.com>, "devel" <devel at ovirt.org>,
> >> "Adam Litke" <alitke at redhat.com>, "Federico
> >> Simoncelli" <fsimonce at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>,
> >> "Allon Mureinik" <amureini at redhat.com>,
> >> "Daniel Erez" <derez at redhat.com>, "Eric Blake" <eblake at redhat.com>
> >> Sent: Monday, June 15, 2015 6:20:34 PM
> >> Subject: Re: Libvirt secrets management - take 2
> >> 
> >> 
> >> On 15 Jun 2015, at 17:06, Nir Soffer wrote:
> >> 
> >>> 
> >>> 
> >>> ----- Original Message -----
> >>>> From: "Francesco Romani" <fromani at redhat.com>
> >>>> To: "devel" <devel at ovirt.org>
> >>>> Cc: "Adam Litke" <alitke at redhat.com>, "Federico Simoncelli"
> >>>> <fsimonce at redhat.com>, "Dan Kenigsberg"
> >>>> <danken at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Daniel
> >>>> Erez"
> >>>> <derez at redhat.com>, "Michal Skrivanek"
> >>>> <mskrivan at redhat.com>, "Eric Blake" <eblake at redhat.com>, "Nir Soffer"
> >>>> <nsoffer at redhat.com>
> >>>> Sent: Monday, June 15, 2015 4:08:09 PM
> >>>> Subject: Re: Libvirt secrets management - take 2
> >>>> 
> >>>> ----- Original Message -----
> >>>>> From: "Nir Soffer" <nsoffer at redhat.com>
> >>>>> To: "Adam Litke" <alitke at redhat.com>
> >>>>> Cc: "devel" <devel at ovirt.org>, "Francesco Romani" <fromani at redhat.com>,
> >>>>> "Federico Simoncelli" <fsimonce at redhat.com>,
> >>>>> "Dan Kenigsberg" <danken at redhat.com>, "Allon Mureinik"
> >>>>> <amureini at redhat.com>, "Daniel Erez" <derez at redhat.com>,
> >>>>> "Michal Skrivanek" <mskrivan at redhat.com>, "Eric Blake"
> >>>>> <eblake at redhat.com>
> >>>>> Sent: Saturday, June 13, 2015 11:14:03 PM
> >>>>> Subject: Re: Libvirt secrets management - take 2
> >>>> 
> >>>> [...]
> >>>>>>> Flows
> >>>>>>> =====
> >>>>>>> 
> >>>>>>> Start vm
> >>>>>>> --------
> >>>>>>> - Engine add required secrets to vm description
> >>>>>>> - Vdsm register temporary secrets with libvirt
> >>>>>>> - When vm is up or if operation failed, Vdsm remove the temporary
> >>>>>>> secret
> >>>>>>> 
> >>>>>>> Migrate vm
> >>>>>>> ----------
> >>>>>>> - Engine add required secrets to vm description
> >>>>>>> - Vdsm add secrets to the vm description sent to the destination
> >>>>>>> - On the destination, Vdsm register temporary secrets with libvirt
> >>>>>>> - On the destination, when vm is up or if operation failed, Vdsm
> >>>>>>> remove
> >>>>>>> the temporary secret
> >>>>> 
> >>>>> On issue with migration - do we have to keep the same auth information
> >>>>> as
> >>>>> in
> >>>>> the original vm xml, or we can create the vm on the destination using
> >>>>> different
> >>>>> auth xml?
> >>>> 
> >>>> I don't have an answer for this. Need to check and play with this a bit.
> >>>> 
> >>>>>>> Hot-plug disk
> >>>>>>> -------------
> >>>>>>> - Engine add secret to disk description
> >>>>>>> - Vdsm register temporary secret with libvirt
> >>>>>>> - When disk is successfully plugged, or if operation failed, Vdsm
> >>>>>>> remove
> >>>>>>> the temporary secret
> >>>> 
> >>>> OK, what if this VM is migrate afterwards? destination VDSM needs to
> >>>> register
> >>>> this secret
> >>>> to libvirt.
> >>>> 
> >>>> The fact here is that is the source VDSM that orchestrates the
> >>>> migration,
> >>>> and
> >>>> if it has
> >>>> forgorgotten the secret, then we need get it back again from Engine and
> >>>> pass
> >>>> it through the destination VM.
> >>>> 
> >>>> On migration, it is the source VM which orchestrates the flow. Engine
> >>>> polls
> >>>> source VM for updates,
> >>>> and consider the VM migrated only after source exited correctly.
> >>>> I don't recall the fine details engine side (need to check), but on VDSM
> >>>> side:
> >>>> 
> >>>> - source VDSM receives VM.migrate(destination_uri)
> >>>> - source VDSM sends migrationCreate to destination VDSM
> >>>> - once destination VDSM created special-purpose VM, source VDSM
> >>>> instructs
> >>>> libvirt to start the migration
> >>>> - source VDSM monitors the migration, report progress, updates the
> >>>> downtime
> >>>> - migration fails: source VDSM automatically send VM.destroy to
> >>>> destination
> >>>> VDSM
> >>>> - migration succeeds: source VDSM destroy source VM, execution now
> >>>> continues
> >>>> on destination VM
> >> 
> >> the source side is "destroyed" by qemu automatically. Engine just sends an
> >> explicit destroy after the migration is deemed successful in the engine
> >> too
> >> 
> >>> 
> >>> Ok, when we create the special purpose vm, do we include all the devices?
> >> 
> >> it's an exact copy of the source vm. yes
> >> 
> >>> 
> >>> If this special vm (waiting for migration) is connected to all the disks,
> >>> then
> >> 
> >> it is (it goes though preparePaths as any other VM creation)
> >> 
> >>> we only need to register the secrets sent from the source in
> >>> migrationCreate,
> >>> start the special vm, and one it is running, delete the secrets.
> >> 
> >> probably. Does the source know those secrets?
> > 
> > At the time of migration, the source does not know much about the secrerts.
> > 
> > - qemu got the secret key and connected to the ceph disks.
> > - libvirt knew about the secret when starting qemu/ hot-plugging a disk,
> > but
> >  now it does not know anything about the secret, except the secret uuid
> >  in the <disk> <auth> element.
> 
> So when the VM loses ceph connection and needs to reconnect is the uuid
> enough?

No, qemu needs the actual authentication key, which it got when the vm was
started, or when hot-plugging a disk.

I did not check yet if qemu can reconnect after breaking the connection
to ceph disk, but it cannot use libvirt secrets - we use private secret
that libvirt will not expose to anyone.

> 
> 
> > - vdsm knows about the auth dicts in the disk devices, but does not know
> > the
> >  key (the key cannot be persisted)
> 
> Assuming it can reconnect (as above) using the auth, can't the new
> destination VM use the same?

The destination vm is created by libvirt with the auth key, if libvirt
have a secret with the specified uuid.

> 
> > 
> > So engine will have to send the secrets again, so vdsm can send them to the
> > destination.
> 
> Doable. But the the vdsm can cache it as well, perhaps exluded from
> saveState(),but otherwise why not?

Cashing the secret is not very useful, since after vdsm crash/restart
we will have no secrets anyway.

> 
> Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common
> have no ppc64le builds. When is this going to be added?

I'm looking at that, got no response yet in #ceph about the missing
package.

How urgent is this issue?

Nir