[ovirt-devel] Libvirt secrets management - take 2

Michal Skrivanek mskrivan at redhat.com
Tue Jun 16 16:34:31 UTC 2015


On 16 Jun 2015, at 18:22, Nir Soffer wrote:

> 
> 
> ----- Original Message -----
>> From: "Michal Skrivanek" <mskrivan at redhat.com>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: "Francesco Romani" <fromani at redhat.com>, "devel" <devel at ovirt.org>, "Adam Litke" <alitke at redhat.com>, "Federico
>> Simoncelli" <fsimonce at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
>> "Daniel Erez" <derez at redhat.com>, "Eric Blake" <eblake at redhat.com>
>> Sent: Tuesday, June 16, 2015 6:57:35 PM
>> Subject: Re: Libvirt secrets management - take 2
>> 
>> 
>> 
>> On 15 Jun 2015, at 17:57, Nir Soffer <nsoffer at redhat.com> wrote:
>> 
>>> ----- Original Message -----
>>>> From: "Michal Skrivanek" <mskrivan at redhat.com>
>>>> To: "Nir Soffer" <nsoffer at redhat.com>
>>>> Cc: "Francesco Romani" <fromani at redhat.com>, "devel" <devel at ovirt.org>,
>>>> "Adam Litke" <alitke at redhat.com>, "Federico
>>>> Simoncelli" <fsimonce at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>,
>>>> "Allon Mureinik" <amureini at redhat.com>,
>>>> "Daniel Erez" <derez at redhat.com>, "Eric Blake" <eblake at redhat.com>
>>>> Sent: Monday, June 15, 2015 6:20:34 PM
>>>> Subject: Re: Libvirt secrets management - take 2
>>>> 
>>>> 
>>>> On 15 Jun 2015, at 17:06, Nir Soffer wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>>> From: "Francesco Romani" <fromani at redhat.com>
>>>>>> To: "devel" <devel at ovirt.org>
>>>>>> Cc: "Adam Litke" <alitke at redhat.com>, "Federico Simoncelli"
>>>>>> <fsimonce at redhat.com>, "Dan Kenigsberg"
>>>>>> <danken at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Daniel
>>>>>> Erez"
>>>>>> <derez at redhat.com>, "Michal Skrivanek"
>>>>>> <mskrivan at redhat.com>, "Eric Blake" <eblake at redhat.com>, "Nir Soffer"
>>>>>> <nsoffer at redhat.com>
>>>>>> Sent: Monday, June 15, 2015 4:08:09 PM
>>>>>> Subject: Re: Libvirt secrets management - take 2
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>>> From: "Nir Soffer" <nsoffer at redhat.com>
>>>>>>> To: "Adam Litke" <alitke at redhat.com>
>>>>>>> Cc: "devel" <devel at ovirt.org>, "Francesco Romani" <fromani at redhat.com>,
>>>>>>> "Federico Simoncelli" <fsimonce at redhat.com>,
>>>>>>> "Dan Kenigsberg" <danken at redhat.com>, "Allon Mureinik"
>>>>>>> <amureini at redhat.com>, "Daniel Erez" <derez at redhat.com>,
>>>>>>> "Michal Skrivanek" <mskrivan at redhat.com>, "Eric Blake"
>>>>>>> <eblake at redhat.com>
>>>>>>> Sent: Saturday, June 13, 2015 11:14:03 PM
>>>>>>> Subject: Re: Libvirt secrets management - take 2
>>>>>> 
>>>>>> [...]
>>>>>>>>> Flows
>>>>>>>>> =====
>>>>>>>>> 
>>>>>>>>> Start vm
>>>>>>>>> --------
>>>>>>>>> - Engine add required secrets to vm description
>>>>>>>>> - Vdsm register temporary secrets with libvirt
>>>>>>>>> - When vm is up or if operation failed, Vdsm remove the temporary
>>>>>>>>> secret
>>>>>>>>> 
>>>>>>>>> Migrate vm
>>>>>>>>> ----------
>>>>>>>>> - Engine add required secrets to vm description
>>>>>>>>> - Vdsm add secrets to the vm description sent to the destination
>>>>>>>>> - On the destination, Vdsm register temporary secrets with libvirt
>>>>>>>>> - On the destination, when vm is up or if operation failed, Vdsm
>>>>>>>>> remove
>>>>>>>>> the temporary secret
>>>>>>> 
>>>>>>> On issue with migration - do we have to keep the same auth information
>>>>>>> as
>>>>>>> in
>>>>>>> the original vm xml, or we can create the vm on the destination using
>>>>>>> different
>>>>>>> auth xml?
>>>>>> 
>>>>>> I don't have an answer for this. Need to check and play with this a bit.
>>>>>> 
>>>>>>>>> Hot-plug disk
>>>>>>>>> -------------
>>>>>>>>> - Engine add secret to disk description
>>>>>>>>> - Vdsm register temporary secret with libvirt
>>>>>>>>> - When disk is successfully plugged, or if operation failed, Vdsm
>>>>>>>>> remove
>>>>>>>>> the temporary secret
>>>>>> 
>>>>>> OK, what if this VM is migrate afterwards? destination VDSM needs to
>>>>>> register
>>>>>> this secret
>>>>>> to libvirt.
>>>>>> 
>>>>>> The fact here is that is the source VDSM that orchestrates the
>>>>>> migration,
>>>>>> and
>>>>>> if it has
>>>>>> forgorgotten the secret, then we need get it back again from Engine and
>>>>>> pass
>>>>>> it through the destination VM.
>>>>>> 
>>>>>> On migration, it is the source VM which orchestrates the flow. Engine
>>>>>> polls
>>>>>> source VM for updates,
>>>>>> and consider the VM migrated only after source exited correctly.
>>>>>> I don't recall the fine details engine side (need to check), but on VDSM
>>>>>> side:
>>>>>> 
>>>>>> - source VDSM receives VM.migrate(destination_uri)
>>>>>> - source VDSM sends migrationCreate to destination VDSM
>>>>>> - once destination VDSM created special-purpose VM, source VDSM
>>>>>> instructs
>>>>>> libvirt to start the migration
>>>>>> - source VDSM monitors the migration, report progress, updates the
>>>>>> downtime
>>>>>> - migration fails: source VDSM automatically send VM.destroy to
>>>>>> destination
>>>>>> VDSM
>>>>>> - migration succeeds: source VDSM destroy source VM, execution now
>>>>>> continues
>>>>>> on destination VM
>>>> 
>>>> the source side is "destroyed" by qemu automatically. Engine just sends an
>>>> explicit destroy after the migration is deemed successful in the engine
>>>> too
>>>> 
>>>>> 
>>>>> Ok, when we create the special purpose vm, do we include all the devices?
>>>> 
>>>> it's an exact copy of the source vm. yes
>>>> 
>>>>> 
>>>>> If this special vm (waiting for migration) is connected to all the disks,
>>>>> then
>>>> 
>>>> it is (it goes though preparePaths as any other VM creation)
>>>> 
>>>>> we only need to register the secrets sent from the source in
>>>>> migrationCreate,
>>>>> start the special vm, and one it is running, delete the secrets.
>>>> 
>>>> probably. Does the source know those secrets?
>>> 
>>> At the time of migration, the source does not know much about the secrerts.
>>> 
>>> - qemu got the secret key and connected to the ceph disks.
>>> - libvirt knew about the secret when starting qemu/ hot-plugging a disk,
>>> but
>>> now it does not know anything about the secret, except the secret uuid
>>> in the <disk> <auth> element.
>> 
>> So when the VM loses ceph connection and needs to reconnect is the uuid
>> enough?
> 
> No, qemu needs the actual authentication key, which it got when the vm was
> started, or when hot-plugging a disk.
> 
> I did not check yet if qemu can reconnect after breaking the connection
> to ceph disk, but it cannot use libvirt secrets - we use private secret
> that libvirt will not expose to anyone.
> 
>> 
>> 
>>> - vdsm knows about the auth dicts in the disk devices, but does not know
>>> the
>>> key (the key cannot be persisted)
>> 
>> Assuming it can reconnect (as above) using the auth, can't the new
>> destination VM use the same?
> 
> The destination vm is created by libvirt with the auth key, if libvirt
> have a secret with the specified uuid.
> 
>> 
>>> 
>>> So engine will have to send the secrets again, so vdsm can send them to the
>>> destination.
>> 
>> Doable. But the the vdsm can cache it as well, perhaps exluded from
>> saveState(),but otherwise why not?
> 
> Cashing the secret is not very useful, since after vdsm crash/restart
> we will have no secrets anyway.

if we want VMs to be able to reconnect after network outage we would require engine to be reachable too
don't know if it's easier to sync/send the secret on vdsm restart, rather than reconnects

> 
>> 
>> Unrelated Q - AFIK it's supposed to work on ppc as well. Current ceph-common
>> have no ppc64le builds. When is this going to be added?
> 
> I'm looking at that, got no response yet in #ceph about the missing
> package.
> 
> How urgent is this issue?

ppc64le is a supported platform in 3.6, any gap from x86 needs to be communicated and acked by IBM

Thanks,
michal

> 
> Nir




More information about the Devel mailing list