[ovirt-users] Hosted engine on Ceph RBD

[Nir Soffer]

On Mon, Jun 20, 2016 at 2:23 AM, Bond, Darryl <dbond at nrggos.com.au> wrote:
> Nir,
> I absolutely understand why you would want to use Cinder for oVirt disk management.
> My question as only about the hosted engine storage which is a 'special case'. The engine-setup process could have the path to the RBD and secrets file.

We are not keeping ceph key in a file. We use libvirt ephemeral private
secrets:
https://libvirt.org/formatsecret.html

This means that only libvirt and the vm it starts can access the ceph key,
and once libvirt is killed or the host reboot, the key is not accessible on
the host.

Once you add a cinder/ceph storage domain and add a key to
ovirt-engine, we register the secrets on the hosts that need access
to the key, and unregister when a host does not need access.

Hosted engine setup can bootstrap the system by adding ceph key
to the first host, but from this point, ovirt-engine must know about
the cinder/ceph storage domain and the key, so it can register the
key on other hosted engine hosts.

But even if we solve this issue, we cannot run yet hosted engine
on ceph, since hosted engine depend on sanlock, and we don't
support sanlock on ceph storage. hosted engine agent also
communicate via shared storage, but it does not support ceph
storage.

We use ceph volumes as network disks - they do not appear on the
host as local devices. qemu is accessing these volumes directly
using librbd and the ceph key provided by libvirt. This is the most
secure way and it give the best performance.

The issue with sanlock and hosted engine agents may be solved by
exposing certain ceph volumes as local devices, we are considering this
for host storage monitoring, which is not implemented yet for ceph.

Nir