4:48 a.m.
Hi,
I upgraded my test environment to 4.3.2 and now I'm trying to set up a
"Managed Block Storage" domain with our Ceph 12.2 cluster. I think I got
all prerequisites, but when saving the configuration for the domain with
volume_driver "cinder.volume.drivers.rbd.RBDDriver" (and a couple of
other options) I get "VolumeBackendAPIException: Bad or unexpected
response from the storage volume backend API: Error connecting to ceph
cluster" in engine log (full error below). Unfortunately this is a
rather generic error message and I don't really know where to look next.
Accessing the rbd pool from the engine host with rbd CLI and the
configured "rbd_user" works flawlessly...
Although I don't think this is directly connected there is one other
question that comes up for me: how are libvirt "Authentication Keys"
handled with Ceph "Managed Block Storage" domains? With "standalone
Cinder" setups like we are using now you have to configure a "provider"
of type "OpenStack Block Storage" where you can configure these keys
that are referenced in cinder.conf as "rbd_secret_uuid". How is this
supposed to work now?
Thanks for any advice, we are using oVirt with Ceph heavily and are very
interested in a tight integration of oVirt and Ceph.
Matthias
2019-04-01 11:14:55,128+02 ERROR
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor]
(default task-22) [b6665621-6b85-438e-8c68-266f33e55d79] cinderlib
execution failed: Traceback (most recent call last):
File "./cinderlib-client.py", line 187, in main
args.command(args)
File "./cinderlib-client.py", line 275, in storage_stats
backend = load_backend(args)
File "./cinderlib-client.py", line 217, in load_backend
return cl.Backend(**json.loads(args.driver))
File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line
87, in __init__
self.driver.check_for_setup_error()
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
line 288, in check_for_setup_error
with RADOSClient(self):
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
line 170, in __init__
self.cluster, self.ioctx = driver._connect_to_rados(pool)
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
line 346, in _connect_to_rados
return _do_conn(pool, remote, timeout)
File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in
_wrapper
return r.call(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
raise attempt.get()
File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
line 344, in _do_conn
raise exception.VolumeBackendAPIException(data=msg)
VolumeBackendAPIException: Bad or unexpected response from the storage
volume backend API: Error connecting to ceph cluster.
5:07 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Hi,
Thanks for trying this out!
We added a separate log file for cinderlib in 4.3.2, it should be available
under /var/log/ovirt-engine/cinderlib/cinderlib.log
They are not perfect yet, and more improvements are coming, but it might
provide some insight about the issue
Now you are supposed to pass the secret in the driver options, something
like this (using REST):
<property>
<name>rbd_ceph_conf</name>
<value>/etc/ceph/ceph.conf</value>
</property>
<property>
<name>rbd_keyring_conf</name>
<value>/etc/ceph/ceph.client.admin.keyring</value>
</property>
On Mon, Apr 1, 2019 at 12:51 PM Matthias Leopold <
matthias.leopold(a)meduniwien.ac.at> wrote:
5:32 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Am 01.04.19 um 12:07 schrieb Benny Zlotnik:
OK, /var/log/ovirt-engine/cinderlib/cinderlib.log says:
2019-04-01 11:14:54,925 - cinder.volume.drivers.rbd - ERROR - Error
connecting to ceph cluster.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
line 337, in _do_conn
client.connect()
File "rados.pyx", line 885, in rados.Rados.connect
(/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/build/src/pybind/rados/pyrex/rados.c:9785)
OSError: [errno 95] error connecting to the cluster
2019-04-01 11:14:54,930 - root - ERROR - Failure occurred when trying to
run command 'storage_stats': Bad or unexpected response from the storage
volume backend API: Error connecting to ceph cluster.
I don't really know what to do with that either.
BTW, the cinder version on engine host is "pike"
(openstack-cinder-11.2.0-1.el7.noarch)
Shall I pass "rbd_secret_uuid" in the driver options? But where is this
UUID created? Where is the ceph secret key stored in oVirt?
thanks
Matthias
6:17 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Not sure if the version is related (I know
it's been tested with
pike), but you can try and install the latest rocky (that's what I use
for development)
Shall I pass "rbd_secret_uuid" in the driver options? But
where is this
UUID created? Where is the ceph secret key stored in oVirt?
I don't think
it's needed as ceph based volumes are no longer a
network disk like in the cinder integration, but it is attached like a
regular block device
The only things that are a must now are "rbd_keyring_conf" and
"rbd_ceph_conf" (you don't need the first if the path to the keyring
is configured in the latter)
And I think you get the error because it's missing or incorrect, since
I manually removed the keyring path from the configuration and got the
same error as you
9:07 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Am 01.04.19 um 13:17 schrieb Benny Zlotnik:
I upgraded cinder on engine and hypervisors to rocky and installed
missing "ceph-common" packages on hypervisors. I set
"rbd_keyring_conf"
and "rbd_ceph_conf" as indicated and got as far as adding a "Managed
Block Storage" domain and creating a disk (which is also visible through
"rbd ls"). I used a keyring that is only authorized for the pool I
specified with "rbd_pool". When I try to start the VM it fails and I see
the following in supervdsm.log on hypervisor:
ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error
executing helper: Command [\'/usr/libexec/vdsm/managedvolume-helper\',
\'attach\'] failed with rc=1 out=\'\' err=\'oslo.privsep.daemon:
Running
privsep helper: [\\\'sudo\\\', \\\'privsep-helper\\\',
\\\'--privsep_context\\\', \\\'os_brick.privileged.default\\\',
\\\'--privsep_sock_path\\\',
\\\'/tmp/tmp5S8zZV/privsep.sock\\\']\\noslo.privsep.daemon: Spawned new
privsep daemon via rootwrap\\noslo.privsep.daemon: privsep daemon
starting\\noslo.privsep.daemon: privsep process running with uid/gid:
0/0\\noslo.privsep.daemon: privsep process running with capabilities
(eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none\\noslo.privsep.daemon:
privsep daemon running as pid 15944\\nTraceback (most recent call
last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 154, in
<module>\\n sys.exit(main(sys.argv[1:]))\\n File
"/usr/libexec/vdsm/managedvolume-helper", line 77, in main\\n
args.command(args)\\n File "/usr/libexec/vdsm/managedvolume-helper",
line 137, in attach\\n attachment =
conn.connect_volume(conn_info[\\\'data\\\'])\\n File
"/usr/lib/python2.7/site-packages/vdsm/storage/nos_brick.py", line 96,
in connect_volume\\n run_as_root=True)\\n File
"/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in
_execute\\n result = self.__execute(*args, **kwargs)\\n File
"/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line
169, in execute\\n return execute_root(*cmd, **kwargs)\\n File
"/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line
207, in _wrap\\n return self.channel.remote_call(name, args,
kwargs)\\n File
"/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in
remote_call\\n raise
exc_type(*result[2])\\noslo_concurrency.processutils.ProcessExecutionError:
Unexpected error while running command.\\nCommand: rbd map
volume-36f5eb75-329e-4bd2-88d0-6f0bfe5d1040 --pool ovirt-test --conf
/tmp/brickrbd_RmBvxA --id None --mon_host xxx.xxx.216.45:6789 --mon_host
xxx.xxx.216.54:6789 --mon_host xxx.xxx.216.55:6789\\nExit code:
22\\nStdout: u\\\'In some cases useful info is found in syslog - try
"dmesg | tail".\\\\n\\\'\\nStderr: u"2019-04-01 15:27:30.743196
7fe0b4632d40 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\\\\nrbd: sysfs write failed\\\\n2019-04-01
15:27:30.746987 7fe0b4632d40 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\\\\n2019-04-01 15:27:30.747896
7fe0b4632d40 -1 monclient: authenticate NOTE: no keyring found; disabled
cephx authentication\\\\n2019-04-01 15:27:30.747903 7fe0b4632d40 0
librados: client.None authentication error (95) Operation not
supported\\\\nrbd: couldn\\\'t connect to the cluster!\\\\nrbd: map
failed: (22) Invalid argument\\\\n"\\n\'',)
I tried to provide a /etc/ceph directory with ceph.conf and client
keyring on hypervisors (as configured in driver options). This didn't
solve it and doesn't seem to be the right way as the mentioned
/tmp/brickrbd_RmBvxA contains the needed keyring data. Please give me
some advice what's wrong.
thx
Matthias
9:24 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Did you pass the rbd_user when creating the storage domain?
On Mon, Apr 1, 2019 at 5:08 PM Matthias Leopold
<matthias.leopold(a)meduniwien.ac.at> wrote:
>
>
> Am 01.04.19 um 13:17 schrieb Benny Zlotnik:
> >> OK, /var/log/ovirt-engine/cinderlib/cinderlib.log says:
> >>
> >> 2019-04-01 11:14:54,925 - cinder.volume.drivers.rbd - ERROR - Error
> >> connecting to ceph cluster.
> >> Traceback (most recent call last):
> >> File
"/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
> >> line 337, in _do_conn
> >> client.connect()
> >> File "rados.pyx", line 885, in rados.Rados.connect
> >>
(/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/build/src/pybind/rados/pyrex/rados.c:9785)
> >> OSError: [errno 95] error connecting to the cluster
> >> 2019-04-01 11:14:54,930 - root - ERROR - Failure occurred when trying to
> >> run command 'storage_stats': Bad or unexpected response from the
storage
> >> volume backend API: Error connecting to ceph cluster.
> >>
> >> I don't really know what to do with that either.
> >> BTW, the cinder version on engine host is "pike"
> >> (openstack-cinder-11.2.0-1.el7.noarch)
> > Not sure if the version is related (I know it's been tested with
> > pike), but you can try and install the latest rocky (that's what I use
> > for development)
>
> I upgraded cinder on engine and hypervisors to rocky and installed
> missing "ceph-common" packages on hypervisors. I set
"rbd_keyring_conf"
> and "rbd_ceph_conf" as indicated and got as far as adding a "Managed
> Block Storage" domain and creating a disk (which is also visible through
> "rbd ls"). I used a keyring that is only authorized for the pool I
> specified with "rbd_pool". When I try to start the VM it fails and I see
> the following in supervdsm.log on hypervisor:
>
> ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error
> executing helper: Command [\'/usr/libexec/vdsm/managedvolume-helper\',
> \'attach\'] failed with rc=1 out=\'\' err=\'oslo.privsep.daemon:
Running
> privsep helper: [\\\'sudo\\\', \\\'privsep-helper\\\',
> \\\'--privsep_context\\\', \\\'os_brick.privileged.default\\\',
> \\\'--privsep_sock_path\\\',
> \\\'/tmp/tmp5S8zZV/privsep.sock\\\']\\noslo.privsep.daemon: Spawned new
> privsep daemon via rootwrap\\noslo.privsep.daemon: privsep daemon
> starting\\noslo.privsep.daemon: privsep process running with uid/gid:
> 0/0\\noslo.privsep.daemon: privsep process running with capabilities
> (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none\\noslo.privsep.daemon:
> privsep daemon running as pid 15944\\nTraceback (most recent call
> last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 154, in
> <module>\\n sys.exit(main(sys.argv[1:]))\\n File
> "/usr/libexec/vdsm/managedvolume-helper", line 77, in main\\n
> args.command(args)\\n File "/usr/libexec/vdsm/managedvolume-helper",
> line 137, in attach\\n attachment =
> conn.connect_volume(conn_info[\\\'data\\\'])\\n File
> "/usr/lib/python2.7/site-packages/vdsm/storage/nos_brick.py", line 96,
> in connect_volume\\n run_as_root=True)\\n File
> "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in
> _execute\\n result = self.__execute(*args, **kwargs)\\n File
> "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line
> 169, in execute\\n return execute_root(*cmd, **kwargs)\\n File
> "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line
> 207, in _wrap\\n return self.channel.remote_call(name, args,
> kwargs)\\n File
> "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in
> remote_call\\n raise
> exc_type(*result[2])\\noslo_concurrency.processutils.ProcessExecutionError:
> Unexpected error while running command.\\nCommand: rbd map
> volume-36f5eb75-329e-4bd2-88d0-6f0bfe5d1040 --pool ovirt-test --conf
> /tmp/brickrbd_RmBvxA --id None --mon_host xxx.xxx.216.45:6789 --mon_host
> xxx.xxx.216.54:6789 --mon_host xxx.xxx.216.55:6789\\nExit code:
> 22\\nStdout: u\\\'In some cases useful info is found in syslog - try
> "dmesg | tail".\\\\n\\\'\\nStderr: u"2019-04-01 15:27:30.743196
> 7fe0b4632d40 -1 auth: unable to find a keyring on
>
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> (2) No such file or directory\\\\nrbd: sysfs write failed\\\\n2019-04-01
> 15:27:30.746987 7fe0b4632d40 -1 auth: unable to find a keyring on
>
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> (2) No such file or directory\\\\n2019-04-01 15:27:30.747896
> 7fe0b4632d40 -1 monclient: authenticate NOTE: no keyring found; disabled
> cephx authentication\\\\n2019-04-01 15:27:30.747903 7fe0b4632d40 0
> librados: client.None authentication error (95) Operation not
> supported\\\\nrbd: couldn\\\'t connect to the cluster!\\\\nrbd: map
> failed: (22) Invalid argument\\\\n"\\n\'',)
>
> I tried to provide a /etc/ceph directory with ceph.conf and client
> keyring on hypervisors (as configured in driver options). This didn't
> solve it and doesn't seem to be the right way as the mentioned
> /tmp/brickrbd_RmBvxA contains the needed keyring data. Please give me
> some advice what's wrong.
>
> thx
> Matthias
>
>
9:59 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
I added an example for ceph[1]
[1] -
https://github.com/oVirt/ovirt-site/blob/468c79a05358e20289e7403d9dd24732...
On Mon, Apr 1, 2019 at 5:24 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
>
> Did you pass the rbd_user when creating the storage domain?
>
> On Mon, Apr 1, 2019 at 5:08 PM Matthias Leopold
> <matthias.leopold(a)meduniwien.ac.at> wrote:
> >
> >
> > Am 01.04.19 um 13:17 schrieb Benny Zlotnik:
> > >> OK, /var/log/ovirt-engine/cinderlib/cinderlib.log says:
> > >>
> > >> 2019-04-01 11:14:54,925 - cinder.volume.drivers.rbd - ERROR - Error
> > >> connecting to ceph cluster.
> > >> Traceback (most recent call last):
> > >> File
"/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
> > >> line 337, in _do_conn
> > >> client.connect()
> > >> File "rados.pyx", line 885, in rados.Rados.connect
> > >>
(/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/build/src/pybind/rados/pyrex/rados.c:9785)
> > >> OSError: [errno 95] error connecting to the cluster
> > >> 2019-04-01 11:14:54,930 - root - ERROR - Failure occurred when trying
to
> > >> run command 'storage_stats': Bad or unexpected response from
the storage
> > >> volume backend API: Error connecting to ceph cluster.
> > >>
> > >> I don't really know what to do with that either.
> > >> BTW, the cinder version on engine host is "pike"
> > >> (openstack-cinder-11.2.0-1.el7.noarch)
> > > Not sure if the version is related (I know it's been tested with
> > > pike), but you can try and install the latest rocky (that's what I use
> > > for development)
> >
> > I upgraded cinder on engine and hypervisors to rocky and installed
> > missing "ceph-common" packages on hypervisors. I set
"rbd_keyring_conf"
> > and "rbd_ceph_conf" as indicated and got as far as adding a
"Managed
> > Block Storage" domain and creating a disk (which is also visible through
> > "rbd ls"). I used a keyring that is only authorized for the pool I
> > specified with "rbd_pool". When I try to start the VM it fails and I
see
> > the following in supervdsm.log on hypervisor:
> >
> > ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error
> > executing helper: Command [\'/usr/libexec/vdsm/managedvolume-helper\',
> > \'attach\'] failed with rc=1 out=\'\'
err=\'oslo.privsep.daemon: Running
> > privsep helper: [\\\'sudo\\\', \\\'privsep-helper\\\',
> > \\\'--privsep_context\\\', \\\'os_brick.privileged.default\\\',
> > \\\'--privsep_sock_path\\\',
> > \\\'/tmp/tmp5S8zZV/privsep.sock\\\']\\noslo.privsep.daemon: Spawned new
> > privsep daemon via rootwrap\\noslo.privsep.daemon: privsep daemon
> > starting\\noslo.privsep.daemon: privsep process running with uid/gid:
> > 0/0\\noslo.privsep.daemon: privsep process running with capabilities
> > (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none\\noslo.privsep.daemon:
> > privsep daemon running as pid 15944\\nTraceback (most recent call
> > last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 154,
in
> > <module>\\n sys.exit(main(sys.argv[1:]))\\n File
> > "/usr/libexec/vdsm/managedvolume-helper", line 77, in main\\n
> > args.command(args)\\n File "/usr/libexec/vdsm/managedvolume-helper",
> > line 137, in attach\\n attachment =
> > conn.connect_volume(conn_info[\\\'data\\\'])\\n File
> > "/usr/lib/python2.7/site-packages/vdsm/storage/nos_brick.py", line
96,
> > in connect_volume\\n run_as_root=True)\\n File
> > "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in
> > _execute\\n result = self.__execute(*args, **kwargs)\\n File
> > "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py",
line
> > 169, in execute\\n return execute_root(*cmd, **kwargs)\\n File
> > "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line
> > 207, in _wrap\\n return self.channel.remote_call(name, args,
> > kwargs)\\n File
> > "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202,
in
> > remote_call\\n raise
> > exc_type(*result[2])\\noslo_concurrency.processutils.ProcessExecutionError:
> > Unexpected error while running command.\\nCommand: rbd map
> > volume-36f5eb75-329e-4bd2-88d0-6f0bfe5d1040 --pool ovirt-test --conf
> > /tmp/brickrbd_RmBvxA --id None --mon_host xxx.xxx.216.45:6789 --mon_host
> > xxx.xxx.216.54:6789 --mon_host xxx.xxx.216.55:6789\\nExit code:
> > 22\\nStdout: u\\\'In some cases useful info is found in syslog - try
> > "dmesg | tail".\\\\n\\\'\\nStderr: u"2019-04-01
15:27:30.743196
> > 7fe0b4632d40 -1 auth: unable to find a keyring on
> >
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> > (2) No such file or directory\\\\nrbd: sysfs write failed\\\\n2019-04-01
> > 15:27:30.746987 7fe0b4632d40 -1 auth: unable to find a keyring on
> >
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> > (2) No such file or directory\\\\n2019-04-01 15:27:30.747896
> > 7fe0b4632d40 -1 monclient: authenticate NOTE: no keyring found; disabled
> > cephx authentication\\\\n2019-04-01 15:27:30.747903 7fe0b4632d40 0
> > librados: client.None authentication error (95) Operation not
> > supported\\\\nrbd: couldn\\\'t connect to the cluster!\\\\nrbd: map
> > failed: (22) Invalid argument\\\\n"\\n\'',)
> >
> > I tried to provide a /etc/ceph directory with ceph.conf and client
> > keyring on hypervisors (as configured in driver options). This didn't
> > solve it and doesn't seem to be the right way as the mentioned
> > /tmp/brickrbd_RmBvxA contains the needed keyring data. Please give me
> > some advice what's wrong.
> >
> > thx
> > Matthias
> >
> >
7:52 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
No, I didn't...
I wasn't used to using both "rbd_user" and "rbd_keyring_conf" (I
don't
use "rbd_keyring_conf" in standalone Cinder), nevermind
After fixing that and dealing with the rbd feature issues I could
proudly start my first VM with a cinderlib provisioned disk :-)
Thanks for help!
I'll keep posting my experiences concerning cinderlib to this list.
Matthias
Am 01.04.19 um 16:24 schrieb Benny Zlotnik:
--
Matthias Leopold
IT Systems & Communications
Medizinische Universität Wien
Spitalgasse 23 / BT 88 /Ebene 00
A-1090 Wien
Tel: +43 1 40160-21241
Fax: +43 1 40160-921200
8 a.m.
New subject: trying to use Managed Block Storage in 4.3.2 with Ceph / Authentication Keys
Glad to hear it!
On Tue, Apr 2, 2019 at 3:53 PM Matthias Leopold
<matthias.leopold(a)meduniwien.ac.at> wrote:
>
> No, I didn't...
> I wasn't used to using both "rbd_user" and "rbd_keyring_conf"
(I don't
> use "rbd_keyring_conf" in standalone Cinder), nevermind
>
> After fixing that and dealing with the rbd feature issues I could
> proudly start my first VM with a cinderlib provisioned disk :-)
>
> Thanks for help!
> I'll keep posting my experiences concerning cinderlib to this list.
>
> Matthias
>
> Am 01.04.19 um 16:24 schrieb Benny Zlotnik:
> > Did you pass the rbd_user when creating the storage domain?
> >
> > On Mon, Apr 1, 2019 at 5:08 PM Matthias Leopold
> > <matthias.leopold(a)meduniwien.ac.at> wrote:
> >>
> >>
> >> Am 01.04.19 um 13:17 schrieb Benny Zlotnik:
> >>>> OK, /var/log/ovirt-engine/cinderlib/cinderlib.log says:
> >>>>
> >>>> 2019-04-01 11:14:54,925 - cinder.volume.drivers.rbd - ERROR - Error
> >>>> connecting to ceph cluster.
> >>>> Traceback (most recent call last):
> >>>> File
"/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py",
> >>>> line 337, in _do_conn
> >>>> client.connect()
> >>>> File "rados.pyx", line 885, in rados.Rados.connect
> >>>>
(/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/build/src/pybind/rados/pyrex/rados.c:9785)
> >>>> OSError: [errno 95] error connecting to the cluster
> >>>> 2019-04-01 11:14:54,930 - root - ERROR - Failure occurred when
trying to
> >>>> run command 'storage_stats': Bad or unexpected response from
the storage
> >>>> volume backend API: Error connecting to ceph cluster.
> >>>>
> >>>> I don't really know what to do with that either.
> >>>> BTW, the cinder version on engine host is "pike"
> >>>> (openstack-cinder-11.2.0-1.el7.noarch)
> >>> Not sure if the version is related (I know it's been tested with
> >>> pike), but you can try and install the latest rocky (that's what I
use
> >>> for development)
> >>
> >> I upgraded cinder on engine and hypervisors to rocky and installed
> >> missing "ceph-common" packages on hypervisors. I set
"rbd_keyring_conf"
> >> and "rbd_ceph_conf" as indicated and got as far as adding a
"Managed
> >> Block Storage" domain and creating a disk (which is also visible
through
> >> "rbd ls"). I used a keyring that is only authorized for the pool
I
> >> specified with "rbd_pool". When I try to start the VM it fails and
I see
> >> the following in supervdsm.log on hypervisor:
> >>
> >> ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error
> >> executing helper: Command
[\'/usr/libexec/vdsm/managedvolume-helper\',
> >> \'attach\'] failed with rc=1 out=\'\'
err=\'oslo.privsep.daemon: Running
> >> privsep helper: [\\\'sudo\\\', \\\'privsep-helper\\\',
> >> \\\'--privsep_context\\\',
\\\'os_brick.privileged.default\\\',
> >> \\\'--privsep_sock_path\\\',
> >> \\\'/tmp/tmp5S8zZV/privsep.sock\\\']\\noslo.privsep.daemon: Spawned
new
> >> privsep daemon via rootwrap\\noslo.privsep.daemon: privsep daemon
> >> starting\\noslo.privsep.daemon: privsep process running with uid/gid:
> >> 0/0\\noslo.privsep.daemon: privsep process running with capabilities
> >> (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none\\noslo.privsep.daemon:
> >> privsep daemon running as pid 15944\\nTraceback (most recent call
> >> last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line
154, in
> >> <module>\\n sys.exit(main(sys.argv[1:]))\\n File
> >> "/usr/libexec/vdsm/managedvolume-helper", line 77, in main\\n
> >> args.command(args)\\n File
"/usr/libexec/vdsm/managedvolume-helper",
> >> line 137, in attach\\n attachment =
> >> conn.connect_volume(conn_info[\\\'data\\\'])\\n File
> >> "/usr/lib/python2.7/site-packages/vdsm/storage/nos_brick.py", line
96,
> >> in connect_volume\\n run_as_root=True)\\n File
> >> "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52,
in
> >> _execute\\n result = self.__execute(*args, **kwargs)\\n File
> >>
"/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line
> >> 169, in execute\\n return execute_root(*cmd, **kwargs)\\n File
> >> "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py",
line
> >> 207, in _wrap\\n return self.channel.remote_call(name, args,
> >> kwargs)\\n File
> >> "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line
202, in
> >> remote_call\\n raise
> >> exc_type(*result[2])\\noslo_concurrency.processutils.ProcessExecutionError:
> >> Unexpected error while running command.\\nCommand: rbd map
> >> volume-36f5eb75-329e-4bd2-88d0-6f0bfe5d1040 --pool ovirt-test --conf
> >> /tmp/brickrbd_RmBvxA --id None --mon_host xxx.xxx.216.45:6789 --mon_host
> >> xxx.xxx.216.54:6789 --mon_host xxx.xxx.216.55:6789\\nExit code:
> >> 22\\nStdout: u\\\'In some cases useful info is found in syslog - try
> >> "dmesg | tail".\\\\n\\\'\\nStderr: u"2019-04-01
15:27:30.743196
> >> 7fe0b4632d40 -1 auth: unable to find a keyring on
> >>
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> >> (2) No such file or directory\\\\nrbd: sysfs write failed\\\\n2019-04-01
> >> 15:27:30.746987 7fe0b4632d40 -1 auth: unable to find a keyring on
> >>
/etc/ceph/ceph.client.None.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> >> (2) No such file or directory\\\\n2019-04-01 15:27:30.747896
> >> 7fe0b4632d40 -1 monclient: authenticate NOTE: no keyring found; disabled
> >> cephx authentication\\\\n2019-04-01 15:27:30.747903 7fe0b4632d40 0
> >> librados: client.None authentication error (95) Operation not
> >> supported\\\\nrbd: couldn\\\'t connect to the cluster!\\\\nrbd: map
> >> failed: (22) Invalid argument\\\\n"\\n\'',)
> >>
> >> I tried to provide a /etc/ceph directory with ceph.conf and client
> >> keyring on hypervisors (as configured in driver options). This didn't
> >> solve it and doesn't seem to be the right way as the mentioned
> >> /tmp/brickrbd_RmBvxA contains the needed keyring data. Please give me
> >> some advice what's wrong.
> >>
> >> thx
> >> Matthias
> >>
> >>
>
> --
> Matthias Leopold
> IT Systems & Communications
> Medizinische Universität Wien
> Spitalgasse 23 / BT 88 /Ebene 00
> A-1090 Wien
> Tel: +43 1 40160-21241
> Fax: +43 1 40160-921200
2255
days inactive
2256
days old
8 comments
2 participants
participants (2)
-
Benny Zlotnik -
Matthias Leopold