Hi, as you might already know I try to use ceph with openstack in an oVirt test environment. I'm able to create and remove volumes. But if I try to run a VM which contains a ceph volume it is in the "Wait for launch" state for a very long time. Then it gets into "down" state again. The qemu log states 2018-05-25T15:03:41.100401Z qemu-kvm: -drive file=rbd:rbd/volume-3bec499e-d0d0-45ef-86ad-2c187cdb2811:id=cinder:auth_supported=cephx\;none:mon_host=[mon0]\:6789\;[mon1]\:6789,file.password-secret=scsi0-0-0-0-secret0,format=raw,if=none,id=drive-scsi0-0-0-0,serial=3bec499e-d0d0-45ef-86ad-2c187cdb2811,cache=none,werror=stop,rerror=stop,aio=threads: error connecting: Connection timed out 2018-05-25 15:03:41.109+0000: shutting down, reason=failed On the monitor hosts I see traffic with the ceph-mon-port, but not on other ports (the osds for example). In the ceph logs however I don't really see what happens. Do you have some tips how to debug this problem? Regards Bernhard
Hi, I found the reason for my timeout problems: It is the version of librbd1 (which is 0.94.5) in conjunction with my CEPH test-environment which is running the luminous release. When I install the librbd1 (and librados2) packages from the centos-ceph-luminous repository (version 12.2.5) I'm able to start and migrate VMs inbetween the hosts. Regards Bernhard Am 25.05.2018 um 17:08 schrieb Bernhard Dick:
Hi,
as you might already know I try to use ceph with openstack in an oVirt test environment. I'm able to create and remove volumes. But if I try to run a VM which contains a ceph volume it is in the "Wait for launch" state for a very long time. Then it gets into "down" state again. The qemu log states
2018-05-25T15:03:41.100401Z qemu-kvm: -drive file=rbd:rbd/volume-3bec499e-d0d0-45ef-86ad-2c187cdb2811:id=cinder:auth_supported=cephx\;none:mon_host=[mon0]\:6789\;[mon1]\:6789,file.password-secret=scsi0-0-0-0-secret0,format=raw,if=none,id=drive-scsi0-0-0-0,serial=3bec499e-d0d0-45ef-86ad-2c187cdb2811,cache=none,werror=stop,rerror=stop,aio=threads: error connecting: Connection timed out
2018-05-25 15:03:41.109+0000: shutting down, reason=failed
On the monitor hosts I see traffic with the ceph-mon-port, but not on other ports (the osds for example). In the ceph logs however I don't really see what happens. Do you have some tips how to debug this problem?
Regards Bernhard
On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de> wrote:
Hi,
I found the reason for my timeout problems: It is the version of librbd1 (which is 0.94.5) in conjunction with my CEPH test-environment which is running the luminous release. When I install the librbd1 (and librados2) packages from the centos-ceph-luminous repository (version 12.2.5) I'm able to start and migrate VMs inbetween the hosts.
vdsm does not require librbd since qemu brings this dependency, and vdsm does not access ceph directly yet. Maybe qemu should require newer version of librbd? Nir
Regards Bernhard
Am 25.05.2018 um 17:08 schrieb Bernhard Dick:
Hi,
as you might already know I try to use ceph with openstack in an oVirt test environment. I'm able to create and remove volumes. But if I try to run a VM which contains a ceph volume it is in the "Wait for launch" state for a very long time. Then it gets into "down" state again. The qemu log states
2018-05-25T15:03:41.100401Z qemu-kvm: -drive
file=rbd:rbd/volume-3bec499e-d0d0-45ef-86ad-2c187cdb2811:id=cinder:auth_supported=cephx\;none:mon_host=[mon0]\:6789\;[mon1]\:6789,file.password-secret=scsi0-0-0-0-secret0,format=raw,if=none,id=drive-scsi0-0-0-0,serial=3bec499e-d0d0-45ef-86ad-2c187cdb2811,cache=none,werror=stop,rerror=stop,aio=threads:
error connecting: Connection timed out
2018-05-25 15:03:41.109+0000: shutting down, reason=failed
On the monitor hosts I see traffic with the ceph-mon-port, but not on other ports (the osds for example). In the ceph logs however I don't really see what happens. Do you have some tips how to debug this problem?
Regards Bernhard
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N6ODADRIIYRJPS...
On Thu, May 31, 2018 at 11:02:01PM +0300, Nir Soffer wrote:
On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de> wrote:
Hi,
I found the reason for my timeout problems: It is the version of librbd1 (which is 0.94.5) in conjunction with my CEPH test-environment which is running the luminous release. When I install the librbd1 (and librados2) packages from the centos-ceph-luminous repository (version 12.2.5) I'm able to start and migrate VMs inbetween the hosts.
vdsm does not require librbd since qemu brings this dependency, and vdsm does not access ceph directly yet.
Maybe qemu should require newer version of librbd?
Upstream QEMU builds against any librbd version that exports the necessary APIs. The choice of library versions is mostly up to distro package maintainers. Have you filed a bug against Ceph on the distro you are using? Stefan
Regards Bernhard
Am 25.05.2018 um 17:08 schrieb Bernhard Dick:
Hi,
as you might already know I try to use ceph with openstack in an oVirt test environment. I'm able to create and remove volumes. But if I try to run a VM which contains a ceph volume it is in the "Wait for launch" state for a very long time. Then it gets into "down" state again. The qemu log states
2018-05-25T15:03:41.100401Z qemu-kvm: -drive
file=rbd:rbd/volume-3bec499e-d0d0-45ef-86ad-2c187cdb2811:id=cinder:auth_supported=cephx\;none:mon_host=[mon0]\:6789\;[mon1]\:6789,file.password-secret=scsi0-0-0-0-secret0,format=raw,if=none,id=drive-scsi0-0-0-0,serial=3bec499e-d0d0-45ef-86ad-2c187cdb2811,cache=none,werror=stop,rerror=stop,aio=threads:
error connecting: Connection timed out
2018-05-25 15:03:41.109+0000: shutting down, reason=failed
On the monitor hosts I see traffic with the ceph-mon-port, but not on other ports (the osds for example). In the ceph logs however I don't really see what happens. Do you have some tips how to debug this problem?
Regards Bernhard
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N6ODADRIIYRJPS...
On Fri, Jun 1, 2018 at 3:54 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de> wrote:
Hi,
I found the reason for my timeout problems: It is the version of
On Thu, May 31, 2018 at 11:02:01PM +0300, Nir Soffer wrote: librbd1
(which is 0.94.5) in conjunction with my CEPH test-environment which is running the luminous release. When I install the librbd1 (and librados2) packages from the centos-ceph-luminous repository (version 12.2.5) I'm able to start and migrate VMs inbetween the hosts.
vdsm does not require librbd since qemu brings this dependency, and vdsm does not access ceph directly yet.
Maybe qemu should require newer version of librbd?
Upstream QEMU builds against any librbd version that exports the necessary APIs.
The choice of library versions is mostly up to distro package maintainers.
Have you filed a bug against Ceph on the distro you are using?
Thanks for clearing this up Stefan. Bernhard, can you give more info about your Linux version and installed packages (.e.g qemu-*)? Nir
Hi,
as you might already know I try to use ceph with openstack in an oVirt test environment. I'm able to create and remove volumes. But if I
Am 25.05.2018 um 17:08 schrieb Bernhard Dick: try to
run a VM which contains a ceph volume it is in the "Wait for launch" state for a very long time. Then it gets into "down" state again. The qemu log states
2018-05-25T15:03:41.100401Z qemu-kvm: -drive
file=rbd:rbd/volume-3bec499e-d0d0-45ef-86ad-2c187cdb2811:id=cinder:auth_supported=cephx\;none:mon_host=[mon0]\:6789\;[mon1]\:6789,file.password-secret=scsi0-0-0-0-secret0,format=raw,if=none,id=drive-scsi0-0-0-0,serial=3bec499e-d0d0-45ef-86ad-2c187cdb2811,cache=none,werror=stop,rerror=stop,aio=threads:
error connecting: Connection timed out
2018-05-25 15:03:41.109+0000: shutting down, reason=failed
On the monitor hosts I see traffic with the ceph-mon-port, but not on other ports (the osds for example). In the ceph logs however I don't really see what happens. Do you have some tips how to debug this problem?
Regards Bernhard
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N6ODADRIIYRJPS...
Hi, Am 05.06.2018 um 22:11 schrieb Nir Soffer:
On Fri, Jun 1, 2018 at 3:54 PM Stefan Hajnoczi <stefanha@gmail.com <mailto:stefanha@gmail.com>> wrote:
On Thu, May 31, 2018 at 11:02:01PM +0300, Nir Soffer wrote: > On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de <mailto:bernhard@bdick.de>> wrote: > > > Hi, > > > > I found the reason for my timeout problems: It is the version of librbd1 > > (which is 0.94.5) in conjunction with my CEPH test-environment which is > > running the luminous release. > > When I install the librbd1 (and librados2) packages from the > > centos-ceph-luminous repository (version 12.2.5) I'm able to start and > > migrate VMs inbetween the hosts. > > > > vdsm does not require librbd since qemu brings this dependency, and vdsm > does not access ceph directly yet. > > Maybe qemu should require newer version of librbd?
Upstream QEMU builds against any librbd version that exports the necessary APIs.
The choice of library versions is mostly up to distro package maintainers.
Have you filed a bug against Ceph on the distro you are using?
Thanks for clearing this up Stefan.
Bernhard, can you give more info about your Linux version and installed packages (.e.g qemu-*)? Sure. I have two test-systems. The first is running a stock oVirt Node 4.3 which states "CentOS Linux release 7.5.1804 (Core)" as version string. The qemu and ceph packages are: Name : qemu-img-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1
Name : qemu-kvm-common-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1 Name : qemu-kvm-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1 Name : librados2 Arch : x86_64 Epoch : 1 Version : 0.94.5 Release : 2.el7 Name : librbd1 Arch : x86_64 Epoch : 1 Version : 0.94.5 Release : 2.el7 The Centos 7 system is a centos minimal installation with the following repos being enabled: CentOS-7 - Base CentOS-7 - Updates CentOS-7 - Extras ovirt-4.2-epel ovirt-4.2-centos-gluster123 ovirt-4.2-virtio-win-latest ovirt-4.2-centos-qemu-ev ovirt-4.2-centos-opstools centos-sclo-rh-release ovirt-4.2-centos-ovirt42 ovirt-4.2 The version numbers for the qemu packages are the same as above as they're from the ovirt-4.2-centos-qemu-ev repository. Also the version numbers for librados2 and librbd1 match, while they're from the centos-base (instead of ovirt-base) repository. When I activate the ceph-centos-luminous repository librbd1 and librados2 get upgraded to the following versions (leaving the qemu packages untouched, what is as expected): Name : librados2 Arch : x86_64 Epoch : 2 Version : 12.2.5 Release : 0.el7 Name : librbd1 Arch : x86_64 Epoch : 2 Version : 12.2.5 Release : 0.el7 So from my perspective on side of oVirt there should be thaught about a way to add a more recent ceph library version into the ovirt node image, as it is not the most common task to add extra repositories here (and I'm not whether that might break the image based upgrade-path). I will go for Centos based hosts in my case, as I'm a bit more flexible than so at least for me there is no real need to get the above noted implemented quickly :-) Regards Bernhard
On Wed, Jun 6, 2018 at 7:20 AM, Bernhard Dick <bernhard@bdick.de> wrote:
Hi,
Am 05.06.2018 um 22:11 schrieb Nir Soffer:
On Fri, Jun 1, 2018 at 3:54 PM Stefan Hajnoczi <stefanha@gmail.com <mailto:stefanha@gmail.com>> wrote:
On Thu, May 31, 2018 at 11:02:01PM +0300, Nir Soffer wrote: > On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de <mailto:bernhard@bdick.de>> wrote: > > > Hi, > > > > I found the reason for my timeout problems: It is the version of librbd1 > > (which is 0.94.5) in conjunction with my CEPH test-environment which is > > running the luminous release. > > When I install the librbd1 (and librados2) packages from the > > centos-ceph-luminous repository (version 12.2.5) I'm able to start and > > migrate VMs inbetween the hosts. > > > > vdsm does not require librbd since qemu brings this dependency, and vdsm > does not access ceph directly yet. > > Maybe qemu should require newer version of librbd?
Upstream QEMU builds against any librbd version that exports the necessary APIs.
The choice of library versions is mostly up to distro package maintainers.
Have you filed a bug against Ceph on the distro you are using?
Thanks for clearing this up Stefan.
Bernhard, can you give more info about your Linux version and installed packages (.e.g qemu-*)?
Sure. I have two test-systems. The first is running a stock oVirt Node 4.3 which states "CentOS Linux release 7.5.1804 (Core)" as version string. The qemu and ceph packages are: Name : qemu-img-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1
Name : qemu-kvm-common-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1
Name : qemu-kvm-ev Arch : x86_64 Epoch : 10 Version : 2.10.0 Release : 21.el7_5.3.1
Name : librados2 Arch : x86_64 Epoch : 1 Version : 0.94.5 Release : 2.el7
Name : librbd1 Arch : x86_64 Epoch : 1 Version : 0.94.5 Release : 2.el7
The Centos 7 system is a centos minimal installation with the following repos being enabled: CentOS-7 - Base CentOS-7 - Updates CentOS-7 - Extras ovirt-4.2-epel ovirt-4.2-centos-gluster123 ovirt-4.2-virtio-win-latest ovirt-4.2-centos-qemu-ev ovirt-4.2-centos-opstools centos-sclo-rh-release ovirt-4.2-centos-ovirt42 ovirt-4.2
The version numbers for the qemu packages are the same as above as they're from the ovirt-4.2-centos-qemu-ev repository. Also the version numbers for librados2 and librbd1 match, while they're from the centos-base (instead of ovirt-base) repository.
When I activate the ceph-centos-luminous repository librbd1 and librados2 get upgraded to the following versions (leaving the qemu packages untouched, what is as expected): Name : librados2 Arch : x86_64 Epoch : 2 Version : 12.2.5 Release : 0.el7
Name : librbd1 Arch : x86_64 Epoch : 2 Version : 12.2.5 Release : 0.el7
So from my perspective on side of oVirt there should be thaught about a way to add a more recent ceph library version into the ovirt node image, as it is not the most common task to add extra repositories here (and I'm not whether that might break the image based upgrade-path).
I will go for Centos based hosts in my case, as I'm a bit more flexible than so at least for me there is no real need to get the above noted implemented quickly :-)
Without knowing more about your setup, I suspect you might be using unsupported tunables which are preventing the Hammer clients from connecting to the Luminous cluster. Note that upstream, we really only test client/server backwards compatibility between N - 2 releases (it was N - 1) [1]. Therefore, in a perfect world, you should be selecting the best Ceph client repo to match your Ceph cluster version. There is an open BZ against RHEL 7 to upgrade the base Ceph client RPMs to a more recent version [2], but in the interim if you need newer client RPMs for CentOS 7, you will need to get them from upstream Ceph by picking the appropriate release repository.
Regards Bernhard
[1] http://docs.ceph.com/docs/master/releases/schedule/#stable-releases-x-2-z [2] https://bugzilla.redhat.com/show_bug.cgi?id=1421783 -- Jason
Hi, Am 01.06.2018 um 14:54 schrieb Stefan Hajnoczi:
On Thu, May 31, 2018 at 11:02:01PM +0300, Nir Soffer wrote:
On Thu, May 31, 2018 at 1:55 AM Bernhard Dick <bernhard@bdick.de> wrote:
Hi,
I found the reason for my timeout problems: It is the version of librbd1 (which is 0.94.5) in conjunction with my CEPH test-environment which is running the luminous release. When I install the librbd1 (and librados2) packages from the centos-ceph-luminous repository (version 12.2.5) I'm able to start and migrate VMs inbetween the hosts.
vdsm does not require librbd since qemu brings this dependency, and vdsm does not access ceph directly yet.
Maybe qemu should require newer version of librbd?
Upstream QEMU builds against any librbd version that exports the necessary APIs.
The choice of library versions is mostly up to distro package maintainers.
Have you filed a bug against Ceph on the distro you are using? At least I didn't file a bug, as I'm not sure whether this might even desired. And if supplying a newer version of librbd within the base repository leads to problems with older clusters. The 0.94.5 version is from the base repository (On Centos 7 and oVirt Node 4.3).
Regards Bernhard
participants (4)
-
Bernhard Dick -
Jason Dillaman -
Nir Soffer -
Stefan Hajnoczi