Hi Strahil, Thanks for your reply, I have 16 nodes for now but more on the way. The reason why Ceph appeals me over Gluster because of the following reasons. 1. I have more experience with Ceph than Gluster. 2. I heard in Managed Block Storage presentation that it leverages storage software to offload storage related tasks. 3. Adding Gluster storage limits to 3 hosts at a time. 4. I read that there is a limit of maximum 12 hosts in Gluster setup. No such limitation if I go via Ceph. In my initial testing I was able to enable Centos repositories in Node Ng but if I remember correctly, there were some librbd versions present in Node Ng which clashed with the version I was trying to install. Does Ceph hyperconverge still make sense? Regards Shantur

wrote: > Hi Shantur, > > the main question is how many nodes you have. > Ceph integration is still in development/experimental and it should be > wise to consider Gluster also. It has a great integration and it's quite > easy to work with). > > > There are users reporting using CEPH with their oVirt , but I can't tell > how good it is. > I doubt that oVirt nodes come with CEPH components , so you most probably > will need to use a full-blown distro. In general, using extra software on > oVirt nodes is quite hard . > > With such setup, you will need much more nodes than a Gluster setup due > to CEPH's requirements. > > Best Regards, > Strahil Nikolov > > > > > > > В неделя, 17 януари 2021 г., 10:37:57 Гринуич+2, Shantur Rathore < > shantur.rathore(a)gmail.com&gt; написа: > > > > > > Hi all, > > I am planning my new oVirt cluster on Apple hosts. These hosts can only > have one disk which I plan to partition and use for hyper converged setup. > As this is my first oVirt cluster I need help in understanding few bits. > > 1. Is Hyper converged setup possible with Ceph using cinderlib? > 2. Can this hyper converged setup be on oVirt Node Next hosts or only > Centos? > 3. Can I install cinderlib on oVirt Node Next hosts? > 4. Are there any pit falls in such a setup? > > > Thanks for your help > > Regards, > Shantur > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/TPQCJSJ3MQO... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/RVKIBASSQW7... >

В 15:51 +0000 на 17.01.2021 (нд), Shantur Rathore написа: Hi Strahil, Thanks for your reply, I have 16 nodes for now but more on the way. The reason why Ceph appeals me over Gluster because of the following reasons. 1. I have more experience with Ceph than Gluster. That is a good reason to pick CEPH. 2. I heard in Managed Block Storage presentation that it leverages storage software to offload storage related tasks. 3. Adding Gluster storage limits to 3 hosts at a time. Only if you wish the nodes to be both Storage and Compute. Yet, you can add as many as you wish as a compute node (won't be part of Gluster) and later you can add them to the Gluster TSP (this requires 3 nodes at a time). 4. I read that there is a limit of maximum 12 hosts in Gluster setup. No such limitation if I go via Ceph. Actually , it's about Red Hat support for RHHI and not for Gluster + oVirt. As both oVirt and Gluster ,that are used, are upstream projects, support is on best effort from the community. In my initial testing I was able to enable Centos repositories in Node Ng but if I remember correctly, there were some librbd versions present in Node Ng which clashed with the version I was trying to install. Does Ceph hyperconverge still make sense? Yes it is. You got the knowledge to run the CEPH part, yet consider talking with some of the devs on the list - as there were some changes recently in oVirt's support for CEPH. Regards Shantur On Sun, Jan 17, 2021, 9:58 AM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: Hi Shantur, the main question is how many nodes you have. Ceph integration is still in development/experimental and it should be wise to consider Gluster also. It has a great integration and it's quite easy to work with). There are users reporting using CEPH with their oVirt , but I can't tell how good it is. I doubt that oVirt nodes come with CEPH components , so you most probably will need to use a full-blown distro. In general, using extra software on oVirt nodes is quite hard . With such setup, you will need much more nodes than a Gluster setup due to CEPH's requirements. Best Regards, Strahil Nikolov В неделя, 17 януари 2021 г., 10:37:57 Гринуич+2, Shantur Rathore < shantur.rathore(a)gmail.com&gt; написа: Hi all, I am planning my new oVirt cluster on Apple hosts. These hosts can only have one disk which I plan to partition and use for hyper converged setup. As this is my first oVirt cluster I need help in understanding few bits. 1. Is Hyper converged setup possible with Ceph using cinderlib? 2. Can this hyper converged setup be on oVirt Node Next hosts or only Centos? 3. Can I install cinderlib on oVirt Node Next hosts? 4. Are there any pit falls in such a setup? Thanks for your help Regards, Shantur _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TPQCJSJ3MQO... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RVKIBASSQW7... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4IBXGXZVXAI...

Beware about Ceph and oVirt Managed Block Storage, current integration is only possible with kernel, not with qemu-rbd. k Sent from my iPhone On 18 Jan 2021, at 13:00, Shantur Rathore <rathore4u(a)gmail.com&gt; wrote:  Thanks Strahil for your reply. Sorry just to confirm, 1. Are you saying Ceph on oVirt Node NG isn't possible? 2. Would you know which devs would be best to ask about the recent Ceph changes? Thanks, Shantur On Sun, Jan 17, 2021 at 4:46 PM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: > В 15:51 +0000 на 17.01.2021 (нд), Shantur Rathore написа: > > Hi Strahil, > > Thanks for your reply, I have 16 nodes for now but more on the way. > > The reason why Ceph appeals me over Gluster because of the following > reasons. > > 1. I have more experience with Ceph than Gluster. > > That is a good reason to pick CEPH. > > 2. I heard in Managed Block Storage presentation that it leverages > storage software to offload storage related tasks. > 3. Adding Gluster storage limits to 3 hosts at a time. > > Only if you wish the nodes to be both Storage and Compute. Yet, you can > add as many as you wish as a compute node (won't be part of Gluster) and > later you can add them to the Gluster TSP (this requires 3 nodes at a time). > > 4. I read that there is a limit of maximum 12 hosts in Gluster setup. No > such limitation if I go via Ceph. > > Actually , it's about Red Hat support for RHHI and not for Gluster + > oVirt. As both oVirt and Gluster ,that are used, are upstream projects, > support is on best effort from the community. > > In my initial testing I was able to enable Centos repositories in Node Ng > but if I remember correctly, there were some librbd versions present in > Node Ng which clashed with the version I was trying to install. > Does Ceph hyperconverge still make sense? > > Yes it is. You got the knowledge to run the CEPH part, yet consider > talking with some of the devs on the list - as there were some changes > recently in oVirt's support for CEPH. > > Regards > Shantur > > On Sun, Jan 17, 2021, 9:58 AM Strahil Nikolov via Users <users(a)ovirt.org&gt; > wrote: > > Hi Shantur, > > the main question is how many nodes you have. > Ceph integration is still in development/experimental and it should be > wise to consider Gluster also. It has a great integration and it's quite > easy to work with). > > > There are users reporting using CEPH with their oVirt , but I can't tell > how good it is. > I doubt that oVirt nodes come with CEPH components , so you most probably > will need to use a full-blown distro. In general, using extra software on > oVirt nodes is quite hard . > > With such setup, you will need much more nodes than a Gluster setup due > to CEPH's requirements. > > Best Regards, > Strahil Nikolov > > > > > > > В неделя, 17 януари 2021 г., 10:37:57 Гринуич+2, Shantur Rathore < > shantur.rathore(a)gmail.com&gt; написа: > > > > > > Hi all, > > I am planning my new oVirt cluster on Apple hosts. These hosts can only > have one disk which I plan to partition and use for hyper converged setup. > As this is my first oVirt cluster I need help in understanding few bits. > > 1. Is Hyper converged setup possible with Ceph using cinderlib? > 2. Can this hyper converged setup be on oVirt Node Next hosts or only > Centos? > 3. Can I install cinderlib on oVirt Node Next hosts? > 4. Are there any pit falls in such a setup? > > > Thanks for your help > > Regards, > Shantur > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/TPQCJSJ3MQO... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/RVKIBASSQW7... > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/4IBXGXZVXAI... > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6WBVRC4GJTA...

В 15:51 +0000 на 17.01.2021 (нд), Shantur Rathore написа: > Hi Strahil, > > Thanks for your reply, I have 16 nodes for now but more on the way. > > The reason why Ceph appeals me over Gluster because of the following reasons. > > 1. I have more experience with Ceph than Gluster. That is a good reason to pick CEPH. > 2. I heard in Managed Block Storage presentation that it leverages storage software to offload storage related tasks.  > 3. Adding Gluster storage limits to 3 hosts at a time. Only if you wish the nodes to be both Storage and Compute. Yet, you can add as many as you wish as a compute node (won't be part of Gluster) and later you can add them to the Gluster TSP (this requires 3 nodes at a time). > 4. I read that there is a limit of maximum 12 hosts in Gluster setup. No such limitation if I go via Ceph. Actually , it's about Red Hat support for RHHI and not for Gluster + oVirt. As both oVirt and Gluster ,that are used, are upstream projects, support is on best effort from the community. > In my initial testing I was able to enable Centos repositories in Node Ng but if I remember correctly, there were some librbd versions present in Node Ng which clashed with the version I was trying to install. > Does Ceph hyperconverge still make sense? Yes it is. You got the knowledge to run the CEPH part, yet consider talking with some of the devs on the list - as there were some changes recently in oVirt's support for CEPH. > Regards > Shantur > > On Sun, Jan 17, 2021, 9:58 AM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: >> Hi Shantur, >> >> the main question is how many nodes you have. >> Ceph integration is still in development/experimental and it should be wise to consider Gluster also. It has a great integration and it's quite easy to work with). >> >> >> There are users reporting using CEPH with their oVirt , but I can't tell how good it is. >> I doubt that oVirt nodes come with CEPH components , so you most probably will need to use a full-blown distro. In general, using extra software on oVirt nodes is quite hard . >> >> With such setup, you will need much more nodes than a Gluster setup due to CEPH's requirements. >> >> Best Regards, >> Strahil Nikolov >> >> >> >> >> >> >> В неделя, 17 януари 2021 г., 10:37:57 Гринуич+2, Shantur Rathore <shantur.rathore(a)gmail.com&gt; написа: >> >> >> >> >> >> Hi all, >> >> I am planning my new oVirt cluster on Apple hosts. These hosts can only have one disk which I plan to partition and use for hyper converged setup. As this is my first oVirt cluster I need help in understanding few bits. >> >> 1. Is Hyper converged setup possible with Ceph using cinderlib? >> 2. Can this hyper converged setup be on oVirt Node Next hosts or only Centos? >> 3. Can I install cinderlib on oVirt Node Next hosts? >> 4. Are there any pit falls in such a setup? >> >> >> Thanks for your help >> >> Regards, >> Shantur >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TPQCJSJ3MQO... >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RVKIBASSQW7... >> > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4IBXGXZVXAI...

Thanks Strahil for your reply. Sorry just to confirm, 1. Are you saying Ceph on oVirt Node NG isn't possible? 2. Would you know which devs would be best to ask about the recent Ceph changes? Thanks, Shantur On Sun, Jan 17, 2021 at 4:46 PM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: > > В 15:51 +0000 на 17.01.2021 (нд), Shantur Rathore написа: > > Hi Strahil, > > Thanks for your reply, I have 16 nodes for now but more on the way. > > The reason why Ceph appeals me over Gluster because of the following reasons. > > 1. I have more experience with Ceph than Gluster. > > That is a good reason to pick CEPH. > > 2. I heard in Managed Block Storage presentation that it leverages storage software to offload storage related tasks. > 3. Adding Gluster storage limits to 3 hosts at a time. > > Only if you wish the nodes to be both Storage and Compute. Yet, you can add as many as you wish as a compute node (won't be part of Gluster) and later you can add them to the Gluster TSP (this requires 3 nodes at a time). > > 4. I read that there is a limit of maximum 12 hosts in Gluster setup. No such limitation if I go via Ceph. > > Actually , it's about Red Hat support for RHHI and not for Gluster + oVirt. As both oVirt and Gluster ,that are used, are upstream projects, support is on best effort from the community. > > In my initial testing I was able to enable Centos repositories in Node Ng but if I remember correctly, there were some librbd versions present in Node Ng which clashed with the version I was trying to install. > Does Ceph hyperconverge still make sense? > > Yes it is. You got the knowledge to run the CEPH part, yet consider talking with some of the devs on the list - as there were some changes recently in oVirt's support for CEPH. > > Regards > Shantur > > On Sun, Jan 17, 2021, 9:58 AM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: > > Hi Shantur, > > the main question is how many nodes you have. > Ceph integration is still in development/experimental and it should be wise to consider Gluster also. It has a great integration and it's quite easy to work with). > > > There are users reporting using CEPH with their oVirt , but I can't tell how good it is. > I doubt that oVirt nodes come with CEPH components , so you most probably will need to use a full-blown distro. In general, using extra software on oVirt nodes is quite hard . > > With such setup, you will need much more nodes than a Gluster setup due to CEPH's requirements. > > Best Regards, > Strahil Nikolov > > > > > > > В неделя, 17 януари 2021 г., 10:37:57 Гринуич+2, Shantur Rathore <shantur.rathore(a)gmail.com&gt; написа: > > > > > > Hi all, > > I am planning my new oVirt cluster on Apple hosts. These hosts can only have one disk which I plan to partition and use for hyper converged setup. As this is my first oVirt cluster I need help in understanding few bits. > > 1. Is Hyper converged setup possible with Ceph using cinderlib? > 2. Can this hyper converged setup be on oVirt Node Next hosts or only Centos? > 3. Can I install cinderlib on oVirt Node Next hosts? > 4. Are there any pit falls in such a setup? > > > Thanks for your help > > Regards, > Shantur > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TPQCJSJ3MQO... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RVKIBASSQW7... > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4IBXGXZVXAI... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6WBVRC4GJTA...

Shantur, I recommend to look to OpenStack or some OpenNebula/Proxmox if you wan’t use Ceph Storage. Current storage team support in oVirt just can break something and do not work with this anymore, take a look what I talking about: in [1], [2], [3] k [1] https://bugzilla.redhat.com/show_bug.cgi?id=1899453 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1899453 [3] https://bugzilla.redhat.com/show_bug.cgi?id=1899453

I recommend to look to OpenStack or some OpenNebula/Proxmox if you wan’t use Ceph Storage.

Most probably it will be easier if you stick with full-blown distro.

Moreover, it is not possible to use a pure Managed Block Storage setup at all, there has to be at least one regular storage domain in a datacenter

Yep, BZ is https://bugzilla.redhat.com/show_bug.cgi?id=1539837 https://bugzilla.redhat.com/show_bug.cgi?id=1904669 https://bugzilla.redhat.com/show_bug.cgi?id=1905113 Thanks, k On 19 Jan 2021, at 11:05, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: perhaps a copy paste error about the bugzilla entries? They are the same number...

Thanks for pointing out the requirement for Master domain. In theory, will I be able to satisfy the requirement with another iSCSI or >maybe Ceph iSCSI as master domain?

So each node has

- oVirt Node NG / Centos - Ceph cluster member - iSCSI or Ceph iSCSI master domain

How practical is such a setup?

Ceph iSCSI gateway should be supported since 4.1, so I think I can use it for configuring the master domain and still leveraging the same overall storage environment provided by Ceph, correct?

I can confirm that Ceph iSCSI can be used for master domain, we are using it together with VM disks on Ceph via Cinder ("old style"). Recent developments concerning Ceph in oVirt are disappointing for me, I think I will have to look elsewhere (OpenStack, Proxmox) for our rather big deployment. At least Nir Soffer's explanation for the move to cinderlib in another thread (dated 20210121) shed some light on the background of this decision. Matthias Am 19.01.21 um 12:57 schrieb Gianluca Cecchi: > On Tue, Jan 19, 2021 at 12:20 PM Benny Zlotnik <bzlotnik(a)redhat.com > <mailto:bzlotnik@redhat.com>> wrote: > > >Thanks for pointing out the requirement for Master domain. In > theory, will I be able to satisfy the requirement with another iSCSI > or >maybe Ceph iSCSI as master domain? > It should work as ovirt sees it as a regular domain, cephFS will > probably work too > > > Ceph iSCSI gateway should be supported since 4.1, so I think I can use > it for configuring the master domain and still leveraging the same > overall storage environment provided by Ceph, correct? > > https://bugzilla.redhat.com/show_bug.cgi?id=1527061 > > Gianluca > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3ASTNEGXSV7... > -- Matthias Leopold IT Systems & Communications Medizinische Universität Wien Spitalgasse 23 / BT 88 / Ebene 00 A-1090 Wien Tel: +43 1 40160-21241 Fax: +43 1 40160-921200

It should work as ovirt sees it as a regular domain, cephFS will probably work too

>Thanks for pointing out the requirement for Master domain. In theory, will I be able to satisfy the requirement with another iSCSI or >maybe Ceph iSCSI as master domain? It should work as ovirt sees it as a regular domain, cephFS will probably work too >So each node has >- oVirt Node NG / Centos >- Ceph cluster member >- iSCSI or Ceph iSCSI master domain >How practical is such a setup? Not sure, it could work, but it hasn't been tested and it's likely you are going to be the first to try it

So, after a quick dive into source code, I cannot see any mention of posix storage in hosted-engine code. I am not sure if there is a manual way of moving the locally created hosted-engine vm to POSIX storage and create a storage domain using API as it does for other types of domains while installing self-hosted engine. Regards, Shantur

Shantur, this is oVirt. You always should make master domain. It’s enough some 1GB NFS on manager side. k On 22 Jan 2021, at 12:02, Shantur Rathore <rathore4u(a)gmail.com&gt; wrote: Just a bump. Any ideas anyone?

Just for clarification: when you say Managed Block Storage you mean cinderlib integration, >correct? Is still this one below the correct reference page for 4.4? https://www.ovirt.org/develop/release-management/features/storage/cinderl...

So are the manual steps still needed (and also repo config that seems against pike)? Or do you have an updated link for configuring cinderlib in 4.4?

Is this true only for Self Hosted Engine Environment or also if I have an external engine?