David, I’m curious what the use case is. :9 you plan on using the disk with three vms at the same time? This isn’t really what shareable disks are meant to do afaik. If you want to share storage with multiple vms I’d probably just setup an nfs share on one of the vms On Thu, Apr 15, 2021 at 7:37 PM David White via Users <users@ovirt.org> wrote:

I found the proper documentation at https://www.ovirt.org/documentation/administration_guide/#Shareable_Disks.

When I tried to edit the disk, I see that sharable is grayed out, and when I hover my mouse over it, I see "Sharable Storage is not supported on Gluster/Offload Domain".

So to confirm, is there any circumstance where a Gluster volume can support sharable storage? Unfortunately, I don't have any other storage available, and I chose to use Gluster, so that I could have a HA environment.

Sent with ProtonMail <https://protonmail.com> Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, April 15, 2021 5:05 PM, David White via Users < users@ovirt.org> wrote:

I need to mount a partition across 3 different VMs. How do I attach a disk to multiple VMs?

This looks like fairly old documentation-not-documentation: https://www.ovirt.org/develop/release-management/features/storage/sharedrawd...

Sent with ProtonMail <https://protonmail.com> Secure Email.

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WZY6OJWBH5KAB5...

David, I’m curious what the use case is This is for a customer who wants as much high availability as possible for their website, which relies on a basic LAMP or LNMP stack.

The plan is to create a MariaDB Galera cluster. Each of the 3 VMs will run MariaDB, as well as Apache or Nginx (I haven't decided which, yet), and will be able to accept web traffic. So the website files will need to be the same across all 3 virtual servers. My original intent was to setup a mount point on all 3 virtual servers that mapped back to the same shared disk. Strahil, 1 idea I had, which I don't think would be ideal at all, was to setup a separate, new, gluster configuration on each of the 3 VMs. Gluster virtualized on top of gluster! If that doesn't make your head spin, what will? But I'm not seriously thinking about that. :) It did occur to me that I could setup a 4th VM to host the NFS share. But I'm trying to stay away from as many single points of failures as possible. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, April 15, 2021 7:40 PM, Strahil Nikolov via Users <users@ovirt.org> wrote:

I know that clusterizing applications (for example corosync/pacemaker Active-Passive or even GFS2) require simultaneous access to the data.

In your case you can create:

- 2 separate VMs replicating over DRBD and sharing the storage over NFS/iSCSI - Using NFS Ganesha (this is just a theory but should work) to export your Gluster volumes in a redundant and highly available way

Best Regards, Strahil Nikolov

В петък, 16 април 2021 г., 01:56:09 ч. Гринуич+3, Jayme jaymef@gmail.com написа:

David, I’m curious what the use case is. :9 you plan on using the disk with three vms at the same time?  This isn’t really what shareable disks are meant to do afaik. If you want to share storage with multiple vms I’d probably just setup an nfs share on one of the vms 

On Thu, Apr 15, 2021 at 7:37 PM David White via Users users@ovirt.org wrote:

...

I found the proper documentation at https://www.ovirt.org/documentation/administration_guide/#Shareable_Disks. When I tried to edit the disk, I see that sharable is grayed out, and when I hover my mouse over it, I see "Sharable Storage is not supported on Gluster/Offload Domain". So to confirm, is there any circumstance where a Gluster volume can support sharable storage? Unfortunately, I don't have any other storage available, and I chose to use Gluster, so that I could have a HA environment. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐   On Thursday, April 15, 2021 5:05 PM, David White via Users users@ovirt.org wrote:

...

...

I need to mount a partition across 3 different VMs. How do I attach a disk to multiple VMs? This looks like fairly old documentation-not-documentation: https://www.ovirt.org/develop/release-management/features/storage/sharedrawd... Sent with ProtonMail Secure Email.

...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WZY6OJWBH5KAB5...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3XAQI4WTRSPEEC...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AFSDOLZCLVEO6N...

Hi, another approach you may consider is to use HA functionality provided by oVirt which re-create VM on another host if original host become unavailable and attach the disk from underlying Gluster (which is already HA) to newly created VM, so you don't have to care about creating HA storage on your VMs. You also save some resources as you don't have to have backup VMs running. The drawback of this approach that re-creating new VM takes more time than switching the traffic to already running VM on load balancer. This depends on your constraints but IMHO worth to consider. Vojta On Friday, 16 April 2021 03:00:23 CEST David White via Users wrote:

I'm currently thinking about just setting up a rsync cron to run every minute.

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, April 15, 2021 8:55 PM, David White via Users <users@ovirt.org> wrote:

...

...

David, I’m curious what the use case is

This is for a customer who wants as much high availability as possible for their website, which relies on a basic LAMP or LNMP stack.

The plan is to create a MariaDB Galera cluster.

Each of the 3 VMs will run MariaDB, as well as Apache or Nginx (I haven't decided which, yet), and will be able to accept web traffic.

So the website files will need to be the same across all 3 virtual servers.

My original intent was to setup a mount point on all 3 virtual servers that mapped back to the same shared disk.

Strahil, 1 idea I had, which I don't think would be ideal at all, was to setup a separate, new, gluster configuration on each of the 3 VMs. Gluster virtualized on top of gluster! If that doesn't make your head spin, what will? But I'm not seriously thinking about that. :)

It did occur to me that I could setup a 4th VM to host the NFS share.

But I'm trying to stay away from as many single points of failures as possible.

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, April 15, 2021 7:40 PM, Strahil Nikolov via Users users@ovirt.org wrote:

...

I know that clusterizing applications (for example corosync/pacemaker Active-Passive or even GFS2) require simultaneous access to the data.

In your case you can create:

- 2 separate VMs replicating over DRBD and sharing the storage over NFS/iSCSI - Using NFS Ganesha (this is just a theory but should work) to export your Gluster volumes in a redundant and highly available way>

...

...

Best Regards, Strahil Nikolov

В петък, 16 април 2021 г., 01:56:09 ч. Гринуич+3, Jayme

jaymef@gmail.com написа:

...

David, I’m curious what the use case is. :9 you plan on using the disk with three vms at the same time? This isn’t really what shareable disks are meant to do afaik. If you want to share storage with multiple vms I’d probably just setup an nfs share on one of the vms> > On Thu, Apr 15, 2021 at 7:37 PM David White via Users users@ovirt.org wrote:

...

I found the proper documentation at https://www.ovirt.org/documentation/administration_guide/#Shareabl e_Disks. When I tried to edit the disk, I see that sharable is grayed out, and when I hover my mouse over it, I see "Sharable Storage is not supported on Gluster/Offload Domain". So to confirm, is there any circumstance where a Gluster volume can support sharable storage? Unfortunately, I don't have any other storage available, and I chose to use Gluster, so that I could have a HA environment. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, April 15, 2021 5:05 PM, David White via Users users@ovirt.org wrote:

...

I need to mount a partition across 3 different VMs. How do I attach a disk to multiple VMs? This looks like fairly old documentation-not-documentation: https://www.ovirt.org/develop/rele ase-management/features/storage/sharedrawdisk.html Sent with ProtonMail Secure Email.

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WZY6OJW BH5KAB5H2XXYJOVI7BLR4Z67F/> > Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3XAQI4WTRS PEECHZV3NEQLUMHRCEPUEZ/

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AFSDOLZCL VEO6NU36C3WOTSONMGPT4EF/> Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3J6KFZM7K2A KDZJDISSQBRVSEGIYBPIX/

Gluster in the VMs... I was thinking to propose it, but I wasn't sure what kind of workload you got. I think that with Redis and Gluster on VM - you will be quite fine. For the Galera - it doesn't need shared storage at all, so you will be quite fine. Don't forget that lattency kills gluster, so keep it as tight as possible , but in the same time keep them on separate hosts. Best Regards, Strahil Nikolov В петък, 16 април 2021 г., 03:57:51 ч. Гринуич+3, David White via Users <users@ovirt.org> написа:

David, I’m curious what the use case is This is for a customer who wants as much high availability as possible for their website, which relies on a basic LAMP or LNMP stack.

The plan is to create a MariaDB Galera cluster. Each of the 3 VMs will run MariaDB, as well as Apache or Nginx (I haven't decided which, yet), and will be able to accept web traffic. So the website files will need to be the same across all 3 virtual servers. My original intent was to setup a mount point on all 3 virtual servers that mapped back to the same shared disk. Strahil, 1 idea I had, which I don't think would be ideal at all, was to setup a separate, new, gluster configuration on each of the 3 VMs. Gluster virtualized on top of gluster! If that doesn't make your head spin, what will? But I'm not seriously thinking about that. :) It did occur to me that I could setup a 4th VM to host the NFS share. But I'm trying to stay away from as many single points of failures as possible. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, April 15, 2021 7:40 PM, Strahil Nikolov via Users <users@ovirt.org> wrote:

I know that clusterizing applications (for example corosync/pacemaker Active-Passive or even GFS2) require simultaneous access to the data.

In your case you can create:

-  2 separate VMs replicating over DRBD and sharing the storage over NFS/iSCSI -  Using NFS Ganesha (this is just a theory but should work) to export your Gluster volumes in a redundant and highly available way    

    Best Regards,     Strahil Nikolov    

    В петък, 16 април 2021 г., 01:56:09 ч. Гринуич+3, Jayme jaymef@gmail.com написа:    

David, I’m curious what the use case is. :9 you plan on using the disk with three vms at the same time?  This isn’t really what shareable disks are meant to do afaik. If you want to share storage with multiple vms I’d probably just setup an nfs share on one of the vms 

On Thu, Apr 15, 2021 at 7:37 PM David White via Users users@ovirt.org wrote:

...

I found the proper documentation at https://www.ovirt.org/documentation/administration_guide/#Shareable_Disks. When I tried to edit the disk, I see that sharable is grayed out, and when I hover my mouse over it, I see "Sharable Storage is not supported on Gluster/Offload Domain". So to confirm, is there any circumstance where a Gluster volume can support sharable storage? Unfortunately, I don't have any other storage available, and I chose to use Gluster, so that I could have a HA environment. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐   On Thursday, April 15, 2021 5:05 PM, David White via Users users@ovirt.org wrote:

...

...

I need to mount a partition across 3 different VMs. How do I attach a disk to multiple VMs? This looks like fairly old documentation-not-documentation: https://www.ovirt.org/develop/release-management/features/storage/sharedrawd... Sent with ProtonMail Secure Email.

...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WZY6OJWBH5KAB5...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3XAQI4WTRSPEEC...

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AFSDOLZCLVEO6N...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3J6KFZM7K2AKDZ...

You're welcome to help with oVirt project design and discuss with the community the parts that you think should benefit from a re-design.

I consider these pesky little comments part of the discussion, even if I know they are not the best style. But how much is there to discuss, if Redhat has already decided to switch to a beta base (CentOS stream) underneath oVirt? Nobody wants bleeding edge on a hypervisor, except those who develop that hypervisor. oVirt is supposed to deliver a higher reliability than bare metal hardware, by providing a fault tolerant design and automatic fault recovery. But if the software stack that the HA engine builds on is more volatile than the hardware below, it simply can't do its work of increasing overall resilience: a beta OS kills the value of a HA management stack above. Only a RHEL downstream CentOS is near solid enough to build on (unless you did fully validated oVirt node images). Bleeding edge is what you put inside the VMs, not underneath. I don't think I have heard a single oVirt *user* advocating the switch to Stream. IMHO it's political and kills oVirt's value proposition. My next major other gripe is that to a newcomer it's not obvious from the start that the oVirt 'classic' and the HCI variant are and will most likely remain very different beasts, because they have a distinct history and essentially incompatible principles. Classic oVirt started with shared storage, which is always turned on. Theat means idle hosts can be turned off and workloads consolidated to minimize energy consumption. It aims for the minimal number of hosts to do the job. Gluster is all about scale out without any choke points, the more hosts the better the performance. And when you combine both in a HCI gluster, turning off hosts requires a much better attention as to whether these hosts contribute bricks to volumes in use or not. For a user it's quite natural to mix both, using a set of HCI nodes to provide storage and base capacity and then add pure compute nodes to provide dynamic workload expansion. But now I'm pretty sure that's unchartered territory, because I see terrible things happening with quota decisions when computing nodes that don't even contribute bricks to volumes are rebooted e.g. during updates. Making the community responsible for providing a unifying vision, is asking for help a bit too late in the game. And then classic oVirt and HCI oVirt have completely different scaling characteristics. Classic will scale from one to N hosts without any issue, but HCI won't even go from 1 to 2 or the more sensible 3 nodes. Nor does it then allow the transition from replicas to the obviously more attractive dispersion volumes as you expand from 3 to 6 or 9 nodes. How much of a discussion will we have, when I say that I want a Gluster volume to expand/grow/shrink and transform from 1 to N bricks and transition between replicas, dispersed volumes, sharded or non sharded with oVirt seamlessly running on top? But unfortunately that is the natural expectation any newcomer will have, just like I did, when I read all the nice things Redhat had to say about the technology. I hope you won't dispute it's still very much a patchwork and with a very small chance of near time resolution.

Thank you Gianluca, for supporting my claim: it's patchwork and not "a solution designed for the entire enterprise". Instead it's more of "a set of assets where two major combinations from a myriad of potential permutations have received a bit of testing and might be useful somewhere in your enterprise". As such, I see very little future for oVirt as anything that doesn't achieve scale these days is doomed to die. I gather IBM is betting on RH in the cloud, but oVirt isn't designed for that (and suffers a license overhead for little if any extra value over the cloud native stack) and HCI doesn't make sense in any existing cloud: its mission is more like bootstrapping your own. Once you achieve any scale, storage will move to specialized appliances. I can see oVirt and especially the HCI variant on the potentially many stopovers from the edge to the cloud core and even in special data center holdouts. There the ability to really deliver the best fault resilient 1-9 node scalability (somewhat bigger shouldn't be a problem anyway) with the ability to carefully tune and mingle between resilience and storage efficiency is key. You don't want to redeploy oVirt in a 100 embedded locations across a bigger geography, just because you've outgrown 3 nodes. I could see oVirt run on ships, (including space ships), in factories, in the military, train networks, schools or just about any place that need to combine some local presence with resilience, flexibility and remote management (but low dependence). But you'd have to go at it strategically and with a truly unified approach between the Gluster and oVirt teams. The management engine, KVM, VDO and Gluster are each brilliant pieces of engineering, the combination of which could be a force to reckon with, everywhere outside the cloud. But not with the current approach, where each component is allowed to trudge along at its own pace, hopefully not breaking as each is evolving independently. And of course, the final product must be available free of charge so money doesn't get in the way of scale. When a nation adopts oVirt to digitalize its school, or its rail systems, or an industry giant to runs its factories, revenue should not be an issue. And at the low-end you really want to beat QNAP with a 3 node HCI at a similar cost and energy footprint e.g. using RASPI modules (or just three last generation smartphones for that matter). That's how you'd get the scale. I hope you'll find something valuable in all this rant! And sorry for the bother.

Thank you Gianluca for your honest assessment. Now if only you'd put that on the home page of oVirt, or better yet, used the opportunity to change things. Yes, after what I know today, I should not have started with oVirt on Gluster, but unfortunately HCI is exactly the most attractive grassroots use case and the biggest growth opportunity for oVirt. AFAIK there is no competition out there. If there was, I probably would have jumped ship already, as attached (habit dependent) as I am to CentOS. In theory oVirt could start with single node HCI, and then go to 3nHCI to gain resilience. That step is already crucial (ideally witha 2nHCI base on a warm storage standby) and currently easy with Gluster, but "not supported" (with no warning or explanation that I could remember) by oVirt. Whatever the reason, it is not at all obvious to the newcomer, nor should it be unsurmountable to my understanding. And as things go on and you're expanding some clusters, there is a natural tendency to go towards dedicated storage say after reaching dual digits on HCI nodes, because it offer better price/performance and controls. Again, that transition should be "organic" of not seamless, with migration of VMs and their disks, perhaps not live, even if gaps there should be closable, too. Nobody else can do that, not VMware, not Nutanix, nor Proxmox. And the cloud guys aren't offering anything right there, even if they probably should, if only to make sure that any such grass-roots projects can be fork lifted easily into their clouds, once their end product takes off. IMHO this sort of opportunity needs serious seed money, organic growth from the community and direct revenues obviously haven't done it yet. But letting things just go on as they do now, I consider a death knell to oVirt. Kind regards, Thomas

On Sat, Apr 24, 2021 at 3:31 PM David White via Users <users@ovirt.org> wrote:

Off topic, but something to address: We need a stable ovirt-guest-agent package. This doesn't seem to be working for me, although I'll take a look at it more closely again when I have some time: https://launchpad.net/ubuntu/focal/+source/ovirt-guest-agent

ovirt-guest-agent is deprecated in 4.4. See also here the downstream documentation: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/htm... Gianluca

Il giorno mer 21 apr 2021 alle ore 11:02 Thomas Hoberg <thomas@hoberg.net> ha scritto:

...

You're welcome to help with oVirt project design and discuss with the community the parts that you think should benefit from a re-design.

I consider these pesky little comments part of the discussion, even if I know they are not the best style.

But how much is there to discuss, if Redhat has already decided to switch to a beta base (CentOS stream) underneath oVirt?

Nobody wants bleeding edge on a hypervisor, except those who develop that hypervisor.

I can understand the position here, but the fact that oVirt is developed and stabilized against CentOS Stream which is upstream to RHEL doesn't prevent you to run oVirt on RHEL or any other RHEL rebuild on production. If you face any issue running oVirt on top of a downstream to CentOS Stream please report a bug for it and we'll be happy to handle.

oVirt is supposed to deliver a higher reliability than bare metal hardware, by providing a fault tolerant design and automatic fault recovery.

But if the software stack that the HA engine builds on is more volatile than the hardware below, it simply can't do its work of increasing overall resilience: a beta OS kills the value of a HA management stack above.

Only a RHEL downstream CentOS is near solid enough to build on (unless you did fully validated oVirt node images). Bleeding edge is what you put inside the VMs, not underneath. I don't think I have heard a single oVirt *user* advocating the switch to Stream. IMHO it's political and kills oVirt's value proposition.

And I understand the point of view on this and I've nothing against this. CentOS Linux 8 is going to reach EOL in a few months and oVirt project can't keep using it for development. CentOS Stream is upstream to whatever CentOS Stream derivative (including RHEL) oVirt users are going to use in production so we need to ensure oVirt will run on what's coming next in order to avoid oVirt users to get broken systems once an update will come to production. So from oVirt development perspective CentOS Stream right now is the only choice. That said, really, on production you are not required to use CentOS Stream if you don't want to.

My next major other gripe is that to a newcomer it's not obvious from the start that the oVirt 'classic' and the HCI variant are and will most likely remain very different beasts, because they have a distinct history and essentially incompatible principles.

Classic oVirt started with shared storage, which is always turned on. Theat means idle hosts can be turned off and workloads consolidated to minimize energy consumption. It aims for the minimal number of hosts to do the job.

Gluster is all about scale out without any choke points, the more hosts the better the performance.

And when you combine both in a HCI gluster, turning off hosts requires a much better attention as to whether these hosts contribute bricks to volumes in use or not.

For a user it's quite natural to mix both, using a set of HCI nodes to provide storage and base capacity and then add pure compute nodes to provide dynamic workload expansion.

But now I'm pretty sure that's unchartered territory, because I see terrible things happening with quota decisions when computing nodes that don't even contribute bricks to volumes are rebooted e.g. during updates.

And this can be made more clear either on the download page or in the installation guide documentation for new comers. Let's do it. I can open a PR on https://github.com/oVirt/ovirt-site or you can do it and we can work together on ensuring this will be clear for new comers.

Making the community responsible for providing a unifying vision, is asking for help a bit too late in the game.

And then classic oVirt and HCI oVirt have completely different scaling characteristics. Classic will scale from one to N hosts without any issue, but HCI won't even go from 1 to 2 or the more sensible 3 nodes. Nor does it then allow the transition from replicas to the obviously more attractive dispersion volumes as you expand from 3 to 6 or 9 nodes.

How much of a discussion will we have, when I say that I want a Gluster volume to expand/grow/shrink and transform from 1 to N bricks and transition between replicas, dispersed volumes, sharded or non sharded with oVirt seamlessly running on top?

I'll let Gluster team discuss this, I lack the needed knowledge to give meaningful replies.

But unfortunately that is the natural expectation any newcomer will have, just like I did, when I read all the nice things Redhat had to say about the technology.

I hope you won't dispute it's still very much a patchwork and with a very small chance of near time resolution. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2CIGCLM5E2CTA2...

-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

Hi Strahil, I've tried to measure the cost or of erasure coding and, more importantly, VDO with de-duplication and compression a bit. Erasure coding should be neglible in terms of CPU power while the vastly more complex LZ4 compression (used inside VDO) really is rather impressive at 1GByte/s single threaded for compression (6Gbyte/s decompression, on a 25GByte/s memory bus) on the 15Watt NUCs I am using for one cluster. The storage I/O overhead of erasure coding shouldn't really matter with NVMe becoming cheaper than SATA SSD. Perhaps the write amplification needs to be watched with SSDs, but a lot of that is writeback tuning and with a Gluster in the back, you can commit to RAM as long as you have a quorum (and a UPS). Actually with Gluster I guess most of the erasure coding would actually be done by the client and the network amplification would also be there, but not really different between erasure coding and replicas: If you write to nine nodes, you write to nine nodes from the client independent of the encoding. There the ability to say "please continue to use the 4:2 dispersion as I expand from 6 to 9 nodes and roll that across on a shard by shard base without me having to set up bricks like that", would certainly help. With all of VDO enabled I get 200MByte/s for a random data workload on FIO via Gluster, which becomes 600MByte/s for reads with 3 replicas on the 10Gbit network I use, 60% of the theoretical maximum with random I/O. That's completely adequate, because we're not running HPC or SAP batches here and I'd be rather sure that using erasure coding with 6 and 9 nodes won't introduce a performance bottleneck, unless I go to 40 or 100GBit on the network. I'd just really want to be able to choose between say 1, 2 or 3 out of 9 bricks being used for redundancy, depending on if it's an HCI block next door, going into a ship with months at sea or into a space station. I'd also probably add an extra node or two to act as warm (even cold) standby in critical or hard-to-reach locations, that act as compute-only nodes initially (to avoid split quotas), but can be promoted to replace a storage node that failed without hands-on intervention. oVirt HCI is as close at it gets to LEGO computers, but right now it's doing LEGO with your hands tied behind your back. Kind regards, Thomas

and you expect newcomers to find that significant bit of information within the reference that you quote as they try to evaluate if oVirt is the right tool for the job? I only found out once I tried to add dispersed volumes to an existing 3 node HCI and dug through the log files. Of course, I eventually managed to remove the nicely commented bits of ansible code that prevented adding the volume, only to find that it could not be used to run VMs from there or use it for disks. I can still mount those volumes from inside the VMs via a GlusterFS client and I'd guess that there is little if any difference in performance. For an enterprise HCI solution, the usable intersection between oVirt and Gluster is so small, it needs a magnifying glass very top and early in the documentation. Gluster advertises itself as a scale-out file system without any metadata choke point as the main differentiator vs. Lustre etc. and with tunable ratio of read amplification via replicas and resilience. Nobody expects scale-out to mean 1 or 3. With perhaps 6 and 9 as a special option. Or only replicas actually supported by oVirt, when erasure coding should at least in theory give you near perfect scalability, which means you can add increments of one or any bigger number and freely allocated between capacity and resilience. It's perfectly legitimate to not support every potential permutation of deployment scenarios of oVirt and Gluster. But the limitations baked in and perhaps even the motivation need to be explained from the very start. It doesn't help oVirt's adoption and success if people only find out after they have invested heavily under the assumption a "scale-out" solution delivers what that term implies.