On Mon, Feb 11, 2019 at 9:57 PM feral <blistovmhz@gmail.com> wrote:
No. I gave up on ovirt 4.2 (node iso and centos) and moved over to the node 4.3 iso, and that's where I found the problem with the JBOD. I then moved back to centos and installed 4.3 there, which comes with gdeploy-2.0.8-1.el7.noarch . I don't know if this version had the issue or not, as I assumed that given I'm using the same ovirt/gluster repos, it'd probably have had the same problem.

I've since wiped out the cluster and have been testing a single ovirt-4.3 node (from centos) to try to rule out a few issues. Found that for some reason, while my bare metal raid array gives me about 1GBps r/w (XFS on top of LVM), as soon as I mount a brick and try to r/w to it, that performance drops to between 5 and 55MBps depending on block size. The native XFS and gluster XFS configs are identical. 1000MBps o the native XFS with 4k blocks, and 5MBps on gluster with 4k blocks. I can get up to 55MBps on gluster with 64k blocks. So something else isn't right, but I haven't had any luck in asking around, as everyone seems to feel this is just an issue of incrimental performance tweaks. I of course argue that all of the recommended tweaks only gained 1-2% performance, and that we'd need about 800 of them to get decent performance.

So at this point I'm not really sure what to do next. When I've used gluster in the past (configured manually), I've never had a problem getting at least 80-90% native performance. I've also noticed that for some reason, ovirt runs my guests like complete garbage and I have no idea why. I get about 1/20th to 1/50th the performance from ovirt, as I get on the same hardware with boring qemu/kvm. 
I'm just bringing up a single host (centos with 4.3) now, and trying local storage instead of gluster, to see if there's any major change in overall VM performance. Just uploading an ISO to the local storage though, is already about 80x faster than uploading it to a gluster volume.

And the gluster volume options are the standard? Set by cockpit deployment?

The gluster volume created via cockpit uses network.remote-dio: disable and performance.strict-o-direct:on to ensure that every write is written to disk. This is to ensure consistency and prevent loss of data
You can turn on client side brick caching by using network.remote-dio: enable - but be sure to have a power backup to ensure yours hosts do not power off and lose data before the cache is flushed


And... somewhat confirmed. Using local storage (just a plain XFS file system) showed major improvement to guest VM performance overall. The disc throughput is still very slow (160MBps vs the 1000MBps of the host), but overall, the entire VM is much more responsive.
Meanwhile, testing a clone of the same VM on my 7 year old laptop, which pushes around 300MBps, the VM is pushing 280MBps average. Both using XFS.

So why is ovirt's guest disc performance (native and gluster) so poor? Why is it consistently giving me about 1/10th to 1/80th of the hosts disc throughput?




On Mon, Feb 11, 2019 at 5:01 AM Sahina Bose <sabose@redhat.com> wrote:


On Wed, Feb 6, 2019 at 10:45 PM feral <blistovmhz@gmail.com> wrote:
On that note, this was already reported several times a few months back, but apparently was fixed in gdeploy-2.0.2-29.el7rhgs.noarch. I'm guessing ovirt-node-4.3 just hasn't updated to that version yet?

Any update on the gdeploy package update in CentOS ?


On Wed, Feb 6, 2019 at 8:51 AM feral <blistovmhz@gmail.com> wrote:
Bugzilla must have lost my account. Requesting password reset on an account that doesn't exist, results in 500 :p.
Created a new one.

However, my options for gdeploy version do not match the version 2.0.2 installed. They only list 0.xx.x. 

On Wed, Feb 6, 2019 at 8:44 AM Greg Sheremeta <gshereme@redhat.com> wrote:

On Wed, Feb 6, 2019 at 11:31 AM feral <blistovmhz@gmail.com> wrote:
Sorry, I tried opening a bug, but RH's Bugzilla is having issues and 500's out when I try to request password reset.

Sorry to hear that. I hate to ask, but could you try after clearing cookies for *redhat.com, or try in an incognito tab? It's done that to me before and clearing cookies worked for me. If it still doesn't work, I'll open the bug. Thanks! 
 

-----------------------------------------------------------------------------------
PLAY [gluster_servers] *********************************************************

TASK [Run a shell script] ******************************************************
changed: [ovirt-431.localdomain] => (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Run a shell script] ******************************************************
changed: [ovirt-432.localdomain] => (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)

PLAY RECAP *********************************************************************
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Run a shell script] ******************************************************
changed: [ovirt-433.localdomain] => (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)

PLAY RECAP *********************************************************************
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create VDO with specified size] ******************************************
changed: [ovirt-431.localdomain] => (item={u'disk': u'/dev/vdb', u'logicalsize': u'11000G', u'name': u'vdo_vdb'})

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create VDO with specified size] ******************************************
changed: [ovirt-432.localdomain] => (item={u'disk': u'/dev/vdb', u'logicalsize': u'11000G', u'name': u'vdo_vdb'})

PLAY RECAP *********************************************************************
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create VDO with specified size] ******************************************
changed: [ovirt-433.localdomain] => (item={u'disk': u'/dev/vdb', u'logicalsize': u'11000G', u'name': u'vdo_vdb'})

PLAY RECAP *********************************************************************
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Enable or disable services] **********************************************
ok: [ovirt-432.localdomain] => (item=chronyd)
ok: [ovirt-431.localdomain] => (item=chronyd)
ok: [ovirt-433.localdomain] => (item=chronyd)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=0    unreachable=0    failed=0   
ovirt-432.localdomain      : ok=1    changed=0    unreachable=0    failed=0   
ovirt-433.localdomain      : ok=1    changed=0    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [start/stop/restart/reload services] **************************************
changed: [ovirt-432.localdomain] => (item=chronyd)
changed: [ovirt-431.localdomain] => (item=chronyd)
changed: [ovirt-433.localdomain] => (item=chronyd)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Run a command in the shell] **********************************************
changed: [ovirt-431.localdomain] => (item=vdsm-tool configure --force)
changed: [ovirt-432.localdomain] => (item=vdsm-tool configure --force)
changed: [ovirt-433.localdomain] => (item=vdsm-tool configure --force)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Run a shell script] ******************************************************
changed: [ovirt-431.localdomain] => (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)
changed: [ovirt-432.localdomain] => (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)
changed: [ovirt-433.localdomain] => (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Clean up filesystem signature] *******************************************
skipping: [ovirt-431.localdomain] => (item=/dev/mapper/vdo_vdb) 

TASK [Create Physical Volume] **************************************************
changed: [ovirt-431.localdomain] => (item=/dev/mapper/vdo_vdb)

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Clean up filesystem signature] *******************************************
skipping: [ovirt-432.localdomain] => (item=/dev/mapper/vdo_vdb) 

TASK [Create Physical Volume] **************************************************
changed: [ovirt-432.localdomain] => (item=/dev/mapper/vdo_vdb)

PLAY RECAP *********************************************************************
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Clean up filesystem signature] *******************************************
skipping: [ovirt-433.localdomain] => (item=/dev/mapper/vdo_vdb) 

TASK [Create Physical Volume] **************************************************
changed: [ovirt-433.localdomain] => (item=/dev/mapper/vdo_vdb)

PLAY RECAP *********************************************************************
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create volume group on the disks] ****************************************
changed: [ovirt-431.localdomain] => (item={u'brick': u'/dev/mapper/vdo_vdb', u'vg': u'gluster_vg_vdb'})

PLAY RECAP *********************************************************************
ovirt-431.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create volume group on the disks] ****************************************
changed: [ovirt-432.localdomain] => (item={u'brick': u'/dev/mapper/vdo_vdb', u'vg': u'gluster_vg_vdb'})

PLAY RECAP *********************************************************************
ovirt-432.localdomain      : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *********************************************************

TASK [Create volume group on the disks] ****************************************
changed: [ovirt-433.localdomain] => (item={u'brick': u'/dev/mapper/vdo_vdb', u'vg': u'gluster_vg_vdb'})

PLAY RECAP *********************************************************************
ovirt-433.localdomain      : ok=1    changed=1    unreachable=0    failed=0   

Error: Section diskcount not found in the configuration file
-----------------------------------------------------------------------------------

So this is just using the default gluster options, except with JBOD selected.

Sounds like a valid bug to me.
 




On Wed, Feb 6, 2019 at 8:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
Please open a bug on this.
Component = Gdeploy

Best wishes,
Greg


On Tue, Feb 5, 2019 at 3:20 PM feral <blistovmhz@gmail.com> wrote:
Vanilla install of ovirt-node-4.3 (iso). During hyperconverged wizard, there is no option for disc count for JBOD. This results in error during deployment.
Disc Count option is available for all other raid levels.

--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7JLLJJM4D2RCOFQLKLW5AVJPUK24J3YY/


--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

gshereme@redhat.com    IRC: gshereme



--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.


--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

gshereme@redhat.com    IRC: gshereme



--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.


--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GA5TBUK75EJZLVIG24QK7A53VCZWVTBS/


--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.