No. I gave up on ovirt 4.2 (node iso and centos) and moved over to
the
node 4.3 iso, and that's where I found the problem with the JBOD. I then
moved back to centos and installed 4.3 there, which comes with
gdeploy-2.0.8-1.el7.noarch
. I don't know if this version had the issue or not, as I assumed that
given I'm using the same ovirt/gluster repos, it'd probably have had the
same problem.
I've since wiped out the cluster and have been testing a single ovirt-4.3
node (from centos) to try to rule out a few issues. Found that for some
reason, while my bare metal raid array gives me about 1GBps r/w (XFS on top
of LVM), as soon as I mount a brick and try to r/w to it, that performance
drops to between 5 and 55MBps depending on block size. The native XFS and
gluster XFS configs are identical. 1000MBps o the native XFS with 4k
blocks, and 5MBps on gluster with 4k blocks. I can get up to 55MBps on
gluster with 64k blocks. So something else isn't right, but I haven't had
any luck in asking around, as everyone seems to feel this is just an issue
of incrimental performance tweaks. I of course argue that all of the
recommended tweaks only gained 1-2% performance, and that we'd need about
800 of them to get decent performance.
So at this point I'm not really sure what to do next. When I've used
gluster in the past (configured manually), I've never had a problem getting
at least 80-90% native performance. I've also noticed that for some reason,
ovirt runs my guests like complete garbage and I have no idea why. I get
about 1/20th to 1/50th the performance from ovirt, as I get on the same
hardware with boring qemu/kvm.
I'm just bringing up a single host (centos with 4.3) now, and trying local
storage instead of gluster, to see if there's any major change in overall
VM performance. Just uploading an ISO to the local storage though, is
already about 80x faster than uploading it to a gluster volume.
And the gluster volume options are the standard? Set by cockpit deployment?
The gluster volume created via cockpit uses network.remote-dio: disable and
performance.strict-o-direct:on to ensure that every write is written to
disk. This is to ensure consistency and prevent loss of data
You can turn on client side brick caching by using network.remote-dio:
enable - but be sure to have a power backup to ensure yours hosts do not
power off and lose data before the cache is flushed
And... somewhat confirmed. Using local storage (just a plain XFS
file
system) showed major improvement to guest VM performance overall. The disc
throughput is still very slow (160MBps vs the 1000MBps of the host), but
overall, the entire VM is much more responsive.
Meanwhile, testing a clone of the same VM on my 7 year old laptop, which
pushes around 300MBps, the VM is pushing 280MBps average. Both using XFS.
So why is ovirt's guest disc performance (native and gluster) so poor? Why
is it consistently giving me about 1/10th to 1/80th of the hosts disc
throughput?
On Mon, Feb 11, 2019 at 5:01 AM Sahina Bose <sabose(a)redhat.com> wrote:
>
>
> On Wed, Feb 6, 2019 at 10:45 PM feral <blistovmhz(a)gmail.com> wrote:
>
>> On that note, this was already reported several times a few months back,
>> but apparently was fixed in gdeploy-2.0.2-29.el7rhgs.noarch. I'm
>> guessing ovirt-node-4.3 just hasn't updated to that version yet?
>>
>
> +Niels de Vos <ndevos(a)redhat.com> +Sachidananda URS <surs(a)redhat.com>
> Any update on the gdeploy package update in CentOS ?
>
>
>> On Wed, Feb 6, 2019 at 8:51 AM feral <blistovmhz(a)gmail.com> wrote:
>>
>>> Bugzilla must have lost my account. Requesting password reset on an
>>> account that doesn't exist, results in 500 :p.
>>> Created a new one.
>>>
>>> However, my options for gdeploy version do not match the version 2.0.2
>>> installed. They only list 0.xx.x.
>>>
>>> On Wed, Feb 6, 2019 at 8:44 AM Greg Sheremeta <gshereme(a)redhat.com>
>>> wrote:
>>>
>>>>
>>>> On Wed, Feb 6, 2019 at 11:31 AM feral <blistovmhz(a)gmail.com>
wrote:
>>>>
>>>>> Sorry, I tried opening a bug, but RH's Bugzilla is having issues
and
>>>>> 500's out when I try to request password reset.
>>>>>
>>>>
>>>> Sorry to hear that. I hate to ask, but could you try after clearing
>>>> cookies for *redhat.com, or try in an incognito tab? It's done that
>>>> to me before and clearing cookies worked for me. If it still doesn't
work,
>>>> I'll open the bug. Thanks!
>>>>
>>>>
>>>>>
>>>>>
>>>>>
-----------------------------------------------------------------------------------
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Run a shell script]
>>>>> ******************************************************
>>>>> changed: [ovirt-431.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h
>>>>> ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Run a shell script]
>>>>> ******************************************************
>>>>> changed: [ovirt-432.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h
>>>>> ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Run a shell script]
>>>>> ******************************************************
>>>>> changed: [ovirt-433.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d vdb -h
>>>>> ovirt-431.localdomain, ovirt-432.localdomain, ovirt-433.localdomain)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create VDO with specified size]
>>>>> ******************************************
>>>>> changed: [ovirt-431.localdomain] => (item={u'disk':
u'/dev/vdb',
>>>>> u'logicalsize': u'11000G', u'name':
u'vdo_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create VDO with specified size]
>>>>> ******************************************
>>>>> changed: [ovirt-432.localdomain] => (item={u'disk':
u'/dev/vdb',
>>>>> u'logicalsize': u'11000G', u'name':
u'vdo_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create VDO with specified size]
>>>>> ******************************************
>>>>> changed: [ovirt-433.localdomain] => (item={u'disk':
u'/dev/vdb',
>>>>> u'logicalsize': u'11000G', u'name':
u'vdo_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Enable or disable services]
>>>>> **********************************************
>>>>> ok: [ovirt-432.localdomain] => (item=chronyd)
>>>>> ok: [ovirt-431.localdomain] => (item=chronyd)
>>>>> ok: [ovirt-433.localdomain] => (item=chronyd)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=0 unreachable=0
>>>>> failed=0
>>>>> ovirt-432.localdomain : ok=1 changed=0 unreachable=0
>>>>> failed=0
>>>>> ovirt-433.localdomain : ok=1 changed=0 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [start/stop/restart/reload services]
>>>>> **************************************
>>>>> changed: [ovirt-432.localdomain] => (item=chronyd)
>>>>> changed: [ovirt-431.localdomain] => (item=chronyd)
>>>>> changed: [ovirt-433.localdomain] => (item=chronyd)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Run a command in the shell]
>>>>> **********************************************
>>>>> changed: [ovirt-431.localdomain] => (item=vdsm-tool configure
--force)
>>>>> changed: [ovirt-432.localdomain] => (item=vdsm-tool configure
--force)
>>>>> changed: [ovirt-433.localdomain] => (item=vdsm-tool configure
--force)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Run a shell script]
>>>>> ******************************************************
>>>>> changed: [ovirt-431.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)
>>>>> changed: [ovirt-432.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)
>>>>> changed: [ovirt-433.localdomain] =>
>>>>> (item=/usr/share/gdeploy/scripts/blacklist_all_disks.sh)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Clean up filesystem signature]
>>>>> *******************************************
>>>>> skipping: [ovirt-431.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> TASK [Create Physical Volume]
>>>>> **************************************************
>>>>> changed: [ovirt-431.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Clean up filesystem signature]
>>>>> *******************************************
>>>>> skipping: [ovirt-432.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> TASK [Create Physical Volume]
>>>>> **************************************************
>>>>> changed: [ovirt-432.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Clean up filesystem signature]
>>>>> *******************************************
>>>>> skipping: [ovirt-433.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> TASK [Create Physical Volume]
>>>>> **************************************************
>>>>> changed: [ovirt-433.localdomain] => (item=/dev/mapper/vdo_vdb)
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create volume group on the disks]
>>>>> ****************************************
>>>>> changed: [ovirt-431.localdomain] => (item={u'brick':
>>>>> u'/dev/mapper/vdo_vdb', u'vg':
u'gluster_vg_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-431.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create volume group on the disks]
>>>>> ****************************************
>>>>> changed: [ovirt-432.localdomain] => (item={u'brick':
>>>>> u'/dev/mapper/vdo_vdb', u'vg':
u'gluster_vg_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-432.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>>
>>>>> PLAY [gluster_servers]
>>>>> *********************************************************
>>>>>
>>>>> TASK [Create volume group on the disks]
>>>>> ****************************************
>>>>> changed: [ovirt-433.localdomain] => (item={u'brick':
>>>>> u'/dev/mapper/vdo_vdb', u'vg':
u'gluster_vg_vdb'})
>>>>>
>>>>> PLAY RECAP
>>>>>
*********************************************************************
>>>>> ovirt-433.localdomain : ok=1 changed=1 unreachable=0
>>>>> failed=0
>>>>>
>>>>> Error: Section diskcount not found in the configuration file
>>>>>
>>>>>
-----------------------------------------------------------------------------------
>>>>>
>>>>> So this is just using the default gluster options, except with JBOD
>>>>> selected.
>>>>>
>>>>
>>>> Sounds like a valid bug to me.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 6, 2019 at 8:15 AM Greg Sheremeta
<gshereme(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Please open a bug on this.
>>>>>>
https://bugzilla.redhat.com/enter_bug.cgi?product=cockpit-ovirt
>>>>>> Component = Gdeploy
>>>>>>
>>>>>> Best wishes,
>>>>>> Greg
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 5, 2019 at 3:20 PM feral <blistovmhz(a)gmail.com>
wrote:
>>>>>>
>>>>>>> Vanilla install of ovirt-node-4.3 (iso). During
hyperconverged
>>>>>>> wizard, there is no option for disc count for JBOD. This
results in error
>>>>>>> during deployment.
>>>>>>> Disc Count option is available for all other raid levels.
>>>>>>>
>>>>>>> --
>>>>>>> _____
>>>>>>> Fact:
>>>>>>> 1. Ninjas are mammals.
>>>>>>> 2. Ninjas fight ALL the time.
>>>>>>> 3. The purpose of the ninja is to flip out and kill people.
>>>>>>> _______________________________________________
>>>>>>> Users mailing list -- users(a)ovirt.org
>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>>>> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>>>>>>> oVirt Code of Conduct:
>>>>>>>
https://www.ovirt.org/community/about/community-guidelines/
>>>>>>> List Archives:
>>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7JLLJJM4D2R...
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> GREG SHEREMETA
>>>>>>
>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>>>>>>
>>>>>> Red Hat NA
>>>>>>
>>>>>> <
https://www.redhat.com/>
>>>>>>
>>>>>> gshereme(a)redhat.com IRC: gshereme
>>>>>> <
https://red.ht/sig>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> _____
>>>>> Fact:
>>>>> 1. Ninjas are mammals.
>>>>> 2. Ninjas fight ALL the time.
>>>>> 3. The purpose of the ninja is to flip out and kill people.
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> GREG SHEREMETA
>>>>
>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>>>>
>>>> Red Hat NA
>>>>
>>>> <
https://www.redhat.com/>
>>>>
>>>> gshereme(a)redhat.com IRC: gshereme
>>>> <
https://red.ht/sig>
>>>>
>>>
>>>
>>> --
>>> _____
>>> Fact:
>>> 1. Ninjas are mammals.
>>> 2. Ninjas fight ALL the time.
>>> 3. The purpose of the ninja is to flip out and kill people.
>>>
>>
>>
>> --
>> _____
>> Fact:
>> 1. Ninjas are mammals.
>> 2. Ninjas fight ALL the time.
>> 3. The purpose of the ninja is to flip out and kill people.
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>>
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GA5TBUK75EJ...
>>
>
--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.