Hi,

I checked SAR data on storage servers and compared loads
yesterday and three weeks ago (May 3). Load values are in
pretty much the same range, yet now most of the time they
are around the "high" mark so we may be nearing a bottleneck,
specifically on I/O where we mostly do writes to the NAS,
not reads and there's quite a bit of overhead:

VM -> QCOW -> file -> network -> NFS -> DRBD -> disk

Surely, using local scratch disks stored on SSDs should greatly
improve performance as at least half of the above steps will be gone.
We don't really need to centrally store (NFS) or mirror (DRBD)
data that slaves write to their disks all the time anyways.

For VMs where we do need redundancy, I'd suggest using
iSCSI storage domains in the long run.

Regards,
Evgheni Dereveanchin

----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com>
To: "Sandro Bonazzola" <sbonazzo@redhat.com>, "Evgheni Dereveanchin" <ederevea@redhat.com>, "Anton Marchukov" <amarchuk@redhat.com>
Cc: "Fabian Deutsch" <fdeutsch@redhat.com>, "infra" <infra@ovirt.org>
Sent: Wednesday, 25 May, 2016 9:31:43 AM
Subject: Re: ngn build jobs take more than twice (x) as long as in the last days

It might be more load on the storage servers with now running much more
jobs.
Evgheni - can you check if the load on the storage servers has changed
significantly to justify this degradation of service?

We need to expedite the enablement of SSDs in the hypervisors and move to
local hooks.
Anton - do we have a test VM that uses a local DISK we can use to test if
it improves the runtime?

On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo@redhat.com>
wrote:

>
> Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
> >
> > Hey,
> >
> > $subj says it all.
> >
> > Affected jobs are:
> > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
> >
> > I.e. 3.6 - before: ~46min, now 1:23hrs
> >
> > In master it's even worse: >1:30hrs
> >
> > Can someone help to idnetify the reason?
>
> I have no numbers but I have the feeling that all jobs are getting slower
> since a couple of weeks ago. Yum install phase takes ages. I thoughtit was
> some temporary storage i/o peak but looks like it's not temporary.
>
> >
> > - fabian
> >
> > --
> > Fabian Deutsch <fdeutsch@redhat.com>
> > RHEV Hypervisor
> > Red Hat
> > _______________________________________________
> > Infra mailing list
> > Infra@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
>
> _______________________________________________
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>


--
Eyal Edri
Associate Manager
RHEV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)