ngn build jobs take more than twice (x) as long as in the last days

[Evgheni Dereveanchin]

Hi,

I checked SAR data on storage servers and compared loads
yesterday and three weeks ago (May 3). Load values are in
pretty much the same range, yet now most of the time they
are around the "high" mark so we may be nearing a bottleneck,
specifically on I/O where we mostly do writes to the NAS,
not reads and there's quite a bit of overhead:

VM -> QCOW -> file -> network -> NFS -> DRBD -> disk

Surely, using local scratch disks stored on SSDs should greatly
improve performance as at least half of the above steps will be gone.
We don't really need to centrally store (NFS) or mirror (DRBD) 
data that slaves write to their disks all the time anyways.

For VMs where we do need redundancy, I'd suggest using 
iSCSI storage domains in the long run.

Regards, 
Evgheni Dereveanchin 

----- Original Message -----
From: "Eyal Edri" <eedri at redhat.com>
To: "Sandro Bonazzola" <sbonazzo at redhat.com>, "Evgheni Dereveanchin" <ederevea at redhat.com>, "Anton Marchukov" <amarchuk at redhat.com>
Cc: "Fabian Deutsch" <fdeutsch at redhat.com>, "infra" <infra at ovirt.org>
Sent: Wednesday, 25 May, 2016 9:31:43 AM
Subject: Re: ngn build jobs take more than twice (x) as long as in the last days

It might be more load on the storage servers with now running much more
jobs.
Evgheni - can you check if the load on the storage servers has changed
significantly to justify this degradation of service?

We need to expedite the enablement of SSDs in the hypervisors and move to
local hooks.
Anton - do we have a test VM that uses a local DISK we can use to test if
it improves the runtime?

On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo at redhat.com>
wrote:

>
> Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch at redhat.com> ha scritto:
> >
> > Hey,
> >
> > $subj says it all.
> >
> > Affected jobs are:
> > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
> >
> > I.e. 3.6 - before: ~46min, now 1:23hrs
> >
> > In master it's even worse: >1:30hrs
> >
> > Can someone help to idnetify the reason?
>
> I have no numbers but I have the feeling that all jobs are getting slower
> since a couple of weeks ago. Yum install phase takes ages. I thoughtit was
> some temporary storage i/o peak but looks like it's not temporary.
>
> >
> > - fabian
> >
> > --
> > Fabian Deutsch <fdeutsch at redhat.com>
> > RHEV Hypervisor
> > Red Hat
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>


-- 
Eyal Edri
Associate Manager
RHEV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)