ngn build jobs take more than twice (x) as long as in the last days

[Eyal Edri]

OK,
I suggest to test using a VM with local disk (preferably on a host with SSD
configured), if its working,
lets expedite moving all VMs or at least a large amount of VMs to it until
we see network load reduced.

e.

On Wed, May 25, 2016 at 12:38 PM, Evgheni Dereveanchin <ederevea at redhat.com>
wrote:

> Hi,
>
> I checked SAR data on storage servers and compared loads
> yesterday and three weeks ago (May 3). Load values are in
> pretty much the same range, yet now most of the time they
> are around the "high" mark so we may be nearing a bottleneck,
> specifically on I/O where we mostly do writes to the NAS,
> not reads and there's quite a bit of overhead:
>
> VM -> QCOW -> file -> network -> NFS -> DRBD -> disk
>
> Surely, using local scratch disks stored on SSDs should greatly
> improve performance as at least half of the above steps will be gone.
> We don't really need to centrally store (NFS) or mirror (DRBD)
> data that slaves write to their disks all the time anyways.
>
> For VMs where we do need redundancy, I'd suggest using
> iSCSI storage domains in the long run.
>
> Regards,
> Evgheni Dereveanchin
>
> ----- Original Message -----
> From: "Eyal Edri" <eedri at redhat.com>
> To: "Sandro Bonazzola" <sbonazzo at redhat.com>, "Evgheni Dereveanchin" <
> ederevea at redhat.com>, "Anton Marchukov" <amarchuk at redhat.com>
> Cc: "Fabian Deutsch" <fdeutsch at redhat.com>, "infra" <infra at ovirt.org>
> Sent: Wednesday, 25 May, 2016 9:31:43 AM
> Subject: Re: ngn build jobs take more than twice (x) as long as in the
> last days
>
> It might be more load on the storage servers with now running much more
> jobs.
> Evgheni - can you check if the load on the storage servers has changed
> significantly to justify this degradation of service?
>
> We need to expedite the enablement of SSDs in the hypervisors and move to
> local hooks.
> Anton - do we have a test VM that uses a local DISK we can use to test if
> it improves the runtime?
>
> On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo at redhat.com>
> wrote:
>
> >
> > Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch at redhat.com> ha scritto:
> > >
> > > Hey,
> > >
> > > $subj says it all.
> > >
> > > Affected jobs are:
> > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
> > >
> > > I.e. 3.6 - before: ~46min, now 1:23hrs
> > >
> > > In master it's even worse: >1:30hrs
> > >
> > > Can someone help to idnetify the reason?
> >
> > I have no numbers but I have the feeling that all jobs are getting slower
> > since a couple of weeks ago. Yum install phase takes ages. I thoughtit
> was
> > some temporary storage i/o peak but looks like it's not temporary.
> >
> > >
> > > - fabian
> > >
> > > --
> > > Fabian Deutsch <fdeutsch at redhat.com>
> > > RHEV Hypervisor
> > > Red Hat
> > > _______________________________________________
> > > Infra mailing list
> > > Infra at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> >
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>



-- 
Eyal Edri
Associate Manager
RHEV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160525/3bd0d813/attachment.html>