ngn build jobs take more than twice (x) as long as in the last days

[Fabian Deutsch]

On Tue, May 24, 2016 at 8:49 PM, Eyal Edri <eedri at redhat.com> wrote:
> btw, feel free to increase history for node builds now, we have the space
> for it.
> I wouldn't got crazy with keeping history of artifacts, but build info with
> console output there is no problem keeping a month old.

~2GB * 3 * 30 == 180GB of disk space, ok?

> I'm adding Barak as this weekly infra owner to help debug this tomorrow if
> needed.
> Also, Evgheni is working on network configuration, might be relevant as
> well.

Actually, one thought:

I think it might make sense to change the standard job template to
support nesting.
Currently i need to create /dev/kvm to get at least that acceleration.
(If the app inside mock s speaking to /dev/kvm directly).
Also for libvirt a mount is needed.

My take would be: Change the standard ci job template so that every
job can use libvirt + geustfosh (w/ acceleration) right away without
needing these hacks.

If that sounds viable, then I can open a ticket.

- fabain

> On Tue, May 24, 2016 at 9:30 PM, Fabian Deutsch <fdeutsch at redhat.com> wrote:
>>
>> I don't know if it was quicker on different salves, because I can not
>> look at the history.
>>
>> There were no Node sided changes, maybe the package set in master got
>> larger - well see that on the first regular build.
>>
>> But: I enabled kvm in mock, and this seems to help. The first job post
>> this chnage looks good.
>>
>> My assumption is that previous builds were running on a beefier slave.
>>
>> - fabian
>>
>> On Tue, May 24, 2016 at 6:50 PM, Eyal Edri <eedri at redhat.com> wrote:
>> > Is it running on the same slave as before?  Can we see a changelog of
>> > things
>> > changed from the Last time it took less time?
>> >
>> > On May 24, 2016 7:06 PM, "David Caro" <dcaro at redhat.com> wrote:
>> >>
>> >> On 05/24 18:03, David Caro wrote:
>> >> > On 05/24 17:57, Fabian Deutsch wrote:
>> >> > > Hey,
>> >> > >
>> >> > > $subj says it all.
>> >> > >
>> >> > > Affected jobs are:
>> >> > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
>> >> > >
>> >> > > I.e. 3.6 - before: ~46min, now 1:23hrs
>> >> > >
>> >> > > In master it's even worse: >1:30hrs
>> >> > >
>> >> > > Can someone help to idnetify the reason?
>> >> >
>> >> >
>> >> > I see that this is where there's a big jump in time:
>> >> >
>> >> > 06:39:38 Domain installation still in progress. You can reconnect to
>> >> > 06:39:38 the console to complete the installation process.
>> >> > 07:21:51
>> >> >
>> >> > .............................................................................................................................................................................................................................................................2016-05-24
>> >> > 03:21:51,341: Install finished. Or at least virt shut down.
>> >> >
>> >> > So it looks as if the code that checks if the domain is shut down is
>> >> > not
>> >> > working properly, or maybe the virt-install is taking very very long
>> >> > to
>> >> > work.
>> >>
>> >>
>> >> It seems that the virt-install log is not being archived, and the
>> >> workdir
>> >> has
>> >> already been cleaned up so I can check the logfile:
>> >>
>> >>
>> >>
>> >> /home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log
>> >>
>> >>
>> >> Maybe you can archive it too on the next run to debug
>> >>
>> >> >
>> >> > >
>> >> > > - fabian
>> >> > >
>> >> > > --
>> >> > > Fabian Deutsch <fdeutsch at redhat.com>
>> >> > > RHEV Hypervisor
>> >> > > Red Hat
>> >> > > _______________________________________________
>> >> > > Infra mailing list
>> >> > > Infra at ovirt.org
>> >> > > http://lists.ovirt.org/mailman/listinfo/infra
>> >> >
>> >> > --
>> >> > David Caro
>> >> >
>> >> > Red Hat S.L.
>> >> > Continuous Integration Engineer - EMEA ENG Virtualization R&D
>> >> >
>> >> > Tel.: +420 532 294 605
>> >> > Email: dcaro at redhat.com
>> >> > IRC: dcaro|dcaroest@{freenode|oftc|redhat}
>> >> > Web: www.redhat.com
>> >> > RHT Global #: 82-62605
>> >>
>> >>
>> >>
>> >> --
>> >> David Caro
>> >>
>> >> Red Hat S.L.
>> >> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>> >>
>> >> Tel.: +420 532 294 605
>> >> Email: dcaro at redhat.com
>> >> IRC: dcaro|dcaroest@{freenode|oftc|redhat}
>> >> Web: www.redhat.com
>> >> RHT Global #: 82-62605
>> >>
>> >> _______________________________________________
>> >> Infra mailing list
>> >> Infra at ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/infra
>> >>
>> >
>>
>>
>>
>> --
>> Fabian Deutsch <fdeutsch at redhat.com>
>> RHEV Hypervisor
>> Red Hat
>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)



-- 
Fabian Deutsch <fdeutsch at redhat.com>
RHEV Hypervisor
Red Hat