6:08 a.m.
Hey,
it seems that all loop devices on fedora-vm02
are taken:
http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
Could someone look into this?
Thanks
fabian
7:25 a.m.
----- Original Message -----
From: "Fabian Deutsch" <fabiand(a)redhat.com>
To: infra(a)ovirt.org
Sent: Monday, March 18, 2013 1:08:12 PM
Subject: Loop devices exhausted on fedora-vm02
Hey,
it seems that all loop devices on fedora-vm02
are taken:
http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
Could someone look into this?
can you explain why it happens? (ovirt-node jobs?)
maybe the job should add a cleanup step at end to release unused loop devices?
Thanks
fabian
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
8:34 a.m.
On 03/18/2013 08:25 AM, Eyal Edri wrote:
----- Original Message -----
> From: "Fabian Deutsch" <fabiand(a)redhat.com>
> To: infra(a)ovirt.org
> Sent: Monday, March 18, 2013 1:08:12 PM
> Subject: Loop devices exhausted on fedora-vm02
>
> Hey,
>
> it seems that all loop devices on fedora-vm02
> are taken:
> http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
>
> Could someone look into this?
can you explain why it happens? (ovirt-node jobs?)
maybe the job should add a cleanup step at end to release unused loop devices?
Looking here [1] there are 2 iso build jobs that could be running on the
host at the same time.
We could throttle the jobs so that they can't run concurrently. We can
also simply add something like max_loops=256 on the kernel commandline
so that there are extra loop devices.
We've seen this happen before, and livecd-tools is pretty good about
cleaning up, even in failure scenarios (as long as you don't ctrl-c or
kill -9 the job). But a single job uses a number of loop devices
(3,4,5,6, not actually sure the number). It also uses lazy umounts in
some cases meaning that there could be a couple loop devices in use that
get cleaned up at a later point.
The simplest fix for this would be to increase the default number of
loop devices.
Mike
[1] http://jenkins.ovirt.org/computer/fedora18-vm02/
>
> Thanks
> fabian
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
11:48 a.m.
Am Montag, den 18.03.2013, 09:34 -0400 schrieb Mike Burns:
On 03/18/2013 08:25 AM, Eyal Edri wrote:
>
>
> ----- Original Message -----
>> From: "Fabian Deutsch" <fabiand(a)redhat.com>
>> To: infra(a)ovirt.org
>> Sent: Monday, March 18, 2013 1:08:12 PM
>> Subject: Loop devices exhausted on fedora-vm02
>>
>> Hey,
>>
>> it seems that all loop devices on fedora-vm02
>> are taken:
>> http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
>>
>> Could someone look into this?
>
> can you explain why it happens? (ovirt-node jobs?)
> maybe the job should add a cleanup step at end to release unused loop devices?
Looking here [1] there are 2 iso build jobs that could be running on the
host at the same time.
Good catch!
We could throttle the jobs so that they can't run concurrently.
We can
also simply add something like max_loops=256 on the kernel commandline
so that there are extra loop devices.
The Node specififc job is throttled but I didn't take into account that
another job might be editing/creating livecds.
We've seen this happen before, and livecd-tools is pretty good
about
cleaning up, even in failure scenarios (as long as you don't ctrl-c or
kill -9 the job). But a single job uses a number of loop devices
(3,4,5,6, not actually sure the number). It also uses lazy umounts in
some cases meaning that there could be a couple loop devices in use that
get cleaned up at a later point.
The simplest fix for this would be to increase the default number of
loop devices.
+1 for this solution from me too.
It seems that two concurrent livecd jobs just need to many loop devices.
- fabian
5:01 a.m.
----- Original Message -----
From: "Fabian Deutsch" <fabiand(a)redhat.com>
To: "Mike Burns" <mburns(a)redhat.com>
Cc: "Eyal Edri" <eedri(a)redhat.com>, infra(a)ovirt.org
Sent: Monday, March 18, 2013 6:48:58 PM
Subject: Re: Loop devices exhausted on fedora-vm02
Am Montag, den 18.03.2013, 09:34 -0400 schrieb Mike Burns:
> On 03/18/2013 08:25 AM, Eyal Edri wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Fabian Deutsch" <fabiand(a)redhat.com>
> >> To: infra(a)ovirt.org
> >> Sent: Monday, March 18, 2013 1:08:12 PM
> >> Subject: Loop devices exhausted on fedora-vm02
> >>
> >> Hey,
> >>
> >> it seems that all loop devices on fedora-vm02
> >> are taken:
> >> http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
> >>
> >> Could someone look into this?
> >
> > can you explain why it happens? (ovirt-node jobs?)
> > maybe the job should add a cleanup step at end to release unused
> > loop devices?
>
> Looking here [1] there are 2 iso build jobs that could be running
> on the
> host at the same time.
Good catch!
> We could throttle the jobs so that they can't run concurrently. We
> can
> also simply add something like max_loops=256 on the kernel
> commandline
> so that there are extra loop devices.
is this still needed?
i don't see a /etc/modprobe.conf file on the f18 slave:
i imagine this needs to be added:
#vim /etc/modprobe.conf
...
options loop max_loop=256.
...
The Node specififc job is throttled but I didn't take into account
that
another job might be editing/creating livecds.
> We've seen this happen before, and livecd-tools is pretty good
> about
> cleaning up, even in failure scenarios (as long as you don't ctrl-c
> or
> kill -9 the job). But a single job uses a number of loop devices
> (3,4,5,6, not actually sure the number). It also uses lazy umounts
> in
> some cases meaning that there could be a couple loop devices in use
> that
> get cleaned up at a later point.
>
> The simplest fix for this would be to increase the default number
> of
> loop devices.
+1 for this solution from me too.
It seems that two concurrent livecd jobs just need to many loop
devices.
- fabian
4053
days inactive
4056
days old
4 comments
3 participants
participants (3)
-
Eyal Edri -
Fabian Deutsch -
Mike Burns