----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Mike Burns" <mburns@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, infra@ovirt.org Sent: Monday, March 18, 2013 6:48:58 PM Subject: Re: Loop devices exhausted on fedora-vm02
Am Montag, den 18.03.2013, 09:34 -0400 schrieb Mike Burns:
On 03/18/2013 08:25 AM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: infra@ovirt.org Sent: Monday, March 18, 2013 1:08:12 PM Subject: Loop devices exhausted on fedora-vm02
Hey,
it seems that all loop devices on fedora-vm02 are taken: http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console
Could someone look into this?
can you explain why it happens? (ovirt-node jobs?) maybe the job should add a cleanup step at end to release unused loop devices?
Looking here [1] there are 2 iso build jobs that could be running on the host at the same time.
Good catch!
We could throttle the jobs so that they can't run concurrently. We can also simply add something like max_loops=256 on the kernel commandline so that there are extra loop devices.
is this still needed? i don't see a /etc/modprobe.conf file on the f18 slave: i imagine this needs to be added: #vim /etc/modprobe.conf ... options loop max_loop=256. ...
The Node specififc job is throttled but I didn't take into account that another job might be editing/creating livecds.
We've seen this happen before, and livecd-tools is pretty good about cleaning up, even in failure scenarios (as long as you don't ctrl-c or kill -9 the job). But a single job uses a number of loop devices (3,4,5,6, not actually sure the number). It also uses lazy umounts in some cases meaning that there could be a couple loop devices in use that get cleaned up at a later point.
The simplest fix for this would be to increase the default number of loop devices.
+1 for this solution from me too. It seems that two concurrent livecd jobs just need to many loop devices.
- fabian