Re: Loop devices exhausted on fedora-vm02

From: "Fabian Deutsch" <fabiand(a)redhat.com&gt; To: "Mike Burns" <mburns(a)redhat.com&gt; Cc: "Eyal Edri" <eedri(a)redhat.com&gt;, infra(a)ovirt.org Sent: Monday, March 18, 2013 6:48:58 PM Subject: Re: Loop devices exhausted on fedora-vm02 Am Montag, den 18.03.2013, 09:34 -0400 schrieb Mike Burns: > On 03/18/2013 08:25 AM, Eyal Edri wrote: > > > > > > ----- Original Message ----- > >> From: "Fabian Deutsch" <fabiand(a)redhat.com&gt; > >> To: infra(a)ovirt.org > >> Sent: Monday, March 18, 2013 1:08:12 PM > >> Subject: Loop devices exhausted on fedora-vm02 > >> > >> Hey, > >> > >> it seems that all loop devices on fedora-vm02 > >> are taken: > >> http://jenkins.ovirt.org/view/ovirt_node/job/ovirt-node-iso/671/console > >> > >> Could someone look into this? > > > > can you explain why it happens? (ovirt-node jobs?) > > maybe the job should add a cleanup step at end to release unused > > loop devices? > > Looking here [1] there are 2 iso build jobs that could be running > on the > host at the same time. Good catch! > We could throttle the jobs so that they can't run concurrently. We > can > also simply add something like max_loops=256 on the kernel > commandline > so that there are extra loop devices.

The Node specififc job is throttled but I didn't take into account that another job might be editing/creating livecds. > We've seen this happen before, and livecd-tools is pretty good > about > cleaning up, even in failure scenarios (as long as you don't ctrl-c > or > kill -9 the job). But a single job uses a number of loop devices > (3,4,5,6, not actually sure the number). It also uses lazy umounts > in > some cases meaning that there could be a couple loop devices in use > that > get cleaned up at a later point. > > The simplest fix for this would be to increase the default number > of > loop devices. +1 for this solution from me too. It seems that two concurrent livecd jobs just need to many loop devices. - fabian