Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this? ----- Original Message ----- > From: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > To: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > Sent: Wednesday, December 17, 2014 1:30:52 PM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template > > > > ----- Original Message ----- > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > Thank you, I will try to increase to 20 and see what happens. Bug is filed: > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > Thank you > and I forgot to mention that you have to restart vdsmd so the change will > apply... > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > > > > > > Hi, > > > You are right, the problem is with the file handlers. > > > You can increase the number of handlers in pool using the vdsm config > > > file, which is supposed to be under the following directory in your > > > system: > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > > > can increase it by a bit, but not too much. > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > temporary relief for your system and this needs much more attention and a > > > proper fix. > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > > > details of your setup and logs? > > > > > > thanks, > > > Yeela > > > > > > ----- Original Message ----- > > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > > > Would this be limits for the vdsm process? Then what is the proper way > > > > to > > > > change ulimits for VDSM? > > > > > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me > wrote: > > > > > > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > > > am no > > > > oVirt pro. > > > > > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > > > Donny > > > > > > > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] > > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > > To: Donny Davis > > > > Cc: Users(a)ovirt.org > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > > creating VM from template > > > > > > > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > > > VM, > > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me > wrote: > > > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org ] On > > > Behalf Of > > > > Tiemen Ruiten > > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > > To: users(a)ovirt.org > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating > > > > VM from template > > > > > > > > Hello, > > > > > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > > > node > > > > where I started a VM creation job, load skyrocketed and some VMs > > > > stopped > > > > responding until and after the job failed. Everything recovered without > > > > intervention, but this obviously shouldn't happen. I have attached the > > > > relevant vdsm log file. The button to create the VM was pressed around > > > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > > > for > > > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > > > The hypervisor nodes are full CentOS 6 installs. > > > > > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file > > > > is > > > key > > > > here. What can I do to prevent this from happening again? Apart from > > > > not > > > > creating new VMs of course :) > > > > > > > > Tiemen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > > > > > -- > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > _______________________________________________ > > > > Users mailing list > > > > Users(a)ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > > > > > -- > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >