[ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

[Yeela Kaplan]

Just another thought,
from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers
and eventually a timeout occurs.
Adam, do you have any idea about this? 

----- Original Message -----
> From: "Yeela Kaplan" <ykaplan at redhat.com>
> To: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> Cc: "Users at ovirt.org" <users at ovirt.org>
> Sent: Wednesday, December 17, 2014 1:30:52 PM
> Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
> 
> 
> 
> ----- Original Message -----
> > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > To: "Yeela Kaplan" <ykaplan at redhat.com>
> > Cc: "Users at ovirt.org" <users at ovirt.org>
> > Sent: Wednesday, December 17, 2014 1:22:59 PM
> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
> > creating VM from template
> > 
> > Thank you, I will try to increase to 20 and see what happens. Bug is filed:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1175255
> 
> Thank you
> and I forgot to mention that you have to restart vdsmd so the change will
> apply...
> 
> 
> > 
> > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan at redhat.com> wrote:
> > >
> > > Hi,
> > > You are right, the problem is with the file handlers.
> > > You can increase the number of handlers in pool using the vdsm config
> > > file, which is supposed to be under the following directory in your
> > > system:
> > >
> > > /usr/lib/python2.6/site-packages/vdsm/config.py
> > >
> > > The default value for 'process_pool_max_slots_per_domain' is 10, so you
> > > can increase it by a bit, but not too much.
> > >
> > > But I suspect the problem is in a larger scale, and this is only a
> > > temporary relief for your system and this needs much more attention and a
> > > proper fix.
> > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the
> > > details of your setup and logs?
> > >
> > > thanks,
> > > Yeela
> > >
> > > ----- Original Message -----
> > > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > > > To: "Users at ovirt.org" <users at ovirt.org>
> > > > Sent: Wednesday, December 17, 2014 10:53:39 AM
> > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
> > > creating VM from template
> > > >
> > > > Would this be limits for the vdsm process? Then what is the proper way
> > > > to
> > > > change ulimits for VDSM?
> > > >
> > > > On 16 December 2014 at 20:45, Donny Davis < donny at cloudspin.me > wrote:
> > > >
> > > >
> > > >
> > > >
> > > > The only thing I can think of would be file hard and soft limits, but I
> > > am no
> > > > oVirt pro.
> > > >
> > > >
> > > >
> > > > 'no free file handlers in pool' that would make sense to me…
> > > >
> > > > Donny
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Tiemen Ruiten [mailto: t.ruiten at rdmedia.com ]
> > > > Sent: Tuesday, December 16, 2014 12:40 PM
> > > > To: Donny Davis
> > > > Cc: Users at ovirt.org
> > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
> > > > creating VM from template
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
> > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the
> > > VM,
> > > > processor load on both nodes was less than 1.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 16 December 2014 at 20:29, Donny Davis < donny at cloudspin.me > wrote:
> > > >
> > > > What is the VM load you are running on your servers?
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: users-bounces at ovirt.org [mailto: users-bounces at ovirt.org ] On
> > > Behalf Of
> > > > Tiemen Ruiten
> > > > Sent: Tuesday, December 16, 2014 12:27 PM
> > > > To: users at ovirt.org
> > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
> > > creating
> > > > VM from template
> > > >
> > > > Hello,
> > > >
> > > > I ran into a nasty problem today when creating a new, cloned VM from a
> > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the
> > > node
> > > > where I started a VM creation job, load skyrocketed and some VMs
> > > > stopped
> > > > responding until and after the job failed. Everything recovered without
> > > > intervention, but this obviously shouldn't happen. I have attached the
> > > > relevant vdsm log file. The button to create the VM was pressed around
> > > > 11:17, the first error in the vdsm log is at 11:23:58.
> > > >
> > > > The ISO domain is a gluster volume exposed via NFS, the storage domain
> > > for
> > > > the VM's is also a gluster volume. The underlying filesystem is ZFS.
> > > > The hypervisor nodes are full CentOS 6 installs.
> > > >
> > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file
> > > > is
> > > key
> > > > here. What can I do to prevent this from happening again? Apart from
> > > > not
> > > > creating new VMs of course :)
> > > >
> > > > Tiemen
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > Tiemen Ruiten
> > > > Systems Engineer
> > > > R&D Media
> > > >
> > > >
> > > > --
> > > > Tiemen Ruiten
> > > > Systems Engineer
> > > > R&D Media
> > > >
> > > > _______________________________________________
> > > > Users mailing list
> > > > Users at ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/users
> > > >
> > >
> > 
> > 
> > --
> > Tiemen Ruiten
> > Systems Engineer
> > R&D Media
> > 
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>