[ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

Mon Jan 5 14:45:23 UTC 2015

Hello,

I just tried creating a thin-provisioned disk and the same problem (load
spike, non responsive VMs) occurred. I may have an idea what is going wrong
now. The following is an excerpt from the engine log after starting the
disk create job:

2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
(ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
LEFT OUTER JOIN storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.id    WHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet. :
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
 vds_groups_storage_domain   LEFT OUTER JOIN
storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.id    WHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet.:
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
 vds_groups_storage_domain   LEFT OUTER JOIN
storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.id    WHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet.

This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
There's also a 3.5 cluster managed by the same engine. Could this be
related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?

On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan at redhat.com> wrote:

> Just another thought,
> from looking at the vdsm logs it looks like there are too many calls to
> getVolumeSize that are eating up all handlers
> and eventually a timeout occurs.
> Adam, do you have any idea about this?
>
> ----- Original Message -----
> > From: "Yeela Kaplan" <ykaplan at redhat.com>
> > To: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > Cc: "Users at ovirt.org" <users at ovirt.org>
> > Sent: Wednesday, December 17, 2014 1:30:52 PM
> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
> creating VM from template
> >
> >
> >
> > ----- Original Message -----
> > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > > To: "Yeela Kaplan" <ykaplan at redhat.com>
> > > Cc: "Users at ovirt.org" <users at ovirt.org>
> > > Sent: Wednesday, December 17, 2014 1:22:59 PM
> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
> > > creating VM from template
> > >
> > > Thank you, I will try to increase to 20 and see what happens. Bug is
> filed:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255
> >
> > Thank you
> > and I forgot to mention that you have to restart vdsmd so the change will
> > apply...
> >
> >
> > >
> > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan at redhat.com> wrote:
> > > >
> > > > Hi,
> > > > You are right, the problem is with the file handlers.
> > > > You can increase the number of handlers in pool using the vdsm config
> > > > file, which is supposed to be under the following directory in your
> > > > system:
> > > >
> > > > /usr/lib/python2.6/site-packages/vdsm/config.py
> > > >
> > > > The default value for 'process_pool_max_slots_per_domain' is 10, so
> you
> > > > can increase it by a bit, but not too much.
> > > >
> > > > But I suspect the problem is in a larger scale, and this is only a
> > > > temporary relief for your system and this needs much more attention
> and a
> > > > proper fix.
> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of
> the
> > > > details of your setup and logs?
> > > >
> > > > thanks,
> > > > Yeela
> > > >
> > > > ----- Original Message -----
> > > > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > > > > To: "Users at ovirt.org" <users at ovirt.org>
> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM
> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
> while
> > > > creating VM from template
> > > > >
> > > > > Would this be limits for the vdsm process? Then what is the proper
> way
> > > > > to
> > > > > change ulimits for VDSM?
> > > > >
> > > > > On 16 December 2014 at 20:45, Donny Davis < donny at cloudspin.me >
> wrote:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The only thing I can think of would be file hard and soft limits,
> but I
> > > > am no
> > > > > oVirt pro.
> > > > >
> > > > >
> > > > >
> > > > > 'no free file handlers in pool' that would make sense to me…
> > > > >
> > > > > Donny
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Tiemen Ruiten [mailto: t.ruiten at rdmedia.com ]
> > > > > Sent: Tuesday, December 16, 2014 12:40 PM
> > > > > To: Donny Davis
> > > > > Cc: Users at ovirt.org
> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
> while
> > > > > creating VM from template
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created
> the
> > > > VM,
> > > > > processor load on both nodes was less than 1.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 16 December 2014 at 20:29, Donny Davis < donny at cloudspin.me >
> wrote:
> > > > >
> > > > > What is the VM load you are running on your servers?
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: users-bounces at ovirt.org [mailto: users-bounces at ovirt.org ]
> On
> > > > Behalf Of
> > > > > Tiemen Ruiten
> > > > > Sent: Tuesday, December 16, 2014 12:27 PM
> > > > > To: users at ovirt.org
> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
> > > > creating
> > > > > VM from template
> > > > >
> > > > > Hello,
> > > > >
> > > > > I ran into a nasty problem today when creating a new, cloned VM
> from a
> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on
> the
> > > > node
> > > > > where I started a VM creation job, load skyrocketed and some VMs
> > > > > stopped
> > > > > responding until and after the job failed. Everything recovered
> without
> > > > > intervention, but this obviously shouldn't happen. I have attached
> the
> > > > > relevant vdsm log file. The button to create the VM was pressed
> around
> > > > > 11:17, the first error in the vdsm log is at 11:23:58.
> > > > >
> > > > > The ISO domain is a gluster volume exposed via NFS, the storage
> domain
> > > > for
> > > > > the VM's is also a gluster volume. The underlying filesystem is
> ZFS.
> > > > > The hypervisor nodes are full CentOS 6 installs.
> > > > >
> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log
> file
> > > > > is
> > > > key
> > > > > here. What can I do to prevent this from happening again? Apart
> from
> > > > > not
> > > > > creating new VMs of course :)
> > > > >
> > > > > Tiemen
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > Tiemen Ruiten
> > > > > Systems Engineer
> > > > > R&D Media
> > > > >
> > > > >
> > > > > --
> > > > > Tiemen Ruiten
> > > > > Systems Engineer
> > > > > R&D Media
> > > > >
> > > > > _______________________________________________
> > > > > Users mailing list
> > > > > Users at ovirt.org
> > > > > http://lists.ovirt.org/mailman/listinfo/users
> > > > >
> > > >
> > >
> > >
> > > --
> > > Tiemen Ruiten
> > > Systems Engineer
> > > R&D Media
> > >
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>

-- 
Tiemen Ruiten
Systems Engineer
R&D Media
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150105/f816e416/attachment-0001.html>