[ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

Tue Jan 6 13:20:59 UTC 2015

Yeela, any chance you could shed any light on this?

On 5 January 2015 at 15:46, Tiemen Ruiten <t.ruiten at rdmedia.com> wrote:

> Sorry, to be clear: no default disk policy exists for the 3.4 cluster
> storage domain.
>
> On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten at rdmedia.com> wrote:
>
>> Hello,
>>
>> I just tried creating a thin-provisioned disk and the same problem (load
>> spike, non responsive VMs) occurred. I may have an idea what is going wrong
>> now. The following is an excerpt from the engine log after starting the
>> disk create job:
>>
>> 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
>> (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
>> StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
>> vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
>> vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
>> LEFT OUTER JOIN storage_pool_with_storage_domain ON
>> vds_groups_storage_domain.storage_pool_id=
>> storage_pool_with_storage_domain.id    WHERE
>> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
>> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
>> org.postgresql.util.PSQLException: The column name disk_storage_type was
>> not found in this ResultSet. :
>> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
>> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
>> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
>>  vds_groups_storage_domain   LEFT OUTER JOIN
>> storage_pool_with_storage_domain ON
>> vds_groups_storage_domain.storage_pool_id=
>> storage_pool_with_storage_domain.id    WHERE
>> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
>> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
>> org.postgresql.util.PSQLException: The column name disk_storage_type was
>> not found in this ResultSet.:
>> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
>> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
>> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
>>  vds_groups_storage_domain   LEFT OUTER JOIN
>> storage_pool_with_storage_domain ON
>> vds_groups_storage_domain.storage_pool_id=
>> storage_pool_with_storage_domain.id    WHERE
>> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
>> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
>> org.postgresql.util.PSQLException: The column name disk_storage_type was
>> not found in this ResultSet.
>>
>> This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
>> There's also a 3.5 cluster managed by the same engine. Could this be
>> related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?
>>
>>
>>
>> On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan at redhat.com> wrote:
>>
>>> Just another thought,
>>> from looking at the vdsm logs it looks like there are too many calls to
>>> getVolumeSize that are eating up all handlers
>>> and eventually a timeout occurs.
>>> Adam, do you have any idea about this?
>>>
>>> ----- Original Message -----
>>> > From: "Yeela Kaplan" <ykaplan at redhat.com>
>>> > To: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>>> > Cc: "Users at ovirt.org" <users at ovirt.org>
>>> > Sent: Wednesday, December 17, 2014 1:30:52 PM
>>> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
>>> creating VM from template
>>> >
>>> >
>>> >
>>> > ----- Original Message -----
>>> > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>>> > > To: "Yeela Kaplan" <ykaplan at redhat.com>
>>> > > Cc: "Users at ovirt.org" <users at ovirt.org>
>>> > > Sent: Wednesday, December 17, 2014 1:22:59 PM
>>> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
>>> while
>>> > > creating VM from template
>>> > >
>>> > > Thank you, I will try to increase to 20 and see what happens. Bug is
>>> filed:
>>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255
>>> >
>>> > Thank you
>>> > and I forgot to mention that you have to restart vdsmd so the change
>>> will
>>> > apply...
>>> >
>>> >
>>> > >
>>> > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan at redhat.com>
>>> wrote:
>>> > > >
>>> > > > Hi,
>>> > > > You are right, the problem is with the file handlers.
>>> > > > You can increase the number of handlers in pool using the vdsm
>>> config
>>> > > > file, which is supposed to be under the following directory in your
>>> > > > system:
>>> > > >
>>> > > > /usr/lib/python2.6/site-packages/vdsm/config.py
>>> > > >
>>> > > > The default value for 'process_pool_max_slots_per_domain' is 10,
>>> so you
>>> > > > can increase it by a bit, but not too much.
>>> > > >
>>> > > > But I suspect the problem is in a larger scale, and this is only a
>>> > > > temporary relief for your system and this needs much more
>>> attention and a
>>> > > > proper fix.
>>> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all
>>> of the
>>> > > > details of your setup and logs?
>>> > > >
>>> > > > thanks,
>>> > > > Yeela
>>> > > >
>>> > > > ----- Original Message -----
>>> > > > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>>> > > > > To: "Users at ovirt.org" <users at ovirt.org>
>>> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM
>>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
>>> while
>>> > > > creating VM from template
>>> > > > >
>>> > > > > Would this be limits for the vdsm process? Then what is the
>>> proper way
>>> > > > > to
>>> > > > > change ulimits for VDSM?
>>> > > > >
>>> > > > > On 16 December 2014 at 20:45, Donny Davis < donny at cloudspin.me
>>> > wrote:
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > The only thing I can think of would be file hard and soft
>>> limits, but I
>>> > > > am no
>>> > > > > oVirt pro.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > 'no free file handlers in pool' that would make sense to me…
>>> > > > >
>>> > > > > Donny
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > From: Tiemen Ruiten [mailto: t.ruiten at rdmedia.com ]
>>> > > > > Sent: Tuesday, December 16, 2014 12:40 PM
>>> > > > > To: Donny Davis
>>> > > > > Cc: Users at ovirt.org
>>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
>>> while
>>> > > > > creating VM from template
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with
>>> 16
>>> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I
>>> created the
>>> > > > VM,
>>> > > > > processor load on both nodes was less than 1.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On 16 December 2014 at 20:29, Donny Davis < donny at cloudspin.me
>>> > wrote:
>>> > > > >
>>> > > > > What is the VM load you are running on your servers?
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > -----Original Message-----
>>> > > > > From: users-bounces at ovirt.org [mailto: users-bounces at ovirt.org
>>> ] On
>>> > > > Behalf Of
>>> > > > > Tiemen Ruiten
>>> > > > > Sent: Tuesday, December 16, 2014 12:27 PM
>>> > > > > To: users at ovirt.org
>>> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool'
>>> while
>>> > > > creating
>>> > > > > VM from template
>>> > > > >
>>> > > > > Hello,
>>> > > > >
>>> > > > > I ran into a nasty problem today when creating a new, cloned VM
>>> from a
>>> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster:
>>> on the
>>> > > > node
>>> > > > > where I started a VM creation job, load skyrocketed and some VMs
>>> > > > > stopped
>>> > > > > responding until and after the job failed. Everything recovered
>>> without
>>> > > > > intervention, but this obviously shouldn't happen. I have
>>> attached the
>>> > > > > relevant vdsm log file. The button to create the VM was pressed
>>> around
>>> > > > > 11:17, the first error in the vdsm log is at 11:23:58.
>>> > > > >
>>> > > > > The ISO domain is a gluster volume exposed via NFS, the storage
>>> domain
>>> > > > for
>>> > > > > the VM's is also a gluster volume. The underlying filesystem is
>>> ZFS.
>>> > > > > The hypervisor nodes are full CentOS 6 installs.
>>> > > > >
>>> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log
>>> file
>>> > > > > is
>>> > > > key
>>> > > > > here. What can I do to prevent this from happening again? Apart
>>> from
>>> > > > > not
>>> > > > > creating new VMs of course :)
>>> > > > >
>>> > > > > Tiemen
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > >
>>> > > > >
>>> > > > > Tiemen Ruiten
>>> > > > > Systems Engineer
>>> > > > > R&D Media
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Tiemen Ruiten
>>> > > > > Systems Engineer
>>> > > > > R&D Media
>>> > > > >
>>> > > > > _______________________________________________
>>> > > > > Users mailing list
>>> > > > > Users at ovirt.org
>>> > > > > http://lists.ovirt.org/mailman/listinfo/users
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Tiemen Ruiten
>>> > > Systems Engineer
>>> > > R&D Media
>>> > >
>>> > _______________________________________________
>>> > Users mailing list
>>> > Users at ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>>
>>
>>
>>
>> --
>> Tiemen Ruiten
>> Systems Engineer
>> R&D Media
>>
>
>
>
> --
> Tiemen Ruiten
> Systems Engineer
> R&D Media
>

-- 
Tiemen Ruiten
Systems Engineer
R&D Media
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150106/399f998c/attachment-0001.html>