[ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Tiemen Ruiten
t.ruiten at rdmedia.com
Mon Jan 5 14:46:21 UTC 2015
Sorry, to be clear: no default disk policy exists for the 3.4 cluster
storage domain.
On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten at rdmedia.com> wrote:
> Hello,
>
> I just tried creating a thin-provisioned disk and the same problem (load
> spike, non responsive VMs) occurred. I may have an idea what is going wrong
> now. The following is an excerpt from the engine log after starting the
> disk create job:
>
> 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
> (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
> StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
> vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
> vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain
> LEFT OUTER JOIN storage_pool_with_storage_domain ON
> vds_groups_storage_domain.storage_pool_id=
> storage_pool_with_storage_domain.id WHERE
> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY
> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
> org.postgresql.util.PSQLException: The column name disk_storage_type was
> not found in this ResultSet. :
> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
> vds_groups_storage_domain LEFT OUTER JOIN
> storage_pool_with_storage_domain ON
> vds_groups_storage_domain.storage_pool_id=
> storage_pool_with_storage_domain.id WHERE
> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY
> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
> org.postgresql.util.PSQLException: The column name disk_storage_type was
> not found in this ResultSet.:
> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
> vds_groups_storage_domain LEFT OUTER JOIN
> storage_pool_with_storage_domain ON
> vds_groups_storage_domain.storage_pool_id=
> storage_pool_with_storage_domain.id WHERE
> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY
> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
> org.postgresql.util.PSQLException: The column name disk_storage_type was
> not found in this ResultSet.
>
> This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
> There's also a 3.5 cluster managed by the same engine. Could this be
> related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?
>
>
>
> On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan at redhat.com> wrote:
>
>> Just another thought,
>> from looking at the vdsm logs it looks like there are too many calls to
>> getVolumeSize that are eating up all handlers
>> and eventually a timeout occurs.
>> Adam, do you have any idea about this?
>>
>> ----- Original Message -----
>> > From: "Yeela Kaplan" <ykaplan at redhat.com>
>> > To: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>> > Cc: "Users at ovirt.org" <users at ovirt.org>
>> > Sent: Wednesday, December 17, 2014 1:30:52 PM
>> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
>> creating VM from template
>> >
>> >
>> >
>> > ----- Original Message -----
>> > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>> > > To: "Yeela Kaplan" <ykaplan at redhat.com>
>> > > Cc: "Users at ovirt.org" <users at ovirt.org>
>> > > Sent: Wednesday, December 17, 2014 1:22:59 PM
>> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
>> > > creating VM from template
>> > >
>> > > Thank you, I will try to increase to 20 and see what happens. Bug is
>> filed:
>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255
>> >
>> > Thank you
>> > and I forgot to mention that you have to restart vdsmd so the change
>> will
>> > apply...
>> >
>> >
>> > >
>> > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan at redhat.com>
>> wrote:
>> > > >
>> > > > Hi,
>> > > > You are right, the problem is with the file handlers.
>> > > > You can increase the number of handlers in pool using the vdsm
>> config
>> > > > file, which is supposed to be under the following directory in your
>> > > > system:
>> > > >
>> > > > /usr/lib/python2.6/site-packages/vdsm/config.py
>> > > >
>> > > > The default value for 'process_pool_max_slots_per_domain' is 10, so
>> you
>> > > > can increase it by a bit, but not too much.
>> > > >
>> > > > But I suspect the problem is in a larger scale, and this is only a
>> > > > temporary relief for your system and this needs much more attention
>> and a
>> > > > proper fix.
>> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of
>> the
>> > > > details of your setup and logs?
>> > > >
>> > > > thanks,
>> > > > Yeela
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
>> > > > > To: "Users at ovirt.org" <users at ovirt.org>
>> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM
>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
>> while
>> > > > creating VM from template
>> > > > >
>> > > > > Would this be limits for the vdsm process? Then what is the
>> proper way
>> > > > > to
>> > > > > change ulimits for VDSM?
>> > > > >
>> > > > > On 16 December 2014 at 20:45, Donny Davis < donny at cloudspin.me >
>> wrote:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > The only thing I can think of would be file hard and soft limits,
>> but I
>> > > > am no
>> > > > > oVirt pro.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 'no free file handlers in pool' that would make sense to me…
>> > > > >
>> > > > > Donny
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > From: Tiemen Ruiten [mailto: t.ruiten at rdmedia.com ]
>> > > > > Sent: Tuesday, December 16, 2014 12:40 PM
>> > > > > To: Donny Davis
>> > > > > Cc: Users at ovirt.org
>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
>> while
>> > > > > creating VM from template
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
>> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I
>> created the
>> > > > VM,
>> > > > > processor load on both nodes was less than 1.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On 16 December 2014 at 20:29, Donny Davis < donny at cloudspin.me >
>> wrote:
>> > > > >
>> > > > > What is the VM load you are running on your servers?
>> > > > >
>> > > > >
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: users-bounces at ovirt.org [mailto: users-bounces at ovirt.org ]
>> On
>> > > > Behalf Of
>> > > > > Tiemen Ruiten
>> > > > > Sent: Tuesday, December 16, 2014 12:27 PM
>> > > > > To: users at ovirt.org
>> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
>> > > > creating
>> > > > > VM from template
>> > > > >
>> > > > > Hello,
>> > > > >
>> > > > > I ran into a nasty problem today when creating a new, cloned VM
>> from a
>> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster:
>> on the
>> > > > node
>> > > > > where I started a VM creation job, load skyrocketed and some VMs
>> > > > > stopped
>> > > > > responding until and after the job failed. Everything recovered
>> without
>> > > > > intervention, but this obviously shouldn't happen. I have
>> attached the
>> > > > > relevant vdsm log file. The button to create the VM was pressed
>> around
>> > > > > 11:17, the first error in the vdsm log is at 11:23:58.
>> > > > >
>> > > > > The ISO domain is a gluster volume exposed via NFS, the storage
>> domain
>> > > > for
>> > > > > the VM's is also a gluster volume. The underlying filesystem is
>> ZFS.
>> > > > > The hypervisor nodes are full CentOS 6 installs.
>> > > > >
>> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log
>> file
>> > > > > is
>> > > > key
>> > > > > here. What can I do to prevent this from happening again? Apart
>> from
>> > > > > not
>> > > > > creating new VMs of course :)
>> > > > >
>> > > > > Tiemen
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > >
>> > > > > Tiemen Ruiten
>> > > > > Systems Engineer
>> > > > > R&D Media
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Tiemen Ruiten
>> > > > > Systems Engineer
>> > > > > R&D Media
>> > > > >
>> > > > > _______________________________________________
>> > > > > Users mailing list
>> > > > > Users at ovirt.org
>> > > > > http://lists.ovirt.org/mailman/listinfo/users
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Tiemen Ruiten
>> > > Systems Engineer
>> > > R&D Media
>> > >
>> > _______________________________________________
>> > Users mailing list
>> > Users at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>>
>
>
>
> --
> Tiemen Ruiten
> Systems Engineer
> R&D Media
>
--
Tiemen Ruiten
Systems Engineer
R&D Media
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150105/267fad2c/attachment-0001.html>
More information about the Users
mailing list