Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

Hello, I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job: 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > Just another thought, > from looking at the vdsm logs it looks like there are too many calls to > getVolumeSize that are eating up all handlers > and eventually a timeout occurs. > Adam, do you have any idea about this? > > ----- Original Message ----- > > From: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > > To: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > Sent: Wednesday, December 17, 2014 1:30:52 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > > > > > ----- Original Message ----- > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > > To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > Thank you, I will try to increase to 20 and see what happens. Bug is > filed: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > > > Thank you > > and I forgot to mention that you have to restart vdsmd so the change > will > > apply... > > > > > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; > wrote: > > > > > > > > Hi, > > > > You are right, the problem is with the file handlers. > > > > You can increase the number of handlers in pool using the vdsm > config > > > > file, which is supposed to be under the following directory in your > > > > system: > > > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so > you > > > > can increase it by a bit, but not too much. > > > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > > temporary relief for your system and this needs much more attention > and a > > > > proper fix. > > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of > the > > > > details of your setup and logs? > > > > > > > > thanks, > > > > Yeela > > > > > > > > ----- Original Message ----- > > > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > > > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' > while > > > > creating VM from template > > > > > > > > > > Would this be limits for the vdsm process? Then what is the > proper way > > > > > to > > > > > change ulimits for VDSM? > > > > > > > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, > but I > > > > am no > > > > > oVirt pro. > > > > > > > > > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > > > > > Donny > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] > > > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > > > To: Donny Davis > > > > > Cc: Users(a)ovirt.org > > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' > while > > > > > creating VM from template > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I > created the > > > > VM, > > > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me > > wrote: > > > > > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org ] > On > > > > Behalf Of > > > > > Tiemen Ruiten > > > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > > > To: users(a)ovirt.org > > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > > > creating > > > > > VM from template > > > > > > > > > > Hello, > > > > > > > > > > I ran into a nasty problem today when creating a new, cloned VM > from a > > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: > on the > > > > node > > > > > where I started a VM creation job, load skyrocketed and some VMs > > > > > stopped > > > > > responding until and after the job failed. Everything recovered > without > > > > > intervention, but this obviously shouldn't happen. I have > attached the > > > > > relevant vdsm log file. The button to create the VM was pressed > around > > > > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > > > > > > > The ISO domain is a gluster volume exposed via NFS, the storage > domain > > > > for > > > > > the VM's is also a gluster volume. The underlying filesystem is > ZFS. > > > > > The hypervisor nodes are full CentOS 6 installs. > > > > > > > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log > file > > > > > is > > > > key > > > > > here. What can I do to prevent this from happening again? Apart > from > > > > > not > > > > > creating new VMs of course :) > > > > > > > > > > Tiemen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Tiemen Ruiten > > > > > Systems Engineer > > > > > R&D Media > > > > > > > > > > > > > > > -- > > > > > Tiemen Ruiten > > > > > Systems Engineer > > > > > R&D Media > > > > > > > > > > _______________________________________________ > > > > > Users mailing list > > > > > Users(a)ovirt.org > > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > > > > > > > > > > -- > > > Tiemen Ruiten > > > Systems Engineer > > > R&D Media > > > > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > -- Tiemen Ruiten Systems Engineer R&D Media