Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain. On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten(a)rdmedia.com&gt; wrote: > Hello, > > I just tried creating a thin-provisioned disk and the same problem (load > spike, non responsive VMs) occurred. I may have an idea what is going wrong > now. The following is an excerpt from the engine log after starting the > disk create job: > > 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] > (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is > StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM > vds_groups_view WHERE ( vds_group_id IN (SELECT distinct > vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain > LEFT OUTER JOIN storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. : > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet.: > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. > > This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. > There's also a 3.5 cluster managed by the same engine. Could this be > related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? > > > > On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > >> Just another thought, >> from looking at the vdsm logs it looks like there are too many calls to >> getVolumeSize that are eating up all handlers >> and eventually a timeout occurs. >> Adam, do you have any idea about this? >> >> ----- Original Message ----- >> > From: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; >> > To: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > Sent: Wednesday, December 17, 2014 1:30:52 PM >> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >> creating VM from template >> > >> > >> > >> > ----- Original Message ----- >> > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > > To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; >> > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > creating VM from template >> > > >> > > Thank you, I will try to increase to 20 and see what happens. Bug is >> filed: >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >> > >> > Thank you >> > and I forgot to mention that you have to restart vdsmd so the change >> will >> > apply... >> > >> > >> > > >> > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; >> wrote: >> > > > >> > > > Hi, >> > > > You are right, the problem is with the file handlers. >> > > > You can increase the number of handlers in pool using the vdsm >> config >> > > > file, which is supposed to be under the following directory in your >> > > > system: >> > > > >> > > > /usr/lib/python2.6/site-packages/vdsm/config.py >> > > > >> > > > The default value for 'process_pool_max_slots_per_domain' is 10, >> so you >> > > > can increase it by a bit, but not too much. >> > > > >> > > > But I suspect the problem is in a larger scale, and this is only a >> > > > temporary relief for your system and this needs much more >> attention and a >> > > > proper fix. >> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all >> of the >> > > > details of your setup and logs? >> > > > >> > > > thanks, >> > > > Yeela >> > > > >> > > > ----- Original Message ----- >> > > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > > > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM >> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > creating VM from template >> > > > > >> > > > > Would this be limits for the vdsm process? Then what is the >> proper way >> > > > > to >> > > > > change ulimits for VDSM? >> > > > > >> > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me >> > wrote: >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > The only thing I can think of would be file hard and soft >> limits, but I >> > > > am no >> > > > > oVirt pro. >> > > > > >> > > > > >> > > > > >> > > > > 'no free file handlers in pool' that would make sense to me… >> > > > > >> > > > > Donny >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] >> > > > > Sent: Tuesday, December 16, 2014 12:40 PM >> > > > > To: Donny Davis >> > > > > Cc: Users(a)ovirt.org >> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > > creating VM from template >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with >> 16 >> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I >> created the >> > > > VM, >> > > > > processor load on both nodes was less than 1. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me >> > wrote: >> > > > > >> > > > > What is the VM load you are running on your servers? >> > > > > >> > > > > >> > > > > >> > > > > -----Original Message----- >> > > > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org >> ] On >> > > > Behalf Of >> > > > > Tiemen Ruiten >> > > > > Sent: Tuesday, December 16, 2014 12:27 PM >> > > > > To: users(a)ovirt.org >> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > creating >> > > > > VM from template >> > > > > >> > > > > Hello, >> > > > > >> > > > > I ran into a nasty problem today when creating a new, cloned VM >> from a >> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: >> on the >> > > > node >> > > > > where I started a VM creation job, load skyrocketed and some VMs >> > > > > stopped >> > > > > responding until and after the job failed. Everything recovered >> without >> > > > > intervention, but this obviously shouldn't happen. I have >> attached the >> > > > > relevant vdsm log file. The button to create the VM was pressed >> around >> > > > > 11:17, the first error in the vdsm log is at 11:23:58. >> > > > > >> > > > > The ISO domain is a gluster volume exposed via NFS, the storage >> domain >> > > > for >> > > > > the VM's is also a gluster volume. The underlying filesystem is >> ZFS. >> > > > > The hypervisor nodes are full CentOS 6 installs. >> > > > > >> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log >> file >> > > > > is >> > > > key >> > > > > here. What can I do to prevent this from happening again? Apart >> from >> > > > > not >> > > > > creating new VMs of course :) >> > > > > >> > > > > Tiemen >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > Tiemen Ruiten >> > > > > Systems Engineer >> > > > > R&D Media >> > > > > >> > > > > >> > > > > -- >> > > > > Tiemen Ruiten >> > > > > Systems Engineer >> > > > > R&D Media >> > > > > >> > > > > _______________________________________________ >> > > > > Users mailing list >> > > > > Users(a)ovirt.org >> > > > > http://lists.ovirt.org/mailman/listinfo/users >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Tiemen Ruiten >> > > Systems Engineer >> > > R&D Media >> > > >> > _______________________________________________ >> > Users mailing list >> > Users(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > >> > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > -- Tiemen Ruiten Systems Engineer R&D Media