From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM? On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me > wrote: The only thing I can think of would be file hard and soft limits, but I am no oVirt pro. 'no free file handlers in pool' that would make sense to me… Donny From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users(a)ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1. On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me > wrote: What is the VM load you are running on your servers? -----Original Message----- From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users(a)ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen -- Tiemen Ruiten Systems Engineer R&D Media -- Tiemen Ruiten Systems Engineer R&D Media _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255

On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > > Hi, > You are right, the problem is with the file handlers. > You can increase the number of handlers in pool using the vdsm config > file, which is supposed to be under the following directory in your system: > > /usr/lib/python2.6/site-packages/vdsm/config.py > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > can increase it by a bit, but not too much. > > But I suspect the problem is in a larger scale, and this is only a > temporary relief for your system and this needs much more attention and a > proper fix. > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > details of your setup and logs? > > thanks, > Yeela > > ----- Original Message ----- > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > Would this be limits for the vdsm process? Then what is the proper way to > > change ulimits for VDSM? > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me > wrote: > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > am no > > oVirt pro. > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > Donny > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] > > Sent: Tuesday, December 16, 2014 12:40 PM > > To: Donny Davis > > Cc: Users(a)ovirt.org > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > VM, > > processor load on both nodes was less than 1. > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me > wrote: > > > > What is the VM load you are running on your servers? > > > > > > > > -----Original Message----- > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org ] On > Behalf Of > > Tiemen Ruiten > > Sent: Tuesday, December 16, 2014 12:27 PM > > To: users(a)ovirt.org > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > creating > > VM from template > > > > Hello, > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > node > > where I started a VM creation job, load skyrocketed and some VMs stopped > > responding until and after the job failed. Everything recovered without > > intervention, but this obviously shouldn't happen. I have attached the > > relevant vdsm log file. The button to create the VM was pressed around > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > for > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > The hypervisor nodes are full CentOS 6 installs. > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is > key > > here. What can I do to prevent this from happening again? Apart from not > > creating new VMs of course :) > > > > Tiemen > > > > > > > > > > > > > > > > > > > > -- > > > > > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > > > > -- > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > -- Tiemen Ruiten Systems Engineer R&D Media

Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this? ----- Original Message ----- > From: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > To: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > Sent: Wednesday, December 17, 2014 1:30:52 PM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template > > > > ----- Original Message ----- > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > Thank you, I will try to increase to 20 and see what happens. Bug is filed: > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > Thank you > and I forgot to mention that you have to restart vdsmd so the change will > apply... > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > > > > > > Hi, > > > You are right, the problem is with the file handlers. > > > You can increase the number of handlers in pool using the vdsm config > > > file, which is supposed to be under the following directory in your > > > system: > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > > > can increase it by a bit, but not too much. > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > temporary relief for your system and this needs much more attention and a > > > proper fix. > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > > > details of your setup and logs? > > > > > > thanks, > > > Yeela > > > > > > ----- Original Message ----- > > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; > > > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > > > Would this be limits for the vdsm process? Then what is the proper way > > > > to > > > > change ulimits for VDSM? > > > > > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me > wrote: > > > > > > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > > > am no > > > > oVirt pro. > > > > > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > > > Donny > > > > > > > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] > > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > > To: Donny Davis > > > > Cc: Users(a)ovirt.org > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > > creating VM from template > > > > > > > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > > > VM, > > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me > wrote: > > > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org ] On > > > Behalf Of > > > > Tiemen Ruiten > > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > > To: users(a)ovirt.org > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating > > > > VM from template > > > > > > > > Hello, > > > > > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > > > node > > > > where I started a VM creation job, load skyrocketed and some VMs > > > > stopped > > > > responding until and after the job failed. Everything recovered without > > > > intervention, but this obviously shouldn't happen. I have attached the > > > > relevant vdsm log file. The button to create the VM was pressed around > > > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > > > for > > > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > > > The hypervisor nodes are full CentOS 6 installs. > > > > > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file > > > > is > > > key > > > > here. What can I do to prevent this from happening again? Apart from > > > > not > > > > creating new VMs of course :) > > > > > > > > Tiemen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > > > > > -- > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > _______________________________________________ > > > > Users mailing list > > > > Users(a)ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > > > > > -- > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain. On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten(a)rdmedia.com&gt; wrote: > Hello, > > I just tried creating a thin-provisioned disk and the same problem (load > spike, non responsive VMs) occurred. I may have an idea what is going wrong > now. The following is an excerpt from the engine log after starting the > disk create job: > > 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] > (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is > StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM > vds_groups_view WHERE ( vds_group_id IN (SELECT distinct > vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain > LEFT OUTER JOIN storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. : > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet.: > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.id WHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. > > This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. > There's also a 3.5 cluster managed by the same engine. Could this be > related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? > > > > On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan(a)redhat.com&gt; wrote: > >> Just another thought, >> from looking at the vdsm logs it looks like there are too many calls to >> getVolumeSize that are eating up all handlers >> and eventually a timeout occurs. >> Adam, do you have any idea about this? >> >> ----- Original Message ----- >> > From: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; >> > To: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > Sent: Wednesday, December 17, 2014 1:30:52 PM >> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >> creating VM from template >> > >> > >> > >> > ----- Original Message ----- >> > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > > To: "Yeela Kaplan" <ykaplan(a)redhat.com&gt; >> > > Cc: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > creating VM from template >> > > >> > > Thank you, I will try to increase to 20 and see what happens. Bug is >> filed: >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >> > >> > Thank you >> > and I forgot to mention that you have to restart vdsmd so the change >> will >> > apply... >> > >> > >> > > >> > > On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan(a)redhat.com&gt; >> wrote: >> > > > >> > > > Hi, >> > > > You are right, the problem is with the file handlers. >> > > > You can increase the number of handlers in pool using the vdsm >> config >> > > > file, which is supposed to be under the following directory in your >> > > > system: >> > > > >> > > > /usr/lib/python2.6/site-packages/vdsm/config.py >> > > > >> > > > The default value for 'process_pool_max_slots_per_domain' is 10, >> so you >> > > > can increase it by a bit, but not too much. >> > > > >> > > > But I suspect the problem is in a larger scale, and this is only a >> > > > temporary relief for your system and this needs much more >> attention and a >> > > > proper fix. >> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all >> of the >> > > > details of your setup and logs? >> > > > >> > > > thanks, >> > > > Yeela >> > > > >> > > > ----- Original Message ----- >> > > > > From: "Tiemen Ruiten" <t.ruiten(a)rdmedia.com&gt; >> > > > > To: &quot;Users(a)ovirt.org&quot; <users(a)ovirt.org&gt; >> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM >> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > creating VM from template >> > > > > >> > > > > Would this be limits for the vdsm process? Then what is the >> proper way >> > > > > to >> > > > > change ulimits for VDSM? >> > > > > >> > > > > On 16 December 2014 at 20:45, Donny Davis < donny(a)cloudspin.me >> > wrote: >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > The only thing I can think of would be file hard and soft >> limits, but I >> > > > am no >> > > > > oVirt pro. >> > > > > >> > > > > >> > > > > >> > > > > 'no free file handlers in pool' that would make sense to me… >> > > > > >> > > > > Donny >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > From: Tiemen Ruiten [mailto: t.ruiten(a)rdmedia.com ] >> > > > > Sent: Tuesday, December 16, 2014 12:40 PM >> > > > > To: Donny Davis >> > > > > Cc: Users(a)ovirt.org >> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > > creating VM from template >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with >> 16 >> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I >> created the >> > > > VM, >> > > > > processor load on both nodes was less than 1. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On 16 December 2014 at 20:29, Donny Davis < donny(a)cloudspin.me >> > wrote: >> > > > > >> > > > > What is the VM load you are running on your servers? >> > > > > >> > > > > >> > > > > >> > > > > -----Original Message----- >> > > > > From: users-bounces(a)ovirt.org [mailto: users-bounces(a)ovirt.org >> ] On >> > > > Behalf Of >> > > > > Tiemen Ruiten >> > > > > Sent: Tuesday, December 16, 2014 12:27 PM >> > > > > To: users(a)ovirt.org >> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' >> while >> > > > creating >> > > > > VM from template >> > > > > >> > > > > Hello, >> > > > > >> > > > > I ran into a nasty problem today when creating a new, cloned VM >> from a >> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: >> on the >> > > > node >> > > > > where I started a VM creation job, load skyrocketed and some VMs >> > > > > stopped >> > > > > responding until and after the job failed. Everything recovered >> without >> > > > > intervention, but this obviously shouldn't happen. I have >> attached the >> > > > > relevant vdsm log file. The button to create the VM was pressed >> around >> > > > > 11:17, the first error in the vdsm log is at 11:23:58. >> > > > > >> > > > > The ISO domain is a gluster volume exposed via NFS, the storage >> domain >> > > > for >> > > > > the VM's is also a gluster volume. The underlying filesystem is >> ZFS. >> > > > > The hypervisor nodes are full CentOS 6 installs. >> > > > > >> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log >> file >> > > > > is >> > > > key >> > > > > here. What can I do to prevent this from happening again? Apart >> from >> > > > > not >> > > > > creating new VMs of course :) >> > > > > >> > > > > Tiemen >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > Tiemen Ruiten >> > > > > Systems Engineer >> > > > > R&D Media >> > > > > >> > > > > >> > > > > -- >> > > > > Tiemen Ruiten >> > > > > Systems Engineer >> > > > > R&D Media >> > > > > >> > > > > _______________________________________________ >> > > > > Users mailing list >> > > > > Users(a)ovirt.org >> > > > > http://lists.ovirt.org/mailman/listinfo/users >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Tiemen Ruiten >> > > Systems Engineer >> > > R&D Media >> > > >> > _______________________________________________ >> > Users mailing list >> > Users(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > >> > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > -- Tiemen Ruiten Systems Engineer R&D Media