ERROR 'no free file handlers in pool' while creating VM from template
Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen
What is the VM load you are running on your servers? -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1. On 16 December 2014 at 20:29, Donny Davis <donny@cloudspin.me> wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
-- Tiemen Ruiten Systems Engineer R&D Media
This is a multipart message in MIME format. ------=_NextPart_000_014F_01D0192E.2E17DC10 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The only thing I can think of would be file hard and soft limits, but I = am no oVirt pro.=20 =20 'no free file handlers in pool' that would make sense to me=E2=80=A6=20 Donny =20 =20 From: Tiemen Ruiten [mailto:t.ruiten@rdmedia.com]=20 Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while = creating VM from template =20 About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 = hyperthreaded cores and 64 GB of RAM each. At the moment I created the = VM, processor load on both nodes was less than 1.=20 =20 On 16 December 2014 at 20:29, Donny Davis <donny@cloudspin.me> wrote: What is the VM load you are running on your servers? -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf = Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while = creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a = template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the = node where I started a VM creation job, load skyrocketed and some VMs = stopped responding until and after the job failed. Everything recovered = without intervention, but this obviously shouldn't happen. I have = attached the relevant vdsm log file. The button to create the VM was = pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain = for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is = key here. What can I do to prevent this from happening again? Apart from = not creating new VMs of course :) Tiemen =20 --=20 Tiemen Ruiten Systems Engineer R&D Media ------=_NextPart_000_014F_01D0192E.2E17DC10 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta = name=3DGenerator content=3D"Microsoft Word 14 (filtered = medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p.MsoAcetate, li.MsoAcetate, div.MsoAcetate {mso-style-priority:99; mso-style-link:"Balloon Text Char"; margin:0in; margin-bottom:.0001pt; font-size:8.0pt; font-family:"Tahoma","sans-serif";} span.EmailStyle17 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} span.BalloonTextChar {mso-style-name:"Balloon Text Char"; mso-style-priority:99; mso-style-link:"Balloon Text"; font-family:"Tahoma","sans-serif";} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif";} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'>The only thing I can think of would be file hard and soft limits, but = I am no oVirt pro. <o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal>'no free file = handlers in pool' that would make sense to me=E2=80=A6 <o:p></o:p></p><p = class=3DMsoNormal>Donny<span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><b><span = style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>= </b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> = Tiemen Ruiten [mailto:t.ruiten@rdmedia.com] <br><b>Sent:</b> Tuesday, = December 16, 2014 12:40 PM<br><b>To:</b> Donny Davis<br><b>Cc:</b> = Users@ovirt.org<br><b>Subject:</b> Re: [ovirt-users] ERROR 'no free file = handlers in pool' while creating VM from = template<o:p></o:p></span></p><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal>About = 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 = hyperthreaded cores and 64 GB of RAM each. At the moment I created the = VM, processor load on both nodes was less than = 1. <o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal>On 16 = December 2014 at 20:29, Donny Davis <<a = href=3D"mailto:donny@cloudspin.me" = target=3D"_blank">donny@cloudspin.me</a>> wrote:<o:p></o:p></p><p = class=3DMsoNormal>What is the VM load you are running on your = servers?<o:p></o:p></p><div><div><p class=3DMsoNormal = style=3D'margin-bottom:12.0pt'><br>-----Original Message-----<br>From: = <a href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a> = [mailto:<a = href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a>] On = Behalf Of Tiemen Ruiten<br>Sent: Tuesday, December 16, 2014 12:27 = PM<br>To: <a = href=3D"mailto:users@ovirt.org">users@ovirt.org</a><br>Subject: = [ovirt-users] ERROR 'no free file handlers in pool' while creating VM = from template<br><br>Hello,<br><br>I ran into a nasty problem today when = creating a new, cloned VM from a template (one virtual 20 GBdisk) on our = two-node oVirt cluster: on the node where I started a VM creation job, = load skyrocketed and some VMs stopped responding until and after the job = failed. Everything recovered without intervention, but this obviously = shouldn't happen. I have attached the relevant vdsm log file. The button = to create the VM was pressed around 11:17, the first error in the vdsm = log is at 11:23:58.<br><br>The ISO domain is a gluster volume exposed = via NFS, the storage domain for the VM's is also a gluster volume. The = underlying filesystem is ZFS.<br>The hypervisor nodes are full CentOS 6 = installs.<br><br>I'm guessing the 'no free file handlers in pool' in the = vdsm log file is key here. What can I do to prevent this from happening = again? Apart from not creating new VMs of course = :)<br><br>Tiemen<br><br><o:p></o:p></p></div></div></div><p = class=3DMsoNormal><br clear=3Dall><o:p></o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p></div><p class=3DMsoNormal>-- = <o:p></o:p></p><div><div><p class=3DMsoNormal>Tiemen Ruiten<br>Systems = Engineer<br>R&D = Media<o:p></o:p></p></div></div></div></div></body></html> ------=_NextPart_000_014F_01D0192E.2E17DC10--
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM? On 16 December 2014 at 20:45, Donny Davis <donny@cloudspin.me> wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
*From:* Tiemen Ruiten [mailto:t.ruiten@rdmedia.com] *Sent:* Tuesday, December 16, 2014 12:40 PM *To:* Donny Davis *Cc:* Users@ovirt.org *Subject:* Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis <donny@cloudspin.me> wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system: /usr/lib/python2.6/site-packages/vdsm/config.py The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much. But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs? thanks, Yeela ----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255 On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs?
thanks, Yeela
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs?
thanks, Yeela
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this? ----- Original Message -----
From: "Yeela Kaplan" <ykaplan@redhat.com> To: "Tiemen Ruiten" <t.ruiten@rdmedia.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:30:52 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs?
thanks, Yeela
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hello, I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job: 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan@redhat.com> wrote:
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this?
From: "Yeela Kaplan" <ykaplan@redhat.com> To: "Tiemen Ruiten" <t.ruiten@rdmedia.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:30:52 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so
you
can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of
details of your setup and logs?
thanks, Yeela
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created
VM,
processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on
node
where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached
----- Original Message ----- the the the the
relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain. On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten@rdmedia.com> wrote:
Hello,
I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job:
2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.
This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?
On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan@redhat.com> wrote:
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this?
From: "Yeela Kaplan" <ykaplan@redhat.com> To: "Tiemen Ruiten" <t.ruiten@rdmedia.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:30:52 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com>
wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm
config
file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of
details of your setup and logs?
thanks, Yeela
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 10:53:39 AM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the
----- Original Message ----- the proper way
to change ulimits for VDSM?
On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me > wrote:
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro.
'no free file handlers in pool' that would make sense to me…
Donny
From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1.
On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me > wrote:
What is the VM load you are running on your servers?
-----Original Message----- From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello,
I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58.
The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs.
I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :)
Tiemen
--
Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
Yeela, any chance you could shed any light on this? On 5 January 2015 at 15:46, Tiemen Ruiten <t.ruiten@rdmedia.com> wrote:
Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain.
On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten@rdmedia.com> wrote:
Hello,
I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job:
2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.
This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?
On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan@redhat.com> wrote:
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this?
From: "Yeela Kaplan" <ykaplan@redhat.com> To: "Tiemen Ruiten" <t.ruiten@rdmedia.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:30:52 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com>
wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm
config
file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs?
thanks, Yeela
----- Original Message ----- > From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> > To: "Users@ovirt.org" <users@ovirt.org> > Sent: Wednesday, December 17, 2014 10:53:39 AM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template > > Would this be limits for the vdsm process? Then what is the
> to > change ulimits for VDSM? > > On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me wrote: > > > > > The only thing I can think of would be file hard and soft
----- Original Message ----- proper way limits, but I
am no > oVirt pro. > > > > 'no free file handlers in pool' that would make sense to me… > > Donny > > > > > > From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] > Sent: Tuesday, December 16, 2014 12:40 PM > To: Donny Davis > Cc: Users@ovirt.org > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, > processor load on both nodes was less than 1. > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me wrote: > > What is the VM load you are running on your servers? > > > > -----Original Message----- > From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of > Tiemen Ruiten > Sent: Tuesday, December 16, 2014 12:27 PM > To: users@ovirt.org > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating > VM from template > > Hello, > > I ran into a nasty problem today when creating a new, cloned VM from a > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node > where I started a VM creation job, load skyrocketed and some VMs > stopped > responding until and after the job failed. Everything recovered without > intervention, but this obviously shouldn't happen. I have attached the > relevant vdsm log file. The button to create the VM was pressed around > 11:17, the first error in the vdsm log is at 11:23:58. > > The ISO domain is a gluster volume exposed via NFS, the storage domain for > the VM's is also a gluster volume. The underlying filesystem is ZFS. > The hypervisor nodes are full CentOS 6 installs. > > I'm guessing the 'no free file handlers in pool' in the vdsm log file > is key > here. What can I do to prevent this from happening again? Apart from > not > creating new VMs of course :) > > Tiemen > > > > > > > > > > -- > > > Tiemen Ruiten > Systems Engineer > R&D Media > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
After creating a default disk profile, I can add new disks and create VMs without issues. I have updated the bug I reported as a duplicate of 1145241. https://bugzilla.redhat.com/show_bug.cgi?id=1175255 On 5 January 2015 at 15:46, Tiemen Ruiten <t.ruiten@rdmedia.com> wrote:
Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain.
On 5 January 2015 at 15:45, Tiemen Ruiten <t.ruiten@rdmedia.com> wrote:
Hello,
I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job:
2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.id WHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.
This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>?
On 17 December 2014 at 12:34, Yeela Kaplan <ykaplan@redhat.com> wrote:
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this?
From: "Yeela Kaplan" <ykaplan@redhat.com> To: "Tiemen Ruiten" <t.ruiten@rdmedia.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:30:52 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
----- Original Message -----
From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: "Users@ovirt.org" <users@ovirt.org> Sent: Wednesday, December 17, 2014 1:22:59 PM Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255
Thank you and I forgot to mention that you have to restart vdsmd so the change will apply...
On 17 December 2014 at 11:48, Yeela Kaplan <ykaplan@redhat.com>
wrote:
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm
config
file, which is supposed to be under the following directory in your system:
/usr/lib/python2.6/site-packages/vdsm/config.py
The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much.
But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs?
thanks, Yeela
----- Original Message ----- > From: "Tiemen Ruiten" <t.ruiten@rdmedia.com> > To: "Users@ovirt.org" <users@ovirt.org> > Sent: Wednesday, December 17, 2014 10:53:39 AM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template > > Would this be limits for the vdsm process? Then what is the
> to > change ulimits for VDSM? > > On 16 December 2014 at 20:45, Donny Davis < donny@cloudspin.me wrote: > > > > > The only thing I can think of would be file hard and soft
----- Original Message ----- proper way limits, but I
am no > oVirt pro. > > > > 'no free file handlers in pool' that would make sense to me… > > Donny > > > > > > From: Tiemen Ruiten [mailto: t.ruiten@rdmedia.com ] > Sent: Tuesday, December 16, 2014 12:40 PM > To: Donny Davis > Cc: Users@ovirt.org > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, > processor load on both nodes was less than 1. > > > > > > On 16 December 2014 at 20:29, Donny Davis < donny@cloudspin.me wrote: > > What is the VM load you are running on your servers? > > > > -----Original Message----- > From: users-bounces@ovirt.org [mailto: users-bounces@ovirt.org ] On Behalf Of > Tiemen Ruiten > Sent: Tuesday, December 16, 2014 12:27 PM > To: users@ovirt.org > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating > VM from template > > Hello, > > I ran into a nasty problem today when creating a new, cloned VM from a > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node > where I started a VM creation job, load skyrocketed and some VMs > stopped > responding until and after the job failed. Everything recovered without > intervention, but this obviously shouldn't happen. I have attached the > relevant vdsm log file. The button to create the VM was pressed around > 11:17, the first error in the vdsm log is at 11:23:58. > > The ISO domain is a gluster volume exposed via NFS, the storage domain for > the VM's is also a gluster volume. The underlying filesystem is ZFS. > The hypervisor nodes are full CentOS 6 installs. > > I'm guessing the 'no free file handlers in pool' in the vdsm log file > is key > here. What can I do to prevent this from happening again? Apart from > not > creating new VMs of course :) > > Tiemen > > > > > > > > > > -- > > > Tiemen Ruiten > Systems Engineer > R&D Media > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
-- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
-- Tiemen Ruiten Systems Engineer R&D Media
participants (3)
-
Donny Davis -
Tiemen Ruiten -
Yeela Kaplan