This is a multipart message in MIME format. ------=_NextPart_000_014F_01D0192E.2E17DC10 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The only thing I can think of would be file hard and soft limits, but I = am no oVirt pro.=20 =20 'no free file handlers in pool' that would make sense to me=E2=80=A6=20 Donny =20 =20 From: Tiemen Ruiten [mailto:t.ruiten@rdmedia.com]=20 Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while = creating VM from template =20 About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 = hyperthreaded cores and 64 GB of RAM each. At the moment I created the = VM, processor load on both nodes was less than 1.=20 =20 On 16 December 2014 at 20:29, Donny Davis <donny@cloudspin.me> wrote: What is the VM load you are running on your servers? -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf = Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while = creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a = template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the = node where I started a VM creation job, load skyrocketed and some VMs = stopped responding until and after the job failed. Everything recovered = without intervention, but this obviously shouldn't happen. I have = attached the relevant vdsm log file. The button to create the VM was = pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain = for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is = key here. What can I do to prevent this from happening again? Apart from = not creating new VMs of course :) Tiemen =20 --=20 Tiemen Ruiten Systems Engineer R&D Media ------=_NextPart_000_014F_01D0192E.2E17DC10 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta = name=3DGenerator content=3D"Microsoft Word 14 (filtered = medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p.MsoAcetate, li.MsoAcetate, div.MsoAcetate {mso-style-priority:99; mso-style-link:"Balloon Text Char"; margin:0in; margin-bottom:.0001pt; font-size:8.0pt; font-family:"Tahoma","sans-serif";} span.EmailStyle17 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} span.BalloonTextChar {mso-style-name:"Balloon Text Char"; mso-style-priority:99; mso-style-link:"Balloon Text"; font-family:"Tahoma","sans-serif";} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif";} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'>The only thing I can think of would be file hard and soft limits, but = I am no oVirt pro. <o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal>'no free file = handlers in pool' that would make sense to me=E2=80=A6 <o:p></o:p></p><p = class=3DMsoNormal>Donny<span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><b><span = style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>= </b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> = Tiemen Ruiten [mailto:t.ruiten@rdmedia.com] <br><b>Sent:</b> Tuesday, = December 16, 2014 12:40 PM<br><b>To:</b> Donny Davis<br><b>Cc:</b> = Users@ovirt.org<br><b>Subject:</b> Re: [ovirt-users] ERROR 'no free file = handlers in pool' while creating VM from = template<o:p></o:p></span></p><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal>About = 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 = hyperthreaded cores and 64 GB of RAM each. At the moment I created the = VM, processor load on both nodes was less than = 1. <o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal>On 16 = December 2014 at 20:29, Donny Davis <<a = href=3D"mailto:donny@cloudspin.me" = target=3D"_blank">donny@cloudspin.me</a>> wrote:<o:p></o:p></p><p = class=3DMsoNormal>What is the VM load you are running on your = servers?<o:p></o:p></p><div><div><p class=3DMsoNormal = style=3D'margin-bottom:12.0pt'><br>-----Original Message-----<br>From: = <a href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a> = [mailto:<a = href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a>] On = Behalf Of Tiemen Ruiten<br>Sent: Tuesday, December 16, 2014 12:27 = PM<br>To: <a = href=3D"mailto:users@ovirt.org">users@ovirt.org</a><br>Subject: = [ovirt-users] ERROR 'no free file handlers in pool' while creating VM = from template<br><br>Hello,<br><br>I ran into a nasty problem today when = creating a new, cloned VM from a template (one virtual 20 GBdisk) on our = two-node oVirt cluster: on the node where I started a VM creation job, = load skyrocketed and some VMs stopped responding until and after the job = failed. Everything recovered without intervention, but this obviously = shouldn't happen. I have attached the relevant vdsm log file. The button = to create the VM was pressed around 11:17, the first error in the vdsm = log is at 11:23:58.<br><br>The ISO domain is a gluster volume exposed = via NFS, the storage domain for the VM's is also a gluster volume. The = underlying filesystem is ZFS.<br>The hypervisor nodes are full CentOS 6 = installs.<br><br>I'm guessing the 'no free file handlers in pool' in the = vdsm log file is key here. What can I do to prevent this from happening = again? Apart from not creating new VMs of course = :)<br><br>Tiemen<br><br><o:p></o:p></p></div></div></div><p = class=3DMsoNormal><br clear=3Dall><o:p></o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p></div><p class=3DMsoNormal>-- = <o:p></o:p></p><div><div><p class=3DMsoNormal>Tiemen Ruiten<br>Systems = Engineer<br>R&D = Media<o:p></o:p></p></div></div></div></div></body></html> ------=_NextPart_000_014F_01D0192E.2E17DC10--