<div dir="ltr">Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain.</div><div class="gmail_extra"><br><div class="gmail_quote">On 5 January 2015 at 15:45, Tiemen Ruiten <span dir="ltr"><<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job:</div><div><br></div><div><div>2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id=<a href="http://storage_pool_with_storage_domain.id" target="_blank">storage_pool_with_storage_domain.id</a> WHERE <a href="http://storage_pool_with_storage_domain.name" target="_blank">storage_pool_with_storage_domain.name</a> LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id=<a href="http://storage_pool_with_storage_domain.id" target="_blank">storage_pool_with_storage_domain.id</a> WHERE <a href="http://storage_pool_with_storage_domain.name" target="_blank">storage_pool_with_storage_domain.name</a> LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id=<a href="http://storage_pool_with_storage_domain.id" target="_blank">storage_pool_with_storage_domain.id</a> WHERE <a href="http://storage_pool_with_storage_domain.name" target="_blank">storage_pool_with_storage_domain.name</a> LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.</div></div><div><br></div><div>This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1145241" target="_blank">BZ</a>?</div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 17 December 2014 at 12:34, Yeela Kaplan <span dir="ltr"><<a href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just another thought,<br>
from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers<br>
and eventually a timeout occurs.<br>
Adam, do you have any idea about this?<br>
<span><br>
----- Original Message -----<br>
> From: "Yeela Kaplan" <<a href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>><br>
> To: "Tiemen Ruiten" <<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>><br>
> Cc: "<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>" <<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
</span><div><div>> Sent: Wednesday, December 17, 2014 1:30:52 PM<br>
> Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Tiemen Ruiten" <<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>><br>
> > To: "Yeela Kaplan" <<a href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>><br>
> > Cc: "<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>" <<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
> > Sent: Wednesday, December 17, 2014 1:22:59 PM<br>
> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while<br>
> > creating VM from template<br>
> ><br>
> > Thank you, I will try to increase to 20 and see what happens. Bug is filed:<br>
> > <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1175255" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1175255</a><br>
><br>
> Thank you<br>
> and I forgot to mention that you have to restart vdsmd so the change will<br>
> apply...<br>
><br>
><br>
> ><br>
> > On 17 December 2014 at 11:48, Yeela Kaplan <<a href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>> wrote:<br>
> > ><br>
> > > Hi,<br>
> > > You are right, the problem is with the file handlers.<br>
> > > You can increase the number of handlers in pool using the vdsm config<br>
> > > file, which is supposed to be under the following directory in your<br>
> > > system:<br>
> > ><br>
> > > /usr/lib/python2.6/site-packages/vdsm/config.py<br>
> > ><br>
> > > The default value for 'process_pool_max_slots_per_domain' is 10, so you<br>
> > > can increase it by a bit, but not too much.<br>
> > ><br>
> > > But I suspect the problem is in a larger scale, and this is only a<br>
> > > temporary relief for your system and this needs much more attention and a<br>
> > > proper fix.<br>
> > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the<br>
> > > details of your setup and logs?<br>
> > ><br>
> > > thanks,<br>
> > > Yeela<br>
> > ><br>
> > > ----- Original Message -----<br>
> > > > From: "Tiemen Ruiten" <<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>><br>
> > > > To: "<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>" <<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
> > > > Sent: Wednesday, December 17, 2014 10:53:39 AM<br>
> > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while<br>
> > > creating VM from template<br>
> > > ><br>
> > > > Would this be limits for the vdsm process? Then what is the proper way<br>
> > > > to<br>
> > > > change ulimits for VDSM?<br>
> > > ><br>
> > > > On 16 December 2014 at 20:45, Donny Davis < <a href="mailto:donny@cloudspin.me" target="_blank">donny@cloudspin.me</a> > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > The only thing I can think of would be file hard and soft limits, but I<br>
> > > am no<br>
> > > > oVirt pro.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > 'no free file handlers in pool' that would make sense to me…<br>
> > > ><br>
> > > > Donny<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > From: Tiemen Ruiten [mailto: <a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a> ]<br>
> > > > Sent: Tuesday, December 16, 2014 12:40 PM<br>
> > > > To: Donny Davis<br>
> > > > Cc: <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while<br>
> > > > creating VM from template<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16<br>
> > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the<br>
> > > VM,<br>
> > > > processor load on both nodes was less than 1.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On 16 December 2014 at 20:29, Donny Davis < <a href="mailto:donny@cloudspin.me" target="_blank">donny@cloudspin.me</a> > wrote:<br>
> > > ><br>
> > > > What is the VM load you are running on your servers?<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > -----Original Message-----<br>
> > > > From: <a href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a> [mailto: <a href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a> ] On<br>
> > > Behalf Of<br>
> > > > Tiemen Ruiten<br>
> > > > Sent: Tuesday, December 16, 2014 12:27 PM<br>
> > > > To: <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
> > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while<br>
> > > creating<br>
> > > > VM from template<br>
> > > ><br>
> > > > Hello,<br>
> > > ><br>
> > > > I ran into a nasty problem today when creating a new, cloned VM from a<br>
> > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the<br>
> > > node<br>
> > > > where I started a VM creation job, load skyrocketed and some VMs<br>
> > > > stopped<br>
> > > > responding until and after the job failed. Everything recovered without<br>
> > > > intervention, but this obviously shouldn't happen. I have attached the<br>
> > > > relevant vdsm log file. The button to create the VM was pressed around<br>
> > > > 11:17, the first error in the vdsm log is at 11:23:58.<br>
> > > ><br>
> > > > The ISO domain is a gluster volume exposed via NFS, the storage domain<br>
> > > for<br>
> > > > the VM's is also a gluster volume. The underlying filesystem is ZFS.<br>
> > > > The hypervisor nodes are full CentOS 6 installs.<br>
> > > ><br>
> > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file<br>
> > > > is<br>
> > > key<br>
> > > > here. What can I do to prevent this from happening again? Apart from<br>
> > > > not<br>
> > > > creating new VMs of course :)<br>
> > > ><br>
> > > > Tiemen<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > ><br>
> > > ><br>
> > > > Tiemen Ruiten<br>
> > > > Systems Engineer<br>
> > > > R&D Media<br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Tiemen Ruiten<br>
> > > > Systems Engineer<br>
> > > > R&D Media<br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Users mailing list<br>
> > > > <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> > > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > > ><br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Tiemen Ruiten<br>
> > Systems Engineer<br>
> > R&D Media<br>
> ><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>