This is a multi-part message in MIME format. --------------E0950BFE735406CA7670182D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead. Thanks to everyone involved, Andrea On 13/02/2017 13:24, Nir Soffer wrote:
On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:
Hi Nir,
the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.
Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs)
Andrea
On 13/02/2017 10:13, Nir Soffer wrote:
On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:
Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.
Thanks, Andrea
I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io.
Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server?
Nir
On 12/02/2017 18:27, Andrea Ghelardi wrote:
Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG
-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com <mailto:a.ghelardi@iontrading.com>>; users@ovirt.org <mailto:users@ovirt.org> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
Hi Andrea,
thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.
Thanks, Andrea
On 10/02/2017 17:17, Andrea Ghelardi wrote:
Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5
All HE hosts were hammering the SAN with 2~4k IOPS each
Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?
Cheers AG
-----Original Message----- From: users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> [mailto:users-bounces@ovirt.org <mailto:users-bounces@ovirt.org>] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org <mailto:users@ovirt.org> Subject: [ovirt-users] high number of IOPS from ovirt hosts
Hi,
We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?
Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.
Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
--------------E0950BFE735406CA7670182D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Hi,</p> <p>just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead.</p> <p>Thanks to everyone involved,<br> Andrea<br> </p> <br> <div class="moz-cite-prefix">On 13/02/2017 13:24, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyytbdFiO1sxofxZ8NxCsiWjgb3BXyonPo4Ah3UM1aDPtGw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Hi Nir,</p> <p>the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.</p> </div> </blockquote> <div><br> </div> <div>Andrea, can you send also logs from ovirt-hosted-engine-ha?</div> <div>(I'm not sure about the name of the service or the location of the logs)</div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"><span class="HOEnZb"><font color="#888888"> <p>Andrea<br> </p> </font></span> <div> <div class="h5"> <br> <div class="m_1481480387959180211moz-cite-prefix">On 13/02/2017 10:13, Nir Soffer wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a><wbr>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the heads up, I'll take a look at Mark's thread.<br> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.<br> <br> Thanks,<br> Andrea</blockquote> <div><br> </div> <div>I think we already have hosted engine bug about doing to many calls to vdsm</div> <div>that cause reloads of lvm data; this can cause increased io.</div> <div><br> </div> Can you attach vdsm and hosted engine agent logs showing the timeframe when <div>you have lot of iops on the server?</div> <div><br> </div> <div>Nir<br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div class="m_1481480387959180211gmail-HOEnZb"> <div class="m_1481480387959180211gmail-h5"><br> <br> <br> On 12/02/2017 18:27, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Andrea,<br> I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well.<br> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities.<br> How many storages are you presenting to how many clusters?<br> Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000<br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: Andrea Fagiani [mailto:<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobil<wbr>iare.it</a>]<br> Sent: Friday, February 10, 2017 5:56 PM<br> To: Andrea Ghelardi <<a moz-do-not-send="true" href="mailto:a.ghelardi@iontrading.com" target="_blank">a.ghelardi@iontrading.com</a>>; <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi Andrea,<br> <br> thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread.<br> We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.<br> <br> Thanks,<br> Andrea<br> <br> <br> On 10/02/2017 17:17, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hello Andrea,<br> I saw that happening on my HE hosts connected to a COMPELLENT SC040<br> when I was running Ovirt 3.5<br> <br> All HE hosts were hammering the SAN with 2~4k IOPS each<br> <br> Upgrading to 4.0.5 solved my case.<br> Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?<br> <br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: <a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a> [mailto:<a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.or<wbr>g</a>] On<br> Behalf Of Andrea Fagiani<br> Sent: Friday, February 10, 2017 12:47 PM<br> To: <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi,<br> <br> We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x).<br> Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?<br> <br> Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.<br> <br> Thanks,<br> Andrea<br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </blockquote> </blockquote> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </div> </div> </blockquote> </div> <br> </div> </div> </div> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------E0950BFE735406CA7670182D--