6:15 a.m.
This is a multi-part message in MIME format.
--------------E0950BFE735406CA7670182D
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hi,
just a quick update on the issue. The high amount of IOPS was indeed due
to having the hosted-engine on the EQL storage; we worked around the
issue by moving the hosted-engine to a gluster storage instead.
Thanks to everyone involved,
Andrea
On 13/02/2017 13:24, Nir Soffer wrote:
On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani
<andrea.fagiani(a)immobiliare.it <mailto:andrea.fagiani@immobiliare.it>>
wrote:
Hi Nir,
the amount of IOPS is pretty much constant, attaching the logs
from the last hour or so.
Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of the logs)
Andrea
On 13/02/2017 10:13, Nir Soffer wrote:
> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
> <andrea.fagiani(a)immobiliare.it
> <mailto:andrea.fagiani@immobiliare.it>> wrote:
>
> Thanks for the heads up, I'll take a look at Mark's thread.
> We are presenting 2 storage LUNs to a single ovirt cluster
> and yes, all hosts are hosted engine hosts.
>
> Thanks,
> Andrea
>
>
> I think we already have hosted engine bug about doing to many
> calls to vdsm
> that cause reloads of lvm data; this can cause increased io.
>
> Can you attach vdsm and hosted engine agent logs showing the
> timeframe when
> you have lot of iops on the server?
>
> Nir
>
>
>
>
> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>
> Andrea,
> I don't want to point any fingers but Mark Greenall faced
> some issues with Ovirt 4.06/4.1 linked to equallogic as well.
> You may want to search posts in this ML with subject "
> Ovirt 4.0.6 guests 'Not Responding' " to see if you find
> any similarities.
> How many storages are you presenting to how many clusters?
> Also, are your nodes all Hosted Engine Hosts? In my old
> configuration, normal hosts read IOPS ranged from 50 to
> 500 while HE ones peaked even ~3000
> Cheers
> AG
>
>
> -----Original Message-----
> From: Andrea Fagiani
> [mailto:andrea.fagiani@immobiliare.it
> <mailto:andrea.fagiani@immobiliare.it>]
> Sent: Friday, February 10, 2017 5:56 PM
> To: Andrea Ghelardi <a.ghelardi(a)iontrading.com
> <mailto:a.ghelardi@iontrading.com>>; users(a)ovirt.org
> <mailto:users@ovirt.org>
> Subject: Re: [ovirt-users] high number of IOPS from ovirt
> hosts
>
> Hi Andrea,
>
> thanks for the insight; no I wasn't able to pinpoint the
> issue to a single host, the load seems to be evenly spread.
> We have another cluster running 4.0.6 on a compellent as
> well and we haven't experienced any similar issues.
>
> Thanks,
> Andrea
>
>
> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>
> Hello Andrea,
> I saw that happening on my HE hosts connected to a
> COMPELLENT SC040
> when I was running Ovirt 3.5
>
> All HE hosts were hammering the SAN with 2~4k IOPS each
>
> Upgrading to 4.0.5 solved my case.
> Are you able to check if IOPS are generated from a
> particular host or rather spread among all nodes?
>
> Cheers
> AG
>
>
> -----Original Message-----
> From: users-bounces(a)ovirt.org
> <mailto:users-bounces@ovirt.org>
> [mailto:users-bounces@ovirt.org
> <mailto:users-bounces@ovirt.org>] On
> Behalf Of Andrea Fagiani
> Sent: Friday, February 10, 2017 12:47 PM
> To: users(a)ovirt.org <mailto:users@ovirt.org>
> Subject: [ovirt-users] high number of IOPS from ovirt
> hosts
>
> Hi,
>
> We've got a 8-host cluster running oVirt 4.1; each
> host is connected via iscsi to a EQL 4100 storage (2x).
> Most of the VMs use barely any IOPS, however on our
> storage controller we've noticed a constant steam of
> reads, about 2000 IOPS; any idea what might cause the
> ovirt hosts to hammer on the storage so much?
>
> Looking at iotop, it seems that systemd-udevd is
> being activated every few seconds (and its IO% is
> usually fairly high) so I assume this might be
> somehow related.
>
> Thanks,
> Andrea
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
--------------E0950BFE735406CA7670182D
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi,</p>
<p>just a quick update on the issue. The high amount of IOPS was
indeed due to having the hosted-engine on the EQL storage; we
worked around the issue by moving the hosted-engine to a gluster
storage instead.</p>
<p>Thanks to everyone involved,<br>
Andrea<br>
</p>
<br>
<div class="moz-cite-prefix">On 13/02/2017 13:24, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyytbdFiO1sxofxZ8NxCsiWjgb3BXyonPo4Ah3UM1aDPtGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Feb 13, 2017 at 12:32 PM,
Andrea Fagiani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi Nir,</p>
<p>the amount of IOPS is pretty much constant, attaching
the logs from the last hour or so.</p>
</div>
</blockquote>
<div><br>
</div>
<div>Andrea, can you send also logs from
ovirt-hosted-engine-ha?</div>
<div>(I'm not sure about the name of the service or the
location of the logs)</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span
class="HOEnZb"><font
color="#888888">
<p>Andrea<br>
</p>
</font></span>
<div>
<div class="h5"> <br>
<div class="m_1481480387959180211moz-cite-prefix">On
13/02/2017 10:13, Nir Soffer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Feb 13,
2017 at 11:05 AM, Andrea Fagiani <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a><wbr>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Thanks
for the heads up, I'll take a look at
Mark's thread.<br>
We are presenting 2 storage LUNs to a
single ovirt cluster and yes, all hosts
are hosted engine hosts.<br>
<br>
Thanks,<br>
Andrea</blockquote>
<div><br>
</div>
<div>I think we already have hosted engine
bug about doing to many calls to vdsm</div>
<div>that cause reloads of lvm data; this
can cause increased io.</div>
<div><br>
</div>
Can you attach vdsm and hosted engine
agent logs showing the timeframe when
<div>you have lot of iops on the server?</div>
<div><br>
</div>
<div>Nir<br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
class="m_1481480387959180211gmail-HOEnZb">
<div
class="m_1481480387959180211gmail-h5"><br>
<br>
<br>
On 12/02/2017 18:27, Andrea Ghelardi
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Andrea,<br>
I don't want to point any fingers
but Mark Greenall faced some
issues with Ovirt 4.06/4.1 linked
to equallogic as well.<br>
You may want to search posts in
this ML with subject " Ovirt 4.0.6
guests 'Not Responding' " to see
if you find any similarities.<br>
How many storages are you
presenting to how many clusters?<br>
Also, are your nodes all Hosted
Engine Hosts? In my old
configuration, normal hosts read
IOPS ranged from 50 to 500 while
HE ones peaked even ~3000<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original Message-----<br>
From: Andrea Fagiani [mailto:<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobil<wbr>iare.it</a>]<br>
Sent: Friday, February 10, 2017
5:56 PM<br>
To: Andrea Ghelardi <<a
moz-do-not-send="true"
href="mailto:a.ghelardi@iontrading.com"
target="_blank">a.ghelardi(a)iontrading.com</a>&gt;;
<a moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject: Re: [ovirt-users] high
number of IOPS from ovirt hosts<br>
<br>
Hi Andrea,<br>
<br>
thanks for the insight; no I
wasn't able to pinpoint the issue
to a single host, the load seems
to be evenly spread.<br>
We have another cluster running
4.0.6 on a compellent as well and
we haven't experienced any similar
issues.<br>
<br>
Thanks,<br>
Andrea<br>
<br>
<br>
On 10/02/2017 17:17, Andrea
Ghelardi wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Hello Andrea,<br>
I saw that happening on my HE
hosts connected to a COMPELLENT
SC040<br>
when I was running Ovirt 3.5<br>
<br>
All HE hosts were hammering the
SAN with 2~4k IOPS each<br>
<br>
Upgrading to 4.0.5 solved my
case.<br>
Are you able to check if IOPS
are generated from a particular
host or rather spread among all
nodes?<br>
<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original Message-----<br>
From: <a moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.org</a>
[mailto:<a
moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.or<wbr>g</a>]
On<br>
Behalf Of Andrea Fagiani<br>
Sent: Friday, February 10, 2017
12:47 PM<br>
To: <a moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject: [ovirt-users] high
number of IOPS from ovirt hosts<br>
<br>
Hi,<br>
<br>
We've got a 8-host cluster
running oVirt 4.1; each host is
connected via iscsi to a EQL
4100 storage (2x).<br>
Most of the VMs use barely any
IOPS, however on our storage
controller we've noticed a
constant steam of reads, about
2000 IOPS; any idea what might
cause the ovirt hosts to hammer
on the storage so much?<br>
<br>
Looking at iotop, it seems that
systemd-udevd is being activated
every few seconds (and its IO%
is usually fairly high) so I
assume this might be somehow
related.<br>
<br>
Thanks,<br>
Andrea<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</blockquote>
</blockquote>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------E0950BFE735406CA7670182D--