5:46 a.m.
Hi,
We've got a 8-host cluster running oVirt 4.1; each host is connected via
iscsi to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller
we've noticed a constant steam of reads, about 2000 IOPS; any idea what
might cause the ovirt hosts to hammer on the storage so much?
Looking at iotop, it seems that systemd-udevd is being activated every
few seconds (and its IO% is usually fairly high) so I assume this might
be somehow related.
Thanks,
Andrea
10:17 a.m.
Hello Andrea,
I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running
Ovirt 3.5
All HE hosts were hammering the SAN with 2~4k IOPS each
Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a particular host or rather spread among
all nodes?
Cheers
AG
-----Original Message-----
From: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea
Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users(a)ovirt.org
Subject: [ovirt-users] high number of IOPS from ovirt hosts
Hi,
We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a
EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller we've noticed a
constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to
hammer on the storage so much?
Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and
its IO% is usually fairly high) so I assume this might be somehow related.
Thanks,
Andrea
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:56 a.m.
Hi Andrea,
thanks for the insight; no I wasn't able to pinpoint the issue to a
single host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we
haven't experienced any similar issues.
Thanks,
Andrea
On 10/02/2017 17:17, Andrea Ghelardi wrote:
11:27 a.m.
Andrea,
I don't want to point any fingers but Mark Greenall faced some issues with Ovirt
4.06/4.1 linked to equallogic as well.
You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not
Responding' " to see if you find any similarities.
How many storages are you presenting to how many clusters?
Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read
IOPS ranged from 50 to 500 while HE ones peaked even ~3000
Cheers
AG
-----Original Message-----
From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it]
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi <a.ghelardi(a)iontrading.com>; users(a)ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
Hi Andrea,
thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the
load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we haven't
experienced any similar issues.
Thanks,
Andrea
On 10/02/2017 17:17, Andrea Ghelardi wrote:
3:13 a.m.
On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <
andrea.fagiani(a)immobiliare.it> wrote:
I think we already have hosted engine bug about doing to many calls to vdsm
that cause reloads of lvm data; this can cause increased io.
Can you attach vdsm and hosted engine agent logs showing the timeframe when
you have lot of iops on the server?
Nir
3:46 a.m.
I think we already have hosted engine bug about doing to many calls
to vdsm
that cause reloads of lvm data; this can cause increased io.
https://bugzilla.redhat.com/show_bug.cgi?id=1337914
Just for the reference.
Martin
On Mon, Feb 13, 2017 at 10:13 AM, Nir Soffer <nsoffer(a)redhat.com> wrote:
6:27 a.m.
Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of the logs)
/var/log/ovirt-hosted-engine-ha/agent.log :)
But you will see all the prepareImage and repostats calls there, we
still haven't tackled that issue.
Martin
On Mon, Feb 13, 2017 at 1:24 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani
> <andrea.fagiani(a)immobiliare.it> wrote:
>>
>> Hi Nir,
>>
>> the amount of IOPS is pretty much constant, attaching the logs from the
>> last hour or so.
>
>
Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of the logs)
>
>>
>> Andrea
>>
>>
>> On 13/02/2017 10:13, Nir Soffer wrote:
>>
>> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
>> <andrea.fagiani(a)immobiliare.it> wrote:
>>>
>>> Thanks for the heads up, I'll take a look at Mark's thread.
>>> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
>>> hosts are hosted engine hosts.
>>>
>>> Thanks,
>>> Andrea
>>
>>
>> I think we already have hosted engine bug about doing to many calls to
>> vdsm
>> that cause reloads of lvm data; this can cause increased io.
>>
>> Can you attach vdsm and hosted engine agent logs showing the timeframe
>> when
>> you have lot of iops on the server?
>>
>> Nir
>>
>>>
>>>
>>>
>>>
>>> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>>>
>>>> Andrea,
>>>> I don't want to point any fingers but Mark Greenall faced some
issues
>>>> with Ovirt 4.06/4.1 linked to equallogic as well.
>>>> You may want to search posts in this ML with subject " Ovirt 4.0.6
>>>> guests 'Not Responding' " to see if you find any
similarities.
>>>> How many storages are you presenting to how many clusters?
>>>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>>>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even
~3000
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it]
>>>> Sent: Friday, February 10, 2017 5:56 PM
>>>> To: Andrea Ghelardi <a.ghelardi(a)iontrading.com>; users(a)ovirt.org
>>>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>>>
>>>> Hi Andrea,
>>>>
>>>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>>>> single host, the load seems to be evenly spread.
>>>> We have another cluster running 4.0.6 on a compellent as well and we
>>>> haven't experienced any similar issues.
>>>>
>>>> Thanks,
>>>> Andrea
>>>>
>>>>
>>>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>>>>
>>>>> Hello Andrea,
>>>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>>>> when I was running Ovirt 3.5
>>>>>
>>>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>>>
>>>>> Upgrading to 4.0.5 solved my case.
>>>>> Are you able to check if IOPS are generated from a particular host
or
>>>>> rather spread among all nodes?
>>>>>
>>>>> Cheers
>>>>> AG
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] On
>>>>> Behalf Of Andrea Fagiani
>>>>> Sent: Friday, February 10, 2017 12:47 PM
>>>>> To: users(a)ovirt.org
>>>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>>>
>>>>> Hi,
>>>>>
>>>>> We've got a 8-host cluster running oVirt 4.1; each host is
connected
>>>>> via iscsi to a EQL 4100 storage (2x).
>>>>> Most of the VMs use barely any IOPS, however on our storage
controller
>>>>> we've noticed a constant steam of reads, about 2000 IOPS; any
idea what
>>>>> might cause the ovirt hosts to hammer on the storage so much?
>>>>>
>>>>> Looking at iotop, it seems that systemd-udevd is being activated
every
>>>>> few seconds (and its IO% is usually fairly high) so I assume this
might be
>>>>> somehow related.
>>>>>
>>>>> Thanks,
>>>>> Andrea
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>
6:15 a.m.
This is a multi-part message in MIME format.
--------------E0950BFE735406CA7670182D
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hi,
just a quick update on the issue. The high amount of IOPS was indeed due
to having the hosted-engine on the EQL storage; we worked around the
issue by moving the hosted-engine to a gluster storage instead.
Thanks to everyone involved,
Andrea
On 13/02/2017 13:24, Nir Soffer wrote:
--------------E0950BFE735406CA7670182D
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi,</p>
<p>just a quick update on the issue. The high amount of IOPS was
indeed due to having the hosted-engine on the EQL storage; we
worked around the issue by moving the hosted-engine to a gluster
storage instead.</p>
<p>Thanks to everyone involved,<br>
Andrea<br>
</p>
<br>
<div class="moz-cite-prefix">On 13/02/2017 13:24, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyytbdFiO1sxofxZ8NxCsiWjgb3BXyonPo4Ah3UM1aDPtGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Feb 13, 2017 at 12:32 PM,
Andrea Fagiani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi Nir,</p>
<p>the amount of IOPS is pretty much constant, attaching
the logs from the last hour or so.</p>
</div>
</blockquote>
<div><br>
</div>
<div>Andrea, can you send also logs from
ovirt-hosted-engine-ha?</div>
<div>(I'm not sure about the name of the service or the
location of the logs)</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span
class="HOEnZb"><font
color="#888888">
<p>Andrea<br>
</p>
</font></span>
<div>
<div class="h5"> <br>
<div class="m_1481480387959180211moz-cite-prefix">On
13/02/2017 10:13, Nir Soffer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Feb 13,
2017 at 11:05 AM, Andrea Fagiani <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a><wbr>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Thanks
for the heads up, I'll take a look at
Mark's thread.<br>
We are presenting 2 storage LUNs to a
single ovirt cluster and yes, all hosts
are hosted engine hosts.<br>
<br>
Thanks,<br>
Andrea</blockquote>
<div><br>
</div>
<div>I think we already have hosted engine
bug about doing to many calls to vdsm</div>
<div>that cause reloads of lvm data; this
can cause increased io.</div>
<div><br>
</div>
Can you attach vdsm and hosted engine
agent logs showing the timeframe when
<div>you have lot of iops on the server?</div>
<div><br>
</div>
<div>Nir<br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
class="m_1481480387959180211gmail-HOEnZb">
<div
class="m_1481480387959180211gmail-h5"><br>
<br>
<br>
On 12/02/2017 18:27, Andrea Ghelardi
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Andrea,<br>
I don't want to point any fingers
but Mark Greenall faced some
issues with Ovirt 4.06/4.1 linked
to equallogic as well.<br>
You may want to search posts in
this ML with subject " Ovirt 4.0.6
guests 'Not Responding' " to see
if you find any similarities.<br>
How many storages are you
presenting to how many clusters?<br>
Also, are your nodes all Hosted
Engine Hosts? In my old
configuration, normal hosts read
IOPS ranged from 50 to 500 while
HE ones peaked even ~3000<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original Message-----<br>
From: Andrea Fagiani [mailto:<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobil<wbr>iare.it</a>]<br>
Sent: Friday, February 10, 2017
5:56 PM<br>
To: Andrea Ghelardi <<a
moz-do-not-send="true"
href="mailto:a.ghelardi@iontrading.com"
target="_blank">a.ghelardi(a)iontrading.com</a>&gt;;
<a moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject: Re: [ovirt-users] high
number of IOPS from ovirt hosts<br>
<br>
Hi Andrea,<br>
<br>
thanks for the insight; no I
wasn't able to pinpoint the issue
to a single host, the load seems
to be evenly spread.<br>
We have another cluster running
4.0.6 on a compellent as well and
we haven't experienced any similar
issues.<br>
<br>
Thanks,<br>
Andrea<br>
<br>
<br>
On 10/02/2017 17:17, Andrea
Ghelardi wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Hello Andrea,<br>
I saw that happening on my HE
hosts connected to a COMPELLENT
SC040<br>
when I was running Ovirt 3.5<br>
<br>
All HE hosts were hammering the
SAN with 2~4k IOPS each<br>
<br>
Upgrading to 4.0.5 solved my
case.<br>
Are you able to check if IOPS
are generated from a particular
host or rather spread among all
nodes?<br>
<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original Message-----<br>
From: <a moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.org</a>
[mailto:<a
moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.or<wbr>g</a>]
On<br>
Behalf Of Andrea Fagiani<br>
Sent: Friday, February 10, 2017
12:47 PM<br>
To: <a moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject: [ovirt-users] high
number of IOPS from ovirt hosts<br>
<br>
Hi,<br>
<br>
We've got a 8-host cluster
running oVirt 4.1; each host is
connected via iscsi to a EQL
4100 storage (2x).<br>
Most of the VMs use barely any
IOPS, however on our storage
controller we've noticed a
constant steam of reads, about
2000 IOPS; any idea what might
cause the ovirt hosts to hammer
on the storage so much?<br>
<br>
Looking at iotop, it seems that
systemd-udevd is being activated
every few seconds (and its IO%
is usually fairly high) so I
assume this might be somehow
related.<br>
<br>
Thanks,<br>
Andrea<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</blockquote>
</blockquote>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------E0950BFE735406CA7670182D--
9:08 a.m.
This is a multi-part message in MIME format.
--------------1A6F0B610863719B77CE98F4
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Correct,
coincidentally, there no instances of systemd-udevd going haywire either.
Andrea
On 14/02/2017 15:20, Yaniv Kaul wrote:
--------------1A6F0B610863719B77CE98F4
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Correct,<br>
coincidentally, there no instances of systemd-udevd going haywire
either.<br>
</p>
<p>Andrea<br>
</p>
<br>
<div class="moz-cite-prefix">On 14/02/2017 15:20, Yaniv Kaul
wrote:<br>
</div>
<blockquote
cite="mid:CAJgorsboDgsDRNHgyuYt_xnZcnL9nAh4x73+XMpgECarJk2gFQ@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 14, 2017 at 2:15 PM,
Andrea Fagiani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi,</p>
<p>just a quick update on the issue. The high amount of
IOPS was indeed due to having the hosted-engine on the
EQL storage; we worked around the issue by moving the
hosted-engine to a gluster storage instead.</p>
</div>
</blockquote>
<div><br>
</div>
<div>And now there are no high amount of IOPS on the Gluster
storage?</div>
<div>Y.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Thanks to everyone involved,<br>
Andrea<br>
</p>
<div>
<div class="h5"> <br>
<div class="m_-7339259678222976733moz-cite-prefix">On
13/02/2017 13:24, Nir Soffer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Feb 13, 2017
at 12:32 PM, Andrea Fagiani <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a><wbr>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p>Hi Nir,</p>
<p>the amount of IOPS is pretty much
constant, attaching the logs from the
last hour or so.</p>
</div>
</blockquote>
<div><br>
</div>
<div>Andrea, can you send also logs from
ovirt-hosted-engine-ha?</div>
<div>(I'm not sure about the name of the
service or the location of the logs)</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"><span
class="m_-7339259678222976733HOEnZb"><font
color="#888888">
<p>Andrea<br>
</p>
</font></span>
<div>
<div class="m_-7339259678222976733h5">
<br>
<div
class="m_-7339259678222976733m_1481480387959180211moz-cite-prefix">On
13/02/2017 10:13, Nir Soffer
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On
Mon, Feb 13, 2017 at 11:05
AM, Andrea Fagiani <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a><wbr>&gt;</span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">Thanks
for the heads up, I'll
take a look at Mark's
thread.<br>
We are presenting 2
storage LUNs to a single
ovirt cluster and yes,
all hosts are hosted
engine hosts.<br>
<br>
Thanks,<br>
Andrea</blockquote>
<div><br>
</div>
<div>I think we already
have hosted engine bug
about doing to many
calls to vdsm</div>
<div>that cause reloads of
lvm data; this can cause
increased io.</div>
<div><br>
</div>
Can you attach vdsm and
hosted engine agent logs
showing the timeframe when
<div>you have lot of iops
on the server?</div>
<div><br>
</div>
<div>Nir<br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div
class="m_-7339259678222976733m_1481480387959180211gmail-HOEnZb">
<div
class="m_-7339259678222976733m_1481480387959180211gmail-h5"><br>
<br>
<br>
On 12/02/2017 18:27,
Andrea Ghelardi
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
Andrea,<br>
I don't want to
point any fingers
but Mark Greenall
faced some issues
with Ovirt
4.06/4.1 linked to
equallogic as
well.<br>
You may want to
search posts in
this ML with
subject " Ovirt
4.0.6 guests 'Not
Responding' " to
see if you find
any similarities.<br>
How many storages
are you presenting
to how many
clusters?<br>
Also, are your
nodes all Hosted
Engine Hosts? In
my old
configuration,
normal hosts read
IOPS ranged from
50 to 500 while HE
ones peaked even
~3000<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original
Message-----<br>
From: Andrea
Fagiani [mailto:<a
moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobil<wbr>iare.it</a>]<br>
Sent: Friday,
February 10, 2017
5:56 PM<br>
To: Andrea
Ghelardi <<a
moz-do-not-send="true"
href="mailto:a.ghelardi@iontrading.com"
target="_blank">a.ghelardi(a)iontrading.com</a>&gt;;
<a
moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject: Re:
[ovirt-users] high
number of IOPS
from ovirt hosts<br>
<br>
Hi Andrea,<br>
<br>
thanks for the
insight; no I
wasn't able to
pinpoint the issue
to a single host,
the load seems to
be evenly spread.<br>
We have another
cluster running
4.0.6 on a
compellent as well
and we haven't
experienced any
similar issues.<br>
<br>
Thanks,<br>
Andrea<br>
<br>
<br>
On 10/02/2017
17:17, Andrea
Ghelardi wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Hello Andrea,<br>
I saw that
happening on my
HE hosts
connected to a
COMPELLENT SC040<br>
when I was
running Ovirt
3.5<br>
<br>
All HE hosts
were hammering
the SAN with
2~4k IOPS each<br>
<br>
Upgrading to
4.0.5 solved my
case.<br>
Are you able to
check if IOPS
are generated
from a
particular host
or rather spread
among all nodes?<br>
<br>
Cheers<br>
AG<br>
<br>
<br>
-----Original
Message-----<br>
From: <a
moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.org</a>
[mailto:<a
moz-do-not-send="true"
href="mailto:users-bounces@ovirt.org"
target="_blank">users-bounces(a)ovirt.or<wbr>g</a>]
On<br>
Behalf Of Andrea
Fagiani<br>
Sent: Friday,
February 10,
2017 12:47 PM<br>
To: <a
moz-do-not-send="true"
href="mailto:users@ovirt.org"
target="_blank">users(a)ovirt.org</a><br>
Subject:
[ovirt-users]
high number of
IOPS from ovirt
hosts<br>
<br>
Hi,<br>
<br>
We've got a
8-host cluster
running oVirt
4.1; each host
is connected via
iscsi to a EQL
4100 storage
(2x).<br>
Most of the VMs
use barely any
IOPS, however on
our storage
controller we've
noticed a
constant steam
of reads, about
2000 IOPS; any
idea what might
cause the ovirt
hosts to hammer
on the storage
so much?<br>
<br>
Looking at
iotop, it seems
that
systemd-udevd is
being activated
every few
seconds (and its
IO% is usually
fairly high) so
I assume this
might be somehow
related.<br>
<br>
Thanks,<br>
Andrea<br>
______________________________<wbr>_________________<br>
Users mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</blockquote>
</blockquote>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a
moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/<wbr>mailman/li...
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------1A6F0B610863719B77CE98F4--
3:18 a.m.
New subject: ovirt 4.1.0 - sparsify ireversibly destroys preallocated disks
Hello,
Not sure if it's a bug or a limitation but, while playing with oVirt 4.1 new
possibilities, I tried to use the sparsity function on a preallocated disk, and this
action irreversibly ruined the data on the disk. I have repeated this experiment multiple
times and always same result. (Sparsifying a thin provisioned disk works great though)
I understand that the "sparsify" function is relevant to thin provision disks
only. Still I think there should be a warning before doing this action on a preallocated
disk? or maybe even forbid this action on a preallocated disk since sparsify has no
meaning on a preallocated disk? (I maybe wrong...)
thanks
cordialement, regards,
Nelson LAMEIRAS
Ingénieur Systèmes et Réseaux / Systems and Networks engineer
Tel: +33 5 32 09 09 70
nelson.lameiras(a)lyra-network.com
www.lyra-network.com | www.payzen.eu
4:14 a.m.
New subject: ovirt 4.1.0 - sparsify ireversibly destroys preallocated disks
Hi Nelson,
We are going to disable the sparsify operation on pre-allocated disks, see
https://bugzilla.redhat.com/show_bug.cgi?id=1414430.
Regards,
Arik
בתאריך 13 בפבר׳ 2017 11:18 AM, "Nelson Lameiras" <
nelson.lameiras(a)lyra-network.com> כתב:
3:43 a.m.
Andrea,
Among other advices, I would also suggest you to re-deploy an host without make it an
host-engine node.
It may be worth checking if HE service is responsible of so many IOPS as it was in my
case.
Ciao
AG
-----Original Message-----
From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it]
Sent: Monday, February 13, 2017 10:05 AM
To: Andrea Ghelardi <a.ghelardi(a)iontrading.com>; users(a)ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted
engine hosts.
Thanks,
Andrea
On 12/02/2017 18:27, Andrea Ghelardi wrote:
3:11 a.m.
This is a multi-part message in MIME format.
--------------432CF2AF75540A06C0552C48
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hi Simone,
we are exposing 2x 5TB LUNs, used solely by the ovirt cluster.
Thanks
Andrea
On 12/02/2017 20:21, Simone Tiraboschi wrote:
--------------432CF2AF75540A06C0552C48
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Simone,<br>
we are exposing 2x 5TB LUNs, used solely by the ovirt cluster.<br>
</p>
Thanks<br>
Andrea<br>
<br>
<div class="moz-cite-prefix">On 12/02/2017 20:21, Simone Tiraboschi
wrote:<br>
</div>
<blockquote
cite="mid:CAN8-ONo1fUe0kqf4u66yVUa5KqC0E-OpY6WvyfrsvYfZ4__YRg@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Feb 10, 2017 at 12:46 PM,
Andrea Fagiani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:andrea.fagiani@immobiliare.it"
target="_blank">andrea.fagiani(a)immobiliare.it</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We've got a 8-host cluster running oVirt 4.1; each host is
connected via iscsi to a EQL 4100 storage (2x).<br>
Most of the VMs use barely any IOPS, however on our
storage controller we've noticed a constant steam of
reads, about 2000 IOPS; any idea what might cause the
ovirt hosts to hammer on the storage so much?<br>
<br>
Looking at iotop, it seems that systemd-udevd is being
activated every few seconds (and its IO% is usually fairly
high) so I assume this might be somehow related.<br>
</blockquote>
<div><br>
</div>
<div>mmmh this could be an hint: how many LUNs are you
exposing on the iSCSI portal used for the hosted-engine
LUN?</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
Andrea<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------432CF2AF75540A06C0552C48--
3027
days inactive
3031
days old
19 comments
8 participants
participants (8)
-
Andrea Fagiani -
Andrea Ghelardi -
Arik Hadas -
Martin Sivak -
Nelson Lameiras -
Nir Soffer -
Simone Tiraboschi -
Yaniv Kaul