Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com>; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

...

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io.

https://bugzilla.redhat.com/show_bug.cgi?id=1337914 Just for the reference. Martin On Mon, Feb 13, 2017 at 10:13 AM, Nir Soffer <nsoffer@redhat.com> wrote:

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <andrea.fagiani@immobiliare.it> wrote:

...

Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.

Thanks, Andrea

Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server?

Nir

...

On 12/02/2017 18:27, Andrea Ghelardi wrote:

...

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com>; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

...

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani < andrea.fagiani@immobiliare.it> wrote:

Hi Nir,

the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.

Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs)

Andrea

On 13/02/2017 10:13, Nir Soffer wrote:

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani < andrea.fagiani@immobiliare.it> wrote:

...

Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.

Thanks, Andrea

I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server?

Nir

...

On 12/02/2017 18:27, Andrea Ghelardi wrote:

...

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com>; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

...

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

This is a multi-part message in MIME format. --------------E0950BFE735406CA7670182D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead. Thanks to everyone involved, Andrea On 13/02/2017 13:24, Nir Soffer wrote:

On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:

Hi Nir,

the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.

Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs)

Andrea

On 13/02/2017 10:13, Nir Soffer wrote:

...

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:

Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.

Thanks, Andrea

I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server?

Nir

On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com <mailto:a.ghelardi@iontrading.com>>; users@ovirt.org <mailto:users@ovirt.org> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> [mailto:users-bounces@ovirt.org <mailto:users-bounces@ovirt.org>] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org <mailto:users@ovirt.org> Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>

--------------E0950BFE735406CA7670182D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Hi,</p> <p>just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead.</p> <p>Thanks to everyone involved,<br> Andrea<br> </p> <br> <div class="moz-cite-prefix">On 13/02/2017 13:24, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyytbdFiO1sxofxZ8NxCsiWjgb3BXyonPo4Ah3UM1aDPtGw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Hi Nir,</p> <p>the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.</p> </div> </blockquote> <div><br> </div> <div>Andrea, can you send also logs from ovirt-hosted-engine-ha?</div> <div>(I'm not sure about the name of the service or the location of the logs)</div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"><span class="HOEnZb"><font color="#888888"> <p>Andrea<br> </p> </font></span> <div> <div class="h5"> <br> <div class="m_1481480387959180211moz-cite-prefix">On 13/02/2017 10:13, Nir Soffer wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a><wbr>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the heads up, I'll take a look at Mark's thread.<br> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.<br> <br> Thanks,<br> Andrea</blockquote> <div><br> </div> <div>I think we already have hosted engine bug about doing to many calls to vdsm</div> <div>that cause reloads of lvm data; this can cause increased io.</div> <div><br> </div> Can you attach vdsm and hosted engine agent logs showing the timeframe when <div>you have lot of iops on the server?</div> <div><br> </div> <div>Nir<br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div class="m_1481480387959180211gmail-HOEnZb"> <div class="m_1481480387959180211gmail-h5"><br> <br> <br> On 12/02/2017 18:27, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Andrea,<br> I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well.<br> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities.<br> How many storages are you presenting to how many clusters?<br> Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000<br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: Andrea Fagiani [mailto:<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobil<wbr>iare.it</a>]<br> Sent: Friday, February 10, 2017 5:56 PM<br> To: Andrea Ghelardi <<a moz-do-not-send="true" href="mailto:a.ghelardi@iontrading.com" target="_blank">a.ghelardi@iontrading.com</a>>; <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi Andrea,<br> <br> thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread.<br> We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.<br> <br> Thanks,<br> Andrea<br> <br> <br> On 10/02/2017 17:17, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hello Andrea,<br> I saw that happening on my HE hosts connected to a COMPELLENT SC040<br> when I was running Ovirt 3.5<br> <br> All HE hosts were hammering the SAN with 2~4k IOPS each<br> <br> Upgrading to 4.0.5 solved my case.<br> Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?<br> <br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: <a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a> [mailto:<a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.or<wbr>g</a>] On<br> Behalf Of Andrea Fagiani<br> Sent: Friday, February 10, 2017 12:47 PM<br> To: <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi,<br> <br> We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x).<br> Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?<br> <br> Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.<br> <br> Thanks,<br> Andrea<br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </blockquote> </blockquote> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </div> </div> </blockquote> </div> <br> </div> </div> </div> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------E0950BFE735406CA7670182D--

This is a multi-part message in MIME format. --------------1A6F0B610863719B77CE98F4 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Correct, coincidentally, there no instances of systemd-udevd going haywire either. Andrea On 14/02/2017 15:20, Yaniv Kaul wrote:

On Tue, Feb 14, 2017 at 2:15 PM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:

Hi,

just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead.

And now there are no high amount of IOPS on the Gluster storage? Y.

Thanks to everyone involved, Andrea

On 13/02/2017 13:24, Nir Soffer wrote:

...

On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:

Hi Nir,

the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.

Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs)

Andrea

On 13/02/2017 10:13, Nir Soffer wrote:

...

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>> wrote:

Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.

Thanks, Andrea

I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server?

Nir

On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it <mailto:andrea.fagiani@immobiliare.it>] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com <mailto:a.ghelardi@iontrading.com>>; users@ovirt.org <mailto:users@ovirt.org> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> [mailto:users-bounces@ovirt.org <mailto:users-bounces@ovirt.org>] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org <mailto:users@ovirt.org> Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>

--------------1A6F0B610863719B77CE98F4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Correct,<br> coincidentally, there no instances of systemd-udevd going haywire either.<br> </p> <p>Andrea<br> </p> <br> <div class="moz-cite-prefix">On 14/02/2017 15:20, Yaniv Kaul wrote:<br> </div> <blockquote cite="mid:CAJgorsboDgsDRNHgyuYt_xnZcnL9nAh4x73+XMpgECarJk2gFQ@mail.gmail.com" type="cite"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Tue, Feb 14, 2017 at 2:15 PM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Hi,</p> <p>just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead.</p> </div> </blockquote> <div><br> </div> <div>And now there are no high amount of IOPS on the Gluster storage?</div> <div>Y.</div> <div><br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Thanks to everyone involved,<br> Andrea<br> </p> <div> <div class="h5"> <br> <div class="m_-7339259678222976733moz-cite-prefix">On 13/02/2017 13:24, Nir Soffer wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a><wbr>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Hi Nir,</p> <p>the amount of IOPS is pretty much constant, attaching the logs from the last hour or so.</p> </div> </blockquote> <div><br> </div> <div>Andrea, can you send also logs from ovirt-hosted-engine-ha?</div> <div>(I'm not sure about the name of the service or the location of the logs)</div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"><span class="m_-7339259678222976733HOEnZb"><font color="#888888"> <p>Andrea<br> </p> </font></span> <div> <div class="m_-7339259678222976733h5"> <br> <div class="m_-7339259678222976733m_1481480387959180211moz-cite-prefix">On 13/02/2017 10:13, Nir Soffer wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobiliare.it</a><wbr>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the heads up, I'll take a look at Mark's thread.<br> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts.<br> <br> Thanks,<br> Andrea</blockquote> <div><br> </div> <div>I think we already have hosted engine bug about doing to many calls to vdsm</div> <div>that cause reloads of lvm data; this can cause increased io.</div> <div><br> </div> Can you attach vdsm and hosted engine agent logs showing the timeframe when <div>you have lot of iops on the server?</div> <div><br> </div> <div>Nir<br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div class="m_-7339259678222976733m_1481480387959180211gmail-HOEnZb"> <div class="m_-7339259678222976733m_1481480387959180211gmail-h5"><br> <br> <br> On 12/02/2017 18:27, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Andrea,<br> I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well.<br> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities.<br> How many storages are you presenting to how many clusters?<br> Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000<br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: Andrea Fagiani [mailto:<a moz-do-not-send="true" href="mailto:andrea.fagiani@immobiliare.it" target="_blank">andrea.fagiani@immobil<wbr>iare.it</a>]<br> Sent: Friday, February 10, 2017 5:56 PM<br> To: Andrea Ghelardi <<a moz-do-not-send="true" href="mailto:a.ghelardi@iontrading.com" target="_blank">a.ghelardi@iontrading.com</a>>; <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi Andrea,<br> <br> thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread.<br> We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.<br> <br> Thanks,<br> Andrea<br> <br> <br> On 10/02/2017 17:17, Andrea Ghelardi wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hello Andrea,<br> I saw that happening on my HE hosts connected to a COMPELLENT SC040<br> when I was running Ovirt 3.5<br> <br> All HE hosts were hammering the SAN with 2~4k IOPS each<br> <br> Upgrading to 4.0.5 solved my case.<br> Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?<br> <br> Cheers<br> AG<br> <br> <br> -----Original Message-----<br> From: <a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a> [mailto:<a moz-do-not-send="true" href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.or<wbr>g</a>] On<br> Behalf Of Andrea Fagiani<br> Sent: Friday, February 10, 2017 12:47 PM<br> To: <a moz-do-not-send="true" href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br> Subject: [ovirt-users] high number of IOPS from ovirt hosts<br> <br> Hi,<br> <br> We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x).<br> Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?<br> <br> Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.<br> <br> Thanks,<br> Andrea<br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </blockquote> </blockquote> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </div> </div> </blockquote> </div> <br> </div> </div> </div> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> </div> </div> </blockquote> <br> </div> </div> </div> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br> <br> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------1A6F0B610863719B77CE98F4--

Hi Nelson, We are going to disable the sparsify operation on pre-allocated disks, see https://bugzilla.redhat.com/show_bug.cgi?id=1414430. Regards, Arik בתאריך 13 בפבר׳ 2017 11:18 AM,‏ "Nelson Lameiras" < nelson.lameiras@lyra-network.com> כתב:

Hello,

Not sure if it's a bug or a limitation but, while playing with oVirt 4.1 new possibilities, I tried to use the sparsity function on a preallocated disk, and this action irreversibly ruined the data on the disk. I have repeated this experiment multiple times and always same result. (Sparsifying a thin provisioned disk works great though)

I understand that the "sparsify" function is relevant to thin provision disks only. Still I think there should be a warning before doing this action on a preallocated disk? or maybe even forbid this action on a preallocated disk since sparsify has no meaning on a preallocated disk? (I maybe wrong...)

thanks

cordialement, regards,

Nelson LAMEIRAS Ingénieur Systèmes et Réseaux / Systems and Networks engineer Tel: +33 5 32 09 09 70 nelson.lameiras@lyra-network.com

www.lyra-network.com | www.payzen.eu _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Andrea, Among other advices, I would also suggest you to re-deploy an host without make it an host-engine node. It may be worth checking if HE service is responsible of so many IOPS as it was in my case. Ciao AG -----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it] Sent: Monday, February 13, 2017 10:05 AM To: Andrea Ghelardi <a.ghelardi@iontrading.com>; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG

-----Original Message----- From: Andrea Fagiani [mailto:andrea.fagiani@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi <a.ghelardi@iontrading.com>; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues.

Thanks, Andrea

On 10/02/2017 17:17, Andrea Ghelardi wrote:

...

Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes?

Cheers AG

-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related.

Thanks, Andrea _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users