[ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Hi,

just a quick update on the issue. The high amount of IOPS was indeed due 
to having the hosted-engine on the EQL storage; we worked around the 
issue by moving the hosted-engine to a gluster storage instead.

Thanks to everyone involved,
Andrea


On 13/02/2017 13:24, Nir Soffer wrote:
> On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani 
> <andrea.fagiani at immobiliare.it <mailto:andrea.fagiani at immobiliare.it>> 
> wrote:
>
>     Hi Nir,
>
>     the amount of IOPS is pretty much constant, attaching the logs
>     from the last hour or so.
>
>
> Andrea, can you send also logs from ovirt-hosted-engine-ha?
> (I'm not sure about the name of the service or the location of the logs)
>
>     Andrea
>
>
>     On 13/02/2017 10:13, Nir Soffer wrote:
>>     On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
>>     <andrea.fagiani at immobiliare.it
>>     <mailto:andrea.fagiani at immobiliare.it>> wrote:
>>
>>         Thanks for the heads up, I'll take a look at Mark's thread.
>>         We are presenting 2 storage LUNs to a single ovirt cluster
>>         and yes, all hosts are hosted engine hosts.
>>
>>         Thanks,
>>         Andrea
>>
>>
>>     I think we already have hosted engine bug about doing to many
>>     calls to vdsm
>>     that cause reloads of lvm data; this can cause increased io.
>>
>>     Can you attach vdsm and hosted engine agent logs showing the
>>     timeframe when
>>     you have lot of iops on the server?
>>
>>     Nir
>>
>>
>>
>>
>>         On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>
>>             Andrea,
>>             I don't want to point any fingers but Mark Greenall faced
>>             some issues with Ovirt 4.06/4.1 linked to equallogic as well.
>>             You may want to search posts in this ML with subject "
>>             Ovirt 4.0.6 guests 'Not Responding' " to see if you find
>>             any similarities.
>>             How many storages are you presenting to how many clusters?
>>             Also, are your nodes all Hosted Engine Hosts? In my old
>>             configuration, normal hosts read IOPS ranged from 50 to
>>             500 while HE ones peaked even ~3000
>>             Cheers
>>             AG
>>
>>
>>             -----Original Message-----
>>             From: Andrea Fagiani
>>             [mailto:andrea.fagiani at immobiliare.it
>>             <mailto:andrea.fagiani at immobiliare.it>]
>>             Sent: Friday, February 10, 2017 5:56 PM
>>             To: Andrea Ghelardi <a.ghelardi at iontrading.com
>>             <mailto:a.ghelardi at iontrading.com>>; users at ovirt.org
>>             <mailto:users at ovirt.org>
>>             Subject: Re: [ovirt-users] high number of IOPS from ovirt
>>             hosts
>>
>>             Hi Andrea,
>>
>>             thanks for the insight; no I wasn't able to pinpoint the
>>             issue to a single host, the load seems to be evenly spread.
>>             We have another cluster running 4.0.6 on a compellent as
>>             well and we haven't experienced any similar issues.
>>
>>             Thanks,
>>             Andrea
>>
>>
>>             On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>
>>                 Hello Andrea,
>>                 I saw that happening on my HE hosts connected to a
>>                 COMPELLENT SC040
>>                 when I was running Ovirt 3.5
>>
>>                 All HE hosts were hammering the SAN with 2~4k IOPS each
>>
>>                 Upgrading to 4.0.5 solved my case.
>>                 Are you able to check if IOPS are generated from a
>>                 particular host or rather spread among all nodes?
>>
>>                 Cheers
>>                 AG
>>
>>
>>                 -----Original Message-----
>>                 From: users-bounces at ovirt.org
>>                 <mailto:users-bounces at ovirt.org>
>>                 [mailto:users-bounces at ovirt.org
>>                 <mailto:users-bounces at ovirt.org>] On
>>                 Behalf Of Andrea Fagiani
>>                 Sent: Friday, February 10, 2017 12:47 PM
>>                 To: users at ovirt.org <mailto:users at ovirt.org>
>>                 Subject: [ovirt-users] high number of IOPS from ovirt
>>                 hosts
>>
>>                 Hi,
>>
>>                 We've got a 8-host cluster running oVirt 4.1; each
>>                 host is connected via iscsi to a EQL 4100 storage (2x).
>>                 Most of the VMs use barely any IOPS, however on our
>>                 storage controller we've noticed a constant steam of
>>                 reads, about 2000 IOPS; any idea what might cause the
>>                 ovirt hosts to hammer on the storage so much?
>>
>>                 Looking at iotop, it seems that systemd-udevd is
>>                 being activated every few seconds (and its IO% is
>>                 usually fairly high) so I assume this might be
>>                 somehow related.
>>
>>                 Thanks,
>>                 Andrea
>>                 _______________________________________________
>>                 Users mailing list
>>                 Users at ovirt.org <mailto:Users at ovirt.org>
>>                 http://lists.ovirt.org/mailman/listinfo/users
>>                 <http://lists.ovirt.org/mailman/listinfo/users>
>>
>>
>>         _______________________________________________
>>         Users mailing list
>>         Users at ovirt.org <mailto:Users at ovirt.org>
>>         http://lists.ovirt.org/mailman/listinfo/users
>>         <http://lists.ovirt.org/mailman/listinfo/users>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170214/15a65380/attachment-0001.html>