7:19 p.m.
hi,
i have prometheus based ovirt hosts monitoring (node_exporter,
smartcl_exporter, ipmi_exporter)
https://prometheus-community.github.io/ansible/branch/main/ and alerts
from https://samber.github.io/awesome-prometheus-alerts/
after i started this monitoring i found that one VM is overloading
local storage (so i must check IO limiting documentation as a homework :) )
but my question is
how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
some qemu/libvirt exporter? some custom text file + node_exporter?
thanks for tips
Marek
7:35 p.m.
For detailed monitoring I use Zabbix. This way I get detailed metrics
on my hypervisors, VMs as well as my network storage.
If a machine starts generating large IO I get alerts highlighting the
responsible machine as well as the impacted services. For example, you
might get high IO on a VM but also the correlated high latency on
systems sharing the storage.
Sometimes users will report the the high latency, masking the real
problem so it's nice to have a holistic view of the entire environment.
Patrick.Dubois
On 2024-02-13 11:19, marek wrote:
> hi,
>
> i have prometheus based ovirt hosts monitoring (node_exporter,
> smartcl_exporter, ipmi_exporter)
>
> https://prometheus-community.github.io/ansible/branch/main/ and alerts
> from https://samber.github.io/awesome-prometheus-alerts/
>
> after i started this monitoring i found that one VM is overloading
> local storage (so i must check IO limiting documentation as a homework
> :) )
>
> but my question is
>
> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>
> some qemu/libvirt exporter? some custom text file + node_exporter?
>
> thanks for tips
>
> Marek
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
10:32 p.m.
Hi.
I also use Zabbix here. Its problem is that it collects metrics in real
time, this is not its function.
There are other alternatives like Elasticsearch + metricbeat, but from what
I've tested, it's very heavy and uses a lot of disk space lol.
I never used Prometheus, I found it interesting. I'll do some tests.
@Patrick Dubois <pat(a)pdubois.com> How often do you collect information with
Zabbix? Every 1 minute?
Because for example... for the information to be used correctly for
analysis, we have to have an IO load of at least 1 continuous minute so
that Zabbix can collect the correct information.
Cheers!
Em ter., 13 de fev. de 2024 às 13:44, Patrick Dubois via Users <
users(a)ovirt.org> escreveu:
For detailed monitoring I use Zabbix. This way I get detailed
metrics
on my hypervisors, VMs as well as my network storage.
If a machine starts generating large IO I get alerts highlighting the
responsible machine as well as the impacted services. For example, you
might get high IO on a VM but also the correlated high latency on
systems sharing the storage.
Sometimes users will report the the high latency, masking the real
problem so it's nice to have a holistic view of the entire environment.
Patrick.Dubois
On 2024-02-13 11:19, marek wrote:
> hi,
>
> i have prometheus based ovirt hosts monitoring (node_exporter,
> smartcl_exporter, ipmi_exporter)
>
> https://prometheus-community.github.io/ansible/branch/main/ and alerts
> from https://samber.github.io/awesome-prometheus-alerts/
>
> after i started this monitoring i found that one VM is overloading
> local storage (so i must check IO limiting documentation as a homework
> :) )
>
> but my question is
>
> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>
> some qemu/libvirt exporter? some custom text file + node_exporter?
>
> thanks for tips
>
> Marek
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L4SU7YZ52PO...
--
Att,
Jorge Visentini
+55 55 98432-9868
11 p.m.
I have different collection schedules depending on the importance of the
data i'm collecting. You can adjust as you need accordingly.
For your IO issues you can easily simply poll your machine's IO load
statistics from the 5/10/15 minute averages. That will not give you
precise intervals but certainly will tell you if something is going wrong.
To be honest, Zabbix is flexible enough to get you what you need even
if you're not monitoring the metric directly. Anything you can do to
raise system visibility is good stuff!
Enjoy!
On 2024-02-13 14:32, Jorge Visentini wrote:
Hi.
I also use Zabbix here. Its problem is that it collects metrics in
real time, this is not its function.
There are other alternatives like Elasticsearch + metricbeat, but from
what I've tested, it's very heavy and uses a lot of disk space lol.
I never used Prometheus, I found it interesting. I'll do some tests.
@Patrick Dubois <mailto:pat@pdubois.com> How often do you collect
information with Zabbix? Every 1 minute?
Because for example... for the information to be used correctly for
analysis, we have to have an IO load of at least 1 continuous minute
so that Zabbix can collect the correct information.
Cheers!
Em ter., 13 de fev. de 2024 às 13:44, Patrick Dubois via Users
<users(a)ovirt.org> escreveu:
For detailed monitoring I use Zabbix. This way I get detailed
metrics
on my hypervisors, VMs as well as my network storage.
If a machine starts generating large IO I get alerts highlighting the
responsible machine as well as the impacted services. For
example, you
might get high IO on a VM but also the correlated high latency on
systems sharing the storage.
Sometimes users will report the the high latency, masking the real
problem so it's nice to have a holistic view of the entire
environment.
Patrick.Dubois
On 2024-02-13 11:19, marek wrote:
> hi,
>
> i have prometheus based ovirt hosts monitoring (node_exporter,
> smartcl_exporter, ipmi_exporter)
>
> https://prometheus-community.github.io/ansible/branch/main/ and
alerts
> from https://samber.github.io/awesome-prometheus-alerts/
>
> after i started this monitoring i found that one VM is overloading
> local storage (so i must check IO limiting documentation as a
homework
> :) )
>
> but my question is
>
> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>
> some qemu/libvirt exporter? some custom text file + node_exporter?
>
> thanks for tips
>
> Marek
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L4SU7YZ52PO...
--
Att,
Jorge Visentini
+55 55 98432-9868
11:04 a.m.
Hi Marek,
In fact all the data you need is already collected by oVirt/VDSM itself
and saved into the DWH database.
I configured sql_exporter for prometheus which does queries on the DWH
database to gather the data I need.
This is exported to prometheus, and there I can query all the data and
do some alerting on for example I/O usage.
Jean-Louis
On 13/02/2024 17:19, marek wrote:
> hi,
>
> i have prometheus based ovirt hosts monitoring (node_exporter,
> smartcl_exporter, ipmi_exporter)
>
> https://prometheus-community.github.io/ansible/branch/main/ and alerts
> from https://samber.github.io/awesome-prometheus-alerts/
>
> after i started this monitoring i found that one VM is overloading
> local storage (so i must check IO limiting documentation as a homework
> :) )
>
> but my question is
>
> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>
> some qemu/libvirt exporter? some custom text file + node_exporter?
>
> thanks for tips
>
> Marek
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
3:24 p.m.
can you publish your sql_exporter configuration?
i found this exporter https://github.com/czerwonk/ovirt_exporter . will
give a try
Marek
Dne 2024-02-14 v 9:04 Jean-Louis Dupond napsal(a):
> Hi Marek,
>
> In fact all the data you need is already collected by oVirt/VDSM
> itself and saved into the DWH database.
> I configured sql_exporter for prometheus which does queries on the DWH
> database to gather the data I need.
> This is exported to prometheus, and there I can query all the data and
> do some alerting on for example I/O usage.
>
> Jean-Louis
>
> On 13/02/2024 17:19, marek wrote:
>> hi,
>>
>> i have prometheus based ovirt hosts monitoring (node_exporter,
>> smartcl_exporter, ipmi_exporter)
>>
>> https://prometheus-community.github.io/ansible/branch/main/ and
>> alerts from https://samber.github.io/awesome-prometheus-alerts/
>>
>> after i started this monitoring i found that one VM is overloading
>> local storage (so i must check IO limiting documentation as a
>> homework :) )
>>
>> but my question is
>>
>> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>>
>> some qemu/libvirt exporter? some custom text file + node_exporter?
>>
>> thanks for tips
>>
>> Marek
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
4:19 p.m.
https://bugzilla.redhat.com/show_bug.cgi?id=1679333
You can find the config I use in this bugreport :)
Jean-Louis
On 14/02/2024 13:24, marek wrote:
> can you publish your sql_exporter configuration?
>
> i found this exporter https://github.com/czerwonk/ovirt_exporter .
> will give a try
>
> Marek
>
>
> Dne 2024-02-14 v 9:04 Jean-Louis Dupond napsal(a):
>> Hi Marek,
>>
>> In fact all the data you need is already collected by oVirt/VDSM
>> itself and saved into the DWH database.
>> I configured sql_exporter for prometheus which does queries on the
>> DWH database to gather the data I need.
>> This is exported to prometheus, and there I can query all the data
>> and do some alerting on for example I/O usage.
>>
>> Jean-Louis
>>
>> On 13/02/2024 17:19, marek wrote:
>>> hi,
>>>
>>> i have prometheus based ovirt hosts monitoring (node_exporter,
>>> smartcl_exporter, ipmi_exporter)
>>>
>>> https://prometheus-community.github.io/ansible/branch/main/ and
>>> alerts from https://samber.github.io/awesome-prometheus-alerts/
>>>
>>> after i started this monitoring i found that one VM is overloading
>>> local storage (so i must check IO limiting documentation as a
>>> homework :) )
>>>
>>> but my question is
>>>
>>> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>>>
>>> some qemu/libvirt exporter? some custom text file + node_exporter?
>>>
>>> thanks for tips
>>>
>>> Marek
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BVXGPID57RF...
1:44 p.m.
btw i found this interesting discussion about libvirt exporter
https://github.com/prometheus-community/community/issues/50
Marek
Dne 2024-02-14 v 14:19 Jean-Louis Dupond napsal(a):
> https://bugzilla.redhat.com/show_bug.cgi?id=1679333
>
> You can find the config I use in this bugreport :)
>
> Jean-Louis
>
> On 14/02/2024 13:24, marek wrote:
>> can you publish your sql_exporter configuration?
>>
>> i found this exporter https://github.com/czerwonk/ovirt_exporter .
>> will give a try
>>
>> Marek
>>
>>
>> Dne 2024-02-14 v 9:04 Jean-Louis Dupond napsal(a):
>>> Hi Marek,
>>>
>>> In fact all the data you need is already collected by oVirt/VDSM
>>> itself and saved into the DWH database.
>>> I configured sql_exporter for prometheus which does queries on the
>>> DWH database to gather the data I need.
>>> This is exported to prometheus, and there I can query all the data
>>> and do some alerting on for example I/O usage.
>>>
>>> Jean-Louis
>>>
>>> On 13/02/2024 17:19, marek wrote:
>>>> hi,
>>>>
>>>> i have prometheus based ovirt hosts monitoring (node_exporter,
>>>> smartcl_exporter, ipmi_exporter)
>>>>
>>>> https://prometheus-community.github.io/ansible/branch/main/ and
>>>> alerts from https://samber.github.io/awesome-prometheus-alerts/
>>>>
>>>> after i started this monitoring i found that one VM is overloading
>>>> local storage (so i must check IO limiting documentation as a
>>>> homework :) )
>>>>
>>>> but my question is
>>>>
>>>> how do you monitor IO traffic per VM? (IOPS, read/write traffic,..)
>>>>
>>>> some qemu/libvirt exporter? some custom text file + node_exporter?
>>>>
>>>> thanks for tips
>>>>
>>>> Marek
>>>> _______________________________________________
>>>> Users mailing list -- users(a)ovirt.org
>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>>> oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> List Archives:
>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HVHFX464QJ...
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BVXGPID57RF...
91
days inactive
97
days old
7 comments
4 participants
participants (4)
-
Jean-Louis Dupond -
Jorge Visentini -
marek -
Patrick Dubois