On Fri, Jun 30, 2017 at 1:55 PM, Florian Schmid <fschmid@ubimet.com> wrote:

Hi Yaniv,

thank you for your answer! I haven't known, that there is already such a monitoring tool on ovirt.

We will sure give it a try, but we have already in our environment a monitoring tool, that's why I wanted to add those values, too.

How does collectd get this data from libvirt, when the corresponding cgroup values are empty?

Specifically for IO statistics, VDSM reads the values from libvirt[1]. cgroup limiting is possible if you define it, but is unrelated. Also note that 7.3 is a bit ancient, I'm not sure how supported it is with latest 4.1 - which I'm sure will pull new dependencies from 7.4 (for example, libvirt!). Y. [1] https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/virt/vmstats.py...

BR Florian

------------------------------ *Von: *"Yaniv Kaul" <ykaul@redhat.com> *An: *"Florian Schmid" <fschmid@ubimet.com> *CC: *"users" <users@ovirt.org> *Gesendet: *Dienstag, 27. Juni 2017 09:08:51 *Betreff: *Re: [ovirt-users] Empty cgroup files on centos 7.3 host

On Mon, Jun 26, 2017 at 11:03 PM, Florian Schmid <fschmid@ubimet.com> wrote:

...

Hi,

I wanted to monitor disk IO and R/W on all of our oVirt centos 7.3 hypervisor hosts, but it looks like that all those files are empty.

We have a very nice integration with Elastic based monitoring and logging - why not use it. On the host, we use collectd for monitoring. See http://www.ovirt.org/develop/release-management/features/ engine/metrics-store/

Y.

...

For example: ls -al /sys/fs/cgroup/blkio/machine.slice/machine-qemu\\x2d14\\ x2dHostedEngine.scope/ insgesamt 0 drwxr-xr-x. 2 root root 0 30. Mai 10:09 . drwxr-xr-x. 16 root root 0 26. Jun 09:25 .. -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_merged -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_merged_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_queued -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_queued_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_bytes -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_bytes_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_serviced -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_serviced_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_time -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_time_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_wait_time -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_wait_time_recursive -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.leaf_weight -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.leaf_weight_device --w-------. 1 root root 0 30. Mai 10:09 blkio.reset_stats -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.sectors -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.sectors_recursive -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.io_service_bytes -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.io_serviced -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.read_bps_device -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.read_iops_device -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.write_bps_device -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.write_iops_device -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.time -r--r--r--. 1 root root 0 30. Mai 10:09 blkio.time_recursive -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.weight -rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.weight_device -rw-r--r--. 1 root root 0 30. Mai 10:09 cgroup.clone_children --w--w--w-. 1 root root 0 30. Mai 10:09 cgroup.event_control -rw-r--r--. 1 root root 0 30. Mai 10:09 cgroup.procs -rw-r--r--. 1 root root 0 30. Mai 10:09 notify_on_release -rw-r--r--. 1 root root 0 30. Mai 10:09 tasks

I thought, I can get my needed values from there, but all files are empty.

Looking at this post: http://lists.ovirt.org/ pipermail/users/2017-January/079011.html this should work.

Is this normal on centos 7.3 with oVirt installed? How can I get those values, without monitoring all VMs directly?

oVirt Version we use: 4.1.1.8-1.el7.centos

BR Florian _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

<br><span style=3D"font-variant: normal;"><span style=3D"color: #b3b3b3;">= <span style=3D"font-family: arial;"><span style=3D"font-size: xx-small;"><s=

</span><span style=3D"font-variant: normal;"><span style=3D"color: #666666= ;"><span style=3D"font-family: Sans serif,sans-serif;"><span style=3D"font-= size: xx-small;"> </span></span></span></span><span style=3D"font-variant: = normal;"><span style=3D"color: #666666;"><span style=3D"font-family: Sans s= erif,sans-serif;"><span style=3D"font-size: xx-small;"><span style=3D"font-=

</span></span></span></span><span style=3D"font-variant: normal;"><span st= yle=3D"color: #666666;"><span style=3D"font-family: Sans serif,sans-serif;"= <span style=3D"font-size: xx-small;"> </span></span></span></span><span st= yle=3D"font-variant: normal;"><span style=3D"color: #666666;"><span style= =3D"font-family: Sans serif,sans-serif;"><span style=3D"font-size: xx-small= ;"><span style=3D"font-style: normal;"><span style=3D"font-weight: normal;"= www.ubimet.com =E2=80=A2 Mobile: +43 664 8323379<br></span></span></span><= /span></span></span></p><p style=3D"margin-bottom: 0cm; font-variant: norma= l; font-style: normal; font-weight: normal;"><span style=3D"color: #666666;= "><span style=3D"font-family: Sans serif,sans-serif;"><span style=3D"font-s= ize: xx-small;">Sitz: Wien =E2=80=A2 Firmenbuchgericht: Handelsgericht Wien= =E2=80=A2 FN 248415 t</span></span></span></p><p style=3D"margin-bottom: 0= cm; font-variant: normal; font-style: normal; font-weight: normal;"><span s= tyle=3D"color: #b3b3b3;"><span style=3D"font-family: arial;"><span style=3D= "font-size: xx-small;">----------------------------------------------------= ----------------------------------------------------------------<br></span>= </span></span></p><div><br></div><p style=3D"margin-bottom: 0cm; font-varia= nt: normal; font-style: normal; font-weight: normal;"><span style=3D"color:= #b3b3b3;"><span style=3D"font-family: arial;"><span style=3D"font-size: xx= -small;"><span style=3D"font-size: xx-small;">The information contained in =

--=_ac5cb6d2-ec02-4080-b902-c991d72cd4e4 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi,=20 yes, still the same issues with oVirt 4.1.7 and CentOS 7.4.=20 Whole cgroup directory has empty files.=20 Actually at the moment, we are blind when it comes to disk IO, only our Net= App give us IO values, but these are per NFS share and there are a lots of = VMs on it.=20 A solution would be great here.=20 BR Florian=20 ---------------------------------------------------------------------------= -----------------------------------------=20 UBIMET GmbH - weather matters=20 Ing. Florian Schmid =E2=80=A2 IT Infrastruktur Austria=20 A-1220 Wien =E2=80=A2 Donau-City-Stra=C3=9Fe 11 =E2=80=A2 Tel +43 1 263 11 = 22 DW 469 =E2=80=A2 Fax +43 1 263 11 22 219=20 fschmid@ubimet.com =E2=80=A2 www.ubimet.com =E2=80=A2 Mobile: +43 664 83233= 79=20 Sitz: Wien =E2=80=A2 Firmenbuchgericht: Handelsgericht Wien =E2=80=A2 FN 24= 8415 t=20 ---------------------------------------------------------------------------= -----------------------------------------=20 The information contained in this message (including any attachments) is co= nfidential and may be legally privileged or otherwise protected from disclo= sure. This message is intended solely for the addressee(s). If you are not = the intended recipient, please notify the sender by return e-mail and delet= e this message from your system. Any unauthorized use, reproduction, or dis= semination of this message is strictly prohibited. Please note that e-mails= are susceptible to change. UBIMET GmbH shall not be liable for the imprope= r or incomplete transmission of the information contained in this communica= tion, nor shall it be liable for any delay in its receipt. UBIMET GmbH acce= pts no liability for loss or damage caused by software viruses and you are = advised to carry out a virus check on any attachments contained in this mes= sage.=20 Von: "Konstantin Shalygin" <k0ste@k0ste.ru>=20 An: "users" <users@ovirt.org>=20 Gesendet: Montag, 18. Dezember 2017 07:17:42=20 Betreff: Re: [ovirt-users] Empty cgroup files on centos 7.3 host=20 Specifically for IO statistics, VDSM reads the values from libvirt[1]. cgroup limiting is possible if you define it, but is unrelated. Also note that 7.3 is a bit ancient, I'm not sure how supported it is with latest 4.1 - which I'm sure will pull new dependencies from 7.4 (for example, libvirt!).=20 We use oVirt 4.1.6 on 7.4 of course.=20 Where I can see IO stat? I never see this on oVirt manager.=20 How can I enable cgroup blk metric collection?=20 Perhaps this is an outdated way and metrics should be collected in a differ= ent way and it should be applied in the netdata project?=20 Thanks.=20 k=20 _______________________________________________=20 Users mailing list=20 Users@ovirt.org=20 http://lists.ovirt.org/mailman/listinfo/users=20 --=_ac5cb6d2-ec02-4080-b902-c991d72cd4e4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hi,</div><div><br data-mce-bogus=3D"1"></di= v><div>yes, still the same issues with oVirt 4.1.7 and CentOS 7.4.</div><di= v>Whole cgroup directory has empty files.</div><div><br data-mce-bogus=3D"1= "></div><div>Actually at the moment, we are blind when it comes to disk IO,= only our NetApp give us IO values, but these are per NFS share and there a= re a lots of VMs on it.</div><div><br data-mce-bogus=3D"1"></div><div>A sol= ution would be great here.</div><div><br data-mce-bogus=3D"1"></div><div>BR= Florian</div><div><br></div><div data-marker=3D"__SIG_PRE__"><div><p style= =3D"margin-bottom: 0cm;"><a class=3D"mceItemAnchor" name=3D"OBJ_PREFIX_DWT8= 057_com_zimbra_url"></a><a class=3D"mceItemAnchor" name=3D"OBJ_PREFIX_DWT10= 86_com_zimbra_url"></a><a class=3D"mceItemAnchor" name=3D"OBJ_PREFIX_DWT805= 8_com_zimbra_url"></a> <span style=3D"color: #b3b3b3;"><img src=3D"http://w= ww.ubimet.com/ubimet_logo.gif" data-mce-src=3D"http://www.ubimet.com/ubimet= _logo.gif" height=3D"18" align=3D"BOTTOM" border=3D"0" width=3D"126"></span= pan style=3D"font-style: normal;"><span style=3D"font-weight: normal;">----= ---------------------------------------------------------------------------= -------------------------------------</span></span></span></span></span></s= pan><span style=3D"font-variant: normal;"><span style=3D"color: #000000;"><= span style=3D"font-family: arial;"><span style=3D"font-size: xx-small;"><sp= an style=3D"font-style: normal;"><span style=3D"font-weight: normal;"><br><= /span></span></span></span></span></span><strong><span style=3D"font-varian= t: normal;"><span style=3D"color: #666666;"><span style=3D"font-family: San= s serif,sans-serif;"><span style=3D"font-size: xx-small;"><span style=3D"fo= nt-style: normal;"><span style=3D"font-weight: normal;">UBIMET GmbH</span><= /span></span></span></span></span></strong><span style=3D"font-variant: nor= mal;"><span style=3D"color: #b3b3b3;"><span style=3D"font-family: Sans seri= f,sans-serif;"><span style=3D"font-size: xx-small;"><span style=3D"font-sty= le: normal;"><span style=3D"font-weight: normal;"> - weather matters&n= bsp;</span></span></span></span></span></span><span style=3D"font-variant: = normal;"><span style=3D"color: #000000;"><span style=3D"font-family: Sans s= erif,sans-serif;"><span style=3D"font-size: xx-small;"><span style=3D"font-= style: normal;"><span style=3D"font-weight: normal;"><br></span></span></sp= an></span></span></span><span style=3D"font-variant: normal;"><span style= =3D"color: #666666;"><span style=3D"font-family: Sans serif,sans-serif;"><s= pan style=3D"font-size: xx-small;"><span style=3D"font-style: normal;"><spa= n style=3D"font-weight: normal;">Ing. Florian Schmid =E2=80=A2 IT Infrastru= ktur Austria</span></span></span></span></span></span><span style=3D"font-v= ariant: normal;"><span style=3D"color: #000000;"><span style=3D"font-family= : Sans serif,sans-serif;"><span style=3D"font-size: xx-small;"><span style= =3D"font-style: normal;"><span style=3D"font-weight: normal;"><br></span></= span></span></span></span></span></p><p style=3D"margin-bottom: 0cm;"><span= style=3D"font-variant: normal;"><span style=3D"color: #000000;"><span styl= e=3D"font-family: Sans serif,sans-serif;"><span style=3D"font-size: xx-smal= l;"><span style=3D"font-style: normal;"><span style=3D"font-weight: normal;= "></span></span></span></span></span></span><span style=3D"font-variant: no= rmal;"><span style=3D"color: #666666;"><span style=3D"font-family: Sans ser= if,sans-serif;"><span style=3D"font-size: xx-small;"><span style=3D"font-st= yle: normal;"><span style=3D"font-weight: normal;">A-1220 Wien =E2=80=A2 Do= nau-City-Stra=C3=9Fe 11 =E2=80=A2 Tel +43 1 263 11 22 DW 469 =E2=80=A2 Fax = +43 1 263 11 22 219<br>fschmid@ubimet.com</span></span></span></span></span= style: normal;"><span style=3D"font-weight: normal;">=E2=80=A2</span></span= this message (including any attachments) is confidential and may be legally= privileged or otherwise protected from disclosure. This message is intende= d solely for the addressee(s). If you are not the intended recipient, pleas= e notify the sender by return e-mail and delete this message from your syst= em. Any unauthorized use, reproduction, or dissemination of this message is= strictly prohibited. Please note that e-mails are susceptible to change. U= BIMET GmbH shall not be liable for the improper or incomplete transmission = of the information contained in this communication, nor shall it be liable = for any delay in its receipt. UBIMET GmbH accepts no liability for loss or = damage caused by software viruses and you are advised to carry out a virus = check on any attachments contained in this message.</span><br></span></span=

</span></p><p style=3D"margin-bottom: 0cm;"><br></p></div></div><br><hr id= =3D"zwchr" data-marker=3D"__DIVIDER__"><div data-marker=3D"__HEADERS__"><b>= Von: </b>"Konstantin Shalygin" <k0ste@k0ste.ru><br><b>An: </b>"users"= <users@ovirt.org><br><b>Gesendet: </b>Montag, 18. Dezember 2017 07:1= 7:42<br><b>Betreff: </b>Re: [ovirt-users] Empty cgroup files on centos 7.3 = host<br></div><br><div data-marker=3D"__QUOTED_TEXT__"><p> </p><blockquote> <pre>Specifically for IO statistics, VDSM reads the values from lib= virt[1]. cgroup limiting is possible if you define it, but is unrelated. Also note that 7.3 is a bit ancient, I'm not sure how supported it is with latest 4.1 - which I'm sure will pull new dependencies from 7.4 (for example, libvirt!).</pre> </blockquote> We use oVirt 4.1.6 on 7.4 of course.<p></p> <p>Where I can see IO stat? I never see this on oVirt manager.</p> <p><span id=3D"result_box" class=3D"" lang=3D"en"><span>How can I enabl= e cgroup blk metric collection?</span><br> <span class=3D"">Perhaps this is an outdated way and metrics should be collected in a different way and it should be applied in the netdata project?</span></span></p> <p><span id=3D"result_box" class=3D"" lang=3D"en"><span class=3D"">Than= ks.</span></span></p> <p><span id=3D"result_box" class=3D"" lang=3D"en"><span class=3D""><br> </span></span></p> <p><span id=3D"result_box" class=3D"" lang=3D"en"><span class=3D"">k<br=

</span></span></p> =20 <br>_______________________________________________<br>Users mailing list<b= r>Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></div=

</div></body></html> --=_ac5cb6d2-ec02-4080-b902-c991d72cd4e4--

This is a multi-part message in MIME format. --------------83640865EABB1E7E1C522067 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 12/18/2017 07:58 PM, Yaniv Kaul wrote:

Indeed. 4.2 provides a comprehensive solution, with integration via Collectd -> fluentd -> Elastic -> Kibana. Y.

E.g. integrated to oVirt or "admin can send metrics to ELK"? --------------83640865EABB1E7E1C522067 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 12/18/2017 07:58 PM, Yaniv Kaul wrote:<br> </div> <blockquote type="cite" cite="mid:CAJgorsZnaFEKSR5iNz24yDwbvTw2_c26TcSemXzSz-WyuNwXbw@mail.gmail.com"> <div dir="ltr"><br> <div class="gmail_extra"> <div class="gmail_quote"> <div>Indeed. 4.2 provides a comprehensive solution, with integration via Collectd -> fluentd -> Elastic -> Kibana.</div> <div>Y.</div> <div> </div> </div> </div> </div> </blockquote> <br> E.g. integrated to oVirt or "admin can send metrics to ELK"?<br> <br> </body> </html> --------------83640865EABB1E7E1C522067--

On Dec 18, 2017 3:06 PM, "Konstantin Shalygin" <k0ste@k0ste.ru> wrote: On 12/18/2017 07:58 PM, Yaniv Kaul wrote: Indeed. 4.2 provides a comprehensive solution, with integration via Collectd -> fluentd -> Elastic -> Kibana. Y. E.g. integrated to oVirt or "admin can send metrics to ELK"? We provide the required scripts to install OpenShift with the EFK stack, configure it and the hosts with all relevant details to connect the two. Note that the metrics store also processes the engine and VDSM logs. Y. Y.

On Mon, Dec 18, 2017 at 4:17 PM, Konstantin Shalygin <k0ste@k0ste.ru> wrote:

On 12/18/2017 09:02 PM, Yaniv Kaul wrote:

...

We provide the required scripts to install OpenShift with the EFK stack, configure it and the hosts with all relevant details to connect the two. Note that the metrics store also processes the engine and VDSM logs.

Good to know. But if I still want to use netdata instead EFK? Some info about how to get blk io metrics from libvirt and/or how to enable cgroups metrics. Anyone?

Get netdata to query libvirt for the IO parameters directly. It should be configured with authentication against libvirt (the same password VDSM uses). Y.

Thanks.

k