This is a multi-part message in MIME format. --------------62610E9073B056888EB20AFD Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 19.01.2017 21:42, Michael Watters wrote:
Does ovirt have any way to monitor disk I/O for each VM or disk in a storage pool? I am receiving disk latency warnings and would like to know which VMs are causing the most disk I/O.
We have a homebrew IO vm monitoring, libvirt uses cgroups which record CPU and IO stats for each VM. It's a little tricky to follow the VM while it migrates, but once done, we have cpu and IO graphs for each VM. Basicly for each hypervisor we periodicky poll cgroup info for all its VMs: for vm in $vms do ( echo -n "$HOST:$vm:" vm=${vm/-/\\\\x2d} egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_serviced | grep ^253:.*Read | cut -f3 -d " " | paste -sd+ | bc echo -n ":" egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_serviced | grep ^253:.*Write | cut -f3 -d " " | paste -sd+ | bc echo -n ":" egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_service_bytes | grep ^253:.*Read | cut -f3 -d " " | paste -sd+ | bc echo -n ":" egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_service_bytes | grep ^253:.*Write | cut -f3 -d " " | paste -sd+ | bc echo -n ":" cat /sys/fs/cgroup/cpuacct/machine.slice/*$vm*/cpuacct.usage ) | tr -d '\n' echo "" done and then we MRTG it. -- Ernest Beinrohr, AXON PRO Ing <http://www.beinrohr.sk/ing.php>, RHCE <http://www.beinrohr.sk/rhce.php>, RHCVA <http://www.beinrohr.sk/rhce.php>, LPIC <http://www.beinrohr.sk/lpic.php>, VCA <http://www.beinrohr.sk/vca.php>, +421-2-62410360 +421-903-482603 --------------62610E9073B056888EB20AFD Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 19.01.2017 21:42, Michael Watters wrote:<br> </div> <blockquote type="cite" cite="mid:429938d7-4c91-e3f6-ee48-a7a6273c38e5@watters.ws"> <pre wrap="">Does ovirt have any way to monitor disk I/O for each VM or disk in a storage pool? I am receiving disk latency warnings and would like to know which VMs are causing the most disk I/O. </pre> </blockquote> We have a homebrew IO vm monitoring, libvirt uses cgroups which record CPU and IO stats for each VM. It's a little tricky to follow the VM while it migrates, but once done, we have cpu and IO graphs for each VM.<br> <br> Basicly for each hypervisor we periodicky poll cgroup info for all its VMs:<br> <br> <tt>for vm in $vms</tt><tt><br> </tt><tt>do</tt><tt><br> </tt><tt> (</tt><tt><br> </tt><tt> echo -n "$HOST:$vm:"</tt><tt><br> </tt><tt> vm=${vm/-/\\\\x2d}</tt><tt><br> </tt><tt> egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_serviced | grep ^253:.*Read | cut -f3 -d " " | paste -sd+ | bc</tt><tt><br> </tt><tt> echo -n ":"</tt><tt><br> </tt><tt> egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_serviced | grep ^253:.*Write | cut -f3 -d " " | paste -sd+ | bc</tt><tt><br> </tt><tt> echo -n ":"</tt><tt><br> </tt><tt> egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_service_bytes | grep ^253:.*Read | cut -f3 -d " " | paste -sd+ | bc</tt><tt><br> </tt><tt> echo -n ":"</tt><tt><br> </tt><tt> egrep -v "$IGNORED_REGEX" /sys/fs/cgroup/blkio/machine.slice/machine-qemu*$vm*/blkio.throttle.io_service_bytes | grep ^253:.*Write | cut -f3 -d " " | paste -sd+ | bc</tt><tt><br> </tt><tt> echo -n ":"</tt><tt><br> </tt><tt> cat /sys/fs/cgroup/cpuacct/machine.slice/*$vm*/cpuacct.usage</tt><tt><br> </tt><tt> ) | tr -d '\n'</tt><tt><br> </tt><tt> echo ""</tt><tt><br> </tt><tt>done</tt><br> <br> and then we MRTG it.<br> <div class="moz-signature">-- <br> <div id="oernii_footer" style="color: gray;"> <span style="font-family: Lucida Console, Luxi Mono, Courier, monospace; font-size: 90%;"> Ernest Beinrohr, AXON PRO<br> <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/ing.php">Ing</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCE</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCVA</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/lpic.php">LPIC</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/vca.php">VCA</a>, <br> +421-2-62410360 +421-903-482603 <br> </span> </div> <img src="http://nojsstats.appspot.com/UA-44497096-1/email.beinrohr.sk" moz-do-not-send="true" border="0" width="1" height="1"> </div> </body> </html> --------------62610E9073B056888EB20AFD--