<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 18 Sep 2016, at 18:16, Nir Soffer <<a href="mailto:nsoffer@redhat.com" class="">nsoffer@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Sat, Sep 17, 2016 at 3:14 AM, Nir Soffer<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:nsoffer@redhat.com" target="_blank" class="">nsoffer@redhat.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-">On Fri, Sep 16, 2016 at 10:49 AM, Michal Skrivanek<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:michal.skrivanek@redhat.com" target="_blank" class="">michal.skrivanek@redhat.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span class=""><br class="">> On 15 Sep 2016, at 18:18, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank" class="">nsoffer@redhat.com</a>> wrote:<br class="">><br class="">> Hi all,<br class="">><br class="">> Vdsm reports apparentsize and truesize disk stats [1] when getting<br class="">> vms stats (every 15 seconds?). These values are update every<br class="">> 60 seconds in vdsm.<br class="">><br class="">> To collect the values, we run risky storage apis in vdsm virt thread<br class="">> pool, and we want to avoid this [2] since one slow or broken domain<br class="">> can cause the entire virt thread pool to get stuck and cause vms<br class="">> using other (healthy) storage domain to become non responsive.<br class="">><br class="">> These can also break block storage thin provisioned disks, since they<br class="">> depend also on the virt thread pool. So one bad NFS storage domain<br class="">> can cause vms using only block storage to be paused.<br class="">><br class="">> If both of these values are not used by anyone, we would like to<br class="">> stop reporting them.<br class=""><br class=""></span>there are only 2 consumers of the monitoring, engine and mom.<br class="">git grep reveals that “apparentsize" is used only for importing HE.<br class="">“truesize" too, but additionally it is used in engine storage code as an actual size of the disk<br class=""><span class=""><br class="">><br class="">> If the values are used, we need to find a safer way to report them,<br class="">> probably in storage thread pool, or maybe we can get these values<br class="">> from libvirt using bulk sampling.<br class=""><br class=""></span>you could have been able to drop apparentsize right away, but the HE import code is expecting that field and won’t be happy if it is missing<br class="">The monitoring code would work but fill in 0 for the actual size<br class=""></blockquote><div class=""><br class=""></div></span><div class="">Returning always 0 can be nice prank for the storage team :-)</div></div></div></div></blockquote></div></div></div></div></blockquote><div><br class=""></div>what about that apparentsize? that seems to be unused except for HE import</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class="">Looking in bulk stats, we already have the required info from libvirt:</div><div class=""><br class=""></div><div class=""><div class=""> {'bcfa00d3-78a7-40c9-990e-<wbr class="">5ffac8886ce0': {'balloon.current': 1048576L,</div><div class=""> <span class="Apple-converted-space"> </span>'balloon.maximum': 1048576L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.allocation': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.fl.reqs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.fl.times': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'<a href="http://block.0.name/" target="_blank" class="">block.0.name</a>': 'hdc',</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.physical': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.rd.bytes': 152L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.rd.reqs': 4L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.rd.times': 539801L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.wr.bytes': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.wr.reqs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.0.wr.times': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.allocation': 131005952L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.capacity': 8589934592L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.fl.reqs': 68L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.fl.times': 1894725112L,</div><div class=""> <span class="Apple-converted-space"> </span>'<a href="http://block.1.name/" target="_blank" class="">block.1.name</a>': 'vda',</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.path': '/rhev/data-center/f9374c0e-<wbr class="">ae24-4bc1-a596-f61d5f05bc5f/<wbr class="">5f35b5c0-17d7-4475-9125-<wbr class="">e97f1cdb06f9/images/c54e7894-<wbr class="">b1dc-4f23-9ff5-1836259adc6d/<wbr class="">133db162-6c6a-4e82-baae-<wbr class="">9ae0e7e3885d',</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.physical': 1073741824L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.rd.bytes': 123849728L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.rd.reqs': 7979L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.rd.times': 10655381303L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.wr.bytes': 16762880L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.wr.reqs': 455L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.1.wr.times': 6021639149L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.allocation': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.capacity': 21474836480L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.fl.reqs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.fl.times': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'<a href="http://block.2.name/" target="_blank" class="">block.2.name</a>': 'vdb',</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.path': '/rhev/data-center/f9374c0e-<wbr class="">ae24-4bc1-a596-f61d5f05bc5f/<wbr class="">bb85ee2f-d674-489f-9377-<wbr class="">3eb1f176e8fb/images/b59304f3-<wbr class="">d19d-40dd-9f04-8c2df37ef6d3/<wbr class="">4df47a96-8a1b-436e-8a3e-<wbr class="">3a638f119b48',</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.physical': 21474836480L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.rd.bytes': 1389056L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.rd.reqs': 331L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.rd.times': 160943568L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.wr.bytes': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.wr.reqs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.2.wr.times': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'block.count': 3,</div><div class=""> <span class="Apple-converted-space"> </span>'cpu.system': 19090000000L,</div><div class=""> <span class="Apple-converted-space"> </span>'cpu.time': 53480823390L,</div><div class=""> <span class="Apple-converted-space"> </span>'cpu.user': 4650000000L,</div><div class=""> <span class="Apple-converted-space"> </span>'<a href="http://net.0.name/" target="_blank" class="">net.0.name</a>': 'vnet0',</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.rx.bytes': 2595857L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.rx.drop': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.rx.errs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.rx.pkts': 39957L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.tx.bytes': 17041L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.tx.drop': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.tx.errs': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.0.tx.pkts': 177L,</div><div class=""> <span class="Apple-converted-space"> </span>'net.count': 1,</div><div class=""> <span class="Apple-converted-space"> </span>'state.reason': 1,</div><div class=""> <span class="Apple-converted-space"> </span>'state.state': 1,</div><div class=""> <span class="Apple-converted-space"> </span>'vcpu.0.state': 1,</div><div class=""> <span class="Apple-converted-space"> </span>'vcpu.0.time': 43040000000L,</div><div class=""> <span class="Apple-converted-space"> </span>'vcpu.0.wait': 0L,</div><div class=""> <span class="Apple-converted-space"> </span>'vcpu.current': 1,</div><div class=""> <span class="Apple-converted-space"> </span>'vcpu.maximum': 16}}</div></div><div class=""><br class=""></div><div class="">So we can extract the values from the stats cache matching them using drive.path.</div><div class=""><br class=""></div><div class="">We are already doing this for block.*.rd.bytes etc.</div><div class=""><br class=""></div><div class="">Francesco, what do you think?</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">I check this in <a href="https://gerrit.ovirt.org/64093" class="">https://gerrit.ovirt.org/64093</a>.</div><div class=""><br class=""></div><div class="">Unfortunately, we cannot use it, since libvirt allocation value</div><div class="">is not compatible with truesize.</div><div class=""><br class=""></div><div class="">truesize is:</div><div class="">- file storage: number of blocks * block size</div><div class="">- block storage: size of lv</div><div class=""><br class=""></div><div class="">Also allocation is available only if qemu has written something</div><div class="">to a volume. When starting a vm with a chain of volumes, all</div><div class="">volumes have allocation=0 except the top volume in the boot</div><div class="">disk, not very useful.</div><div class=""><br class=""></div><div class="">So we will have to use the storage apis that do the right thing</div><div class="">for the storage type, but run them in a way that cannot affect</div><div class="">unrelated vms.</div><div class=""><br class=""></div><div class="">Nir</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-"><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span class=""><br class="">><br class="">> Please update if these values are used in engine/dwh.<br class="">><br class="">> [1]<span class="Apple-converted-space"> </span><a href="https://github.com/oVirt/vdsm/blob/master/lib/vdsm/virt/vmstats.py#L364" rel="noreferrer" target="_blank" class="">https://github.com/oVirt/vdsm/<wbr class="">blob/master/lib/vdsm/virt/vmst<wbr class="">ats.py#L364</a><br class="">> [2]<span class="Apple-converted-space"> </span><a href="https://gerrit.ovirt.org/59801" rel="noreferrer" target="_blank" class="">https://gerrit.ovirt.org/59801</a><br class="">><br class="">> Nir<br class=""></span>> ______________________________<wbr class="">_________________<br class="">> Devel mailing list<br class="">><span class="Apple-converted-space"> </span><a href="mailto:Devel@ovirt.org" target="_blank" class="">Devel@ovirt.org</a><br class="">><span class="Apple-converted-space"> </span><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/mailman<wbr class="">/listinfo/devel</a></blockquote></span></div></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></body></html>