2:50 a.m.
--Apple-Mail=_FD55A6CD-BBAC-47AB-A607-BEABC954D30B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 18 Sep 2016, at 18:16, Nir Soffer <nsoffer(a)redhat.com>
wrote:
=20
On Sat, Sep 17, 2016 at 3:14 AM, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
On Fri, Sep 16, 2016 at 10:49 AM, Michal Skrivanek =
<michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
=20
> On 15 Sep 2016, at 18:18, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
>
> Hi all,
>
> Vdsm reports apparentsize and truesize disk stats [1] when getting
> vms stats (every 15 seconds?). These values are update every
> 60 seconds in vdsm.
>
> To collect the values, we run risky storage apis in vdsm virt thread
> pool, and we want to avoid this [2] since one slow or broken domain
> can cause the entire virt thread pool to get stuck and cause vms
> using other (healthy) storage domain to become non responsive.
>
> These can also break block storage thin provisioned disks, since =
they
> depend also on the virt thread pool. So one bad NFS storage
domain
> can cause vms using only block storage to be paused.
>
> If both of these values are not used by anyone, we would like to
> stop reporting them.
=20
there are only 2 consumers of the monitoring, engine and mom.
git grep reveals that =E2=80=9Capparentsize" is used only for =
importing HE.
=E2=80=9Ctruesize" too, but additionally it is used in engine
storage =
code as an actual size of the disk
=20
>
> If the values are used, we need to find a safer way to report them,
> probably in storage thread pool, or maybe we can get these values
> from libvirt using bulk sampling.
=20
you could have been able to drop apparentsize right away, but the HE =
import code
is expecting that field and won=E2=80=99t be happy if it is =
missing
The monitoring code would work but fill in 0 for the actual size
=20
Returning always 0 can be nice prank for the storage team :-)
what about that apparentsize? that seems to be unused except for HE =
import
=20
Looking in bulk stats, we already have the required info from libvirt:
=20
{'bcfa00d3-78a7-40c9-990e-5ffac8886ce0': {'balloon.current': =
1048576L,
'balloon.maximum':
1048576L,
'block.0.allocation': 0L,
'block.0.fl.reqs': 0L,
'block.0.fl.times': 0L,
'block.0.name =
<http://block.0.name/>';: 'hdc',
'block.0.physical':
0L,
'block.0.rd.bytes': 152L,
'block.0.rd.reqs': 4L,
'block.0.rd.times': 539801L,
'block.0.wr.bytes': 0L,
'block.0.wr.reqs': 0L,
'block.0.wr.times': 0L,
'block.1.allocation': =
131005952L,
'block.1.capacity':
=
8589934592L,
'block.1.fl.reqs':
68L,
'block.1.fl.times': =
1894725112L,
'block.1.name =
<http://block.1.name/>';: 'vda',
'block.1.path': =
'/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/5f35b5c0-17d7-4475=
-9125-e97f1cdb06f9/images/c54e7894-b1dc-4f23-9ff5-1836259adc6d/133db162-6c=
6a-4e82-baae-9ae0e7e3885d',
'block.1.physical':
=
1073741824L,
'block.1.rd.bytes':
=
123849728L,
'block.1.rd.reqs':
7979L,
'block.1.rd.times': =
10655381303L,
'block.1.wr.bytes':
=
16762880L,
'block.1.wr.reqs':
455L,
'block.1.wr.times': =
6021639149L,
'block.2.allocation': 0L,
'block.2.capacity': =
21474836480L,
'block.2.fl.reqs':
0L,
'block.2.fl.times': 0L,
'block.2.name =
<http://block.2.name/>';: 'vdb',
'block.2.path': =
'/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/bb85ee2f-d674-489f=
-9377-3eb1f176e8fb/images/b59304f3-d19d-40dd-9f04-8c2df37ef6d3/4df47a96-8a=
1b-436e-8a3e-3a638f119b48',
'block.2.physical':
=
21474836480L,
'block.2.rd.bytes':
=
1389056L,
'block.2.rd.reqs':
331L,
'block.2.rd.times': =
160943568L,
'block.2.wr.bytes':
0L,
'block.2.wr.reqs': 0L,
'block.2.wr.times': 0L,
'block.count': 3,
'cpu.system': 19090000000L,
'cpu.time': 53480823390L,
'cpu.user': 4650000000L,
'net.0.name =
<http://net.0.name/>';: 'vnet0',
'net.0.rx.bytes':
2595857L,
'net.0.rx.drop': 0L,
'net.0.rx.errs': 0L,
'net.0.rx.pkts': 39957L,
'net.0.tx.bytes': 17041L,
'net.0.tx.drop': 0L,
'net.0.tx.errs': 0L,
'net.0.tx.pkts': 177L,
'net.count': 1,
'state.reason': 1,
'state.state': 1,
'vcpu.0.state': 1,
'vcpu.0.time': 43040000000L,
'vcpu.0.wait': 0L,
'vcpu.current': 1,
'vcpu.maximum': 16}}
=20
So we can extract the values from the stats cache matching them using =
drive.path.
=20
We are already doing this for block.*.rd.bytes etc.
=20
Francesco, what do you think?
=20
I check this in https://gerrit.ovirt.org/64093 =
<https://gerrit.ovirt.org/64093>;.
=20
Unfortunately, we cannot use it, since libvirt allocation value
is not compatible with truesize.
=20
truesize is:
- file storage: number of blocks * block size
- block storage: size of lv
=20
Also allocation is available only if qemu has written something
to a volume. When starting a vm with a chain of volumes, all
volumes have allocation=3D0 except the top volume in the boot
disk, not very useful.
=20
So we will have to use the storage apis that do the right thing
for the storage type, but run them in a way that cannot affect
unrelated vms.
=20
Nir
=20
=20
=20
=20
>
> Please update if these values are used in engine/dwh.
>
> [1] =
https://github.com/oVirt/vdsm/blob/master/lib/vdsm/virt/vmstats.py#L364
=
<https://github.com/oVirt/vdsm/blob/master/lib/vdsm/virt/vmstats.py#L364>
> [2] https://gerrit.ovirt.org/59801
<https://gerrit.ovirt.org/59801>
>
> Nir
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
--Apple-Mail=_FD55A6CD-BBAC-47AB-A607-BEABC954D30B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 18 Sep 2016, at 18:16, Nir Soffer <<a =
href=3D"mailto:nsoffer@redhat.com"
class=3D"">nsoffer(a)redhat.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D""><div =
dir=3D"ltr" style=3D"font-family: Helvetica; font-size: 12px; =
font-style: normal; font-variant-caps: normal; font-weight: normal; =
letter-spacing: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; =
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><div
=
class=3D"gmail_extra"><div class=3D"gmail_quote">On Sat, Sep
17, 2016 at =
3:14 AM, Nir Soffer<span =
class=3D"Apple-converted-space"> </span><span
dir=3D"ltr" =
class=3D""><<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt;</span><span =
class=3D"Apple-converted-space"> </span>wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin: 0px 0px =
0px 0.8ex; border-left-width: 1px; border-left-style: solid; =
border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div =
dir=3D"ltr" class=3D""><div
class=3D"gmail_extra"><div =
class=3D"gmail_quote"><span class=3D"gmail-">On Fri, Sep 16,
2016 at =
10:49 AM, Michal Skrivanek<span =
class=3D"Apple-converted-space"> </span><span
dir=3D"ltr" =
class=3D""><<a
href=3D"mailto:michal.skrivanek@redhat.com" =
target=3D"_blank" =
class=3D"">michal.skrivanek(a)redhat.com</a>&gt;</span><span
=
class=3D"Apple-converted-space"> </span>wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin: 0px 0px =
0px 0.8ex; border-left-width: 1px; border-left-style: solid; =
border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span =
class=3D""><br class=3D"">> On 15 Sep 2016, at 18:18,
Nir Soffer =
<<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt; wrote:<br
class=3D"">><br =
class=3D"">> Hi all,<br class=3D"">><br
class=3D"">> Vdsm =
reports apparentsize and truesize disk stats [1] when getting<br =
class=3D"">> vms stats (every 15 seconds?). These values are update =
every<br class=3D"">> 60 seconds in vdsm.<br
class=3D"">><br =
class=3D"">> To collect the values, we run risky storage apis in vdsm
=
virt thread<br class=3D"">> pool, and we want to avoid this [2]
since =
one slow or broken domain<br class=3D"">> can cause the entire virt
=
thread pool to get stuck and cause vms<br class=3D"">> using other
=
(healthy) storage domain to become non responsive.<br
class=3D"">><br =
class=3D"">> These can also break block storage thin provisioned =
disks, since they<br class=3D"">> depend also on the virt thread =
pool. So one bad NFS storage domain<br class=3D"">> can cause vms
=
using only block storage to be paused.<br class=3D"">><br =
class=3D"">> If both of these values are not used by anyone, we would
=
like to<br class=3D"">> stop reporting them.<br
class=3D""><br =
class=3D""></span>there are only 2 consumers of the monitoring, engine
=
and mom.<br class=3D"">git grep reveals that =E2=80=9Capparentsize"
is =
used only for importing HE.<br class=3D"">=E2=80=9Ctruesize" too, but
=
additionally it is used in engine storage code as an actual size of the =
disk<br class=3D""><span class=3D""><br
class=3D"">><br class=3D"">>=
If the values are used, we need to find a safer way to report them,<br =
class=3D"">> probably in storage thread pool, or maybe we can get =
these values<br class=3D"">> from libvirt using bulk
sampling.<br =
class=3D""><br class=3D""></span>you could have been
able to drop =
apparentsize right away, but the HE import code is expecting that field =
and won=E2=80=99t be happy if it is missing<br class=3D"">The monitoring
=
code would work but fill in 0 for the actual size<br =
class=3D""></blockquote><div class=3D""><br
class=3D""></div></span><div =
class=3D"">Returning always 0 can be nice prank for the storage team =
:-)</div></div></div></div></blockquote></div></div></div></div></blockquo=
te><div><br class=3D""></div>what about that apparentsize?
that seems to =
be unused except for HE import</div><div><br
class=3D""><blockquote =
type=3D"cite" class=3D""><div class=3D""><div
dir=3D"ltr" =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px;" class=3D""><div =
class=3D"gmail_extra"><div
class=3D"gmail_quote"><blockquote =
class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; =
border-left-width: 1px; border-left-style: solid; border-left-color: =
rgb(204, 204, 204); padding-left: 1ex;"><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote"><div
class=3D""><br =
class=3D""></div><div class=3D"">Looking in bulk stats,
we already have =
the required info from libvirt:</div><div class=3D""><br =
class=3D""></div><div class=3D""><div =
class=3D""> {'bcfa00d3-78a7-40c9-990e-<wbr
class=3D"">5ffac8886ce0': =
{'balloon.current': 1048576L,</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'balloon.maximum':
=
1048576L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.allocation':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.fl.reqs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.fl.times':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'<a =
href=3D"http://block.0.name/" target=3D"_blank" =
class=3D"">block.0.name</a>': 'hdc',</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.physical':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.rd.bytes':
=
152L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.rd.reqs':
=
4L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.rd.times':
=
539801L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.wr.bytes':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.wr.reqs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.0.wr.times':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.allocation':
=
131005952L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.capacity':
=
8589934592L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.fl.reqs':
=
68L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.fl.times':
=
1894725112L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'<a =
href=3D"http://block.1.name/" target=3D"_blank" =
class=3D"">block.1.name</a>': 'vda',</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.path':
=
'/rhev/data-center/f9374c0e-<wbr =
class=3D"">ae24-4bc1-a596-f61d5f05bc5f/<wbr =
class=3D"">5f35b5c0-17d7-4475-9125-<wbr =
class=3D"">e97f1cdb06f9/images/c54e7894-<wbr =
class=3D"">b1dc-4f23-9ff5-1836259adc6d/<wbr =
class=3D"">133db162-6c6a-4e82-baae-<wbr =
class=3D"">9ae0e7e3885d',</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.physical':
=
1073741824L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.rd.bytes':
=
123849728L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.rd.reqs':
=
7979L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.rd.times':
=
10655381303L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.wr.bytes':
=
16762880L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.wr.reqs':
=
455L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.1.wr.times':
=
6021639149L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.allocation':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.capacity':
=
21474836480L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.fl.reqs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.fl.times':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'<a =
href=3D"http://block.2.name/" target=3D"_blank" =
class=3D"">block.2.name</a>': 'vdb',</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.path':
=
'/rhev/data-center/f9374c0e-<wbr =
class=3D"">ae24-4bc1-a596-f61d5f05bc5f/<wbr =
class=3D"">bb85ee2f-d674-489f-9377-<wbr =
class=3D"">3eb1f176e8fb/images/b59304f3-<wbr =
class=3D"">d19d-40dd-9f04-8c2df37ef6d3/<wbr =
class=3D"">4df47a96-8a1b-436e-8a3e-<wbr =
class=3D"">3a638f119b48',</div><div
class=3D""> =
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.physical':
=
21474836480L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.rd.bytes':
=
1389056L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.rd.reqs':
=
331L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.rd.times':
=
160943568L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.wr.bytes':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.wr.reqs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.2.wr.times':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'block.count':
3,</div><div =
class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'cpu.system':
=
19090000000L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'cpu.time': =
53480823390L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'cpu.user': =
4650000000L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'<a =
href=3D"http://net.0.name/" target=3D"_blank"
class=3D"">net.0.name</a>': =
'vnet0',</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.rx.bytes':
=
2595857L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.rx.drop':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.rx.errs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.rx.pkts':
=
39957L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.tx.bytes':
=
17041L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.tx.drop':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.tx.errs':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.0.tx.pkts':
=
177L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'net.count':
1,</div><div =
class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'state.reason':
=
1,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'state.state':
1,</div><div =
class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'vcpu.0.state':
=
1,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'vcpu.0.time':
=
43040000000L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'vcpu.0.wait':
=
0L,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'vcpu.current':
=
1,</div><div class=3D"">
=
=
<span =
class=3D"Apple-converted-space"> </span>'vcpu.maximum':
=
16}}</div></div><div class=3D""><br
class=3D""></div><div class=3D"">So =
we can extract the values from the stats cache matching them using =
drive.path.</div><div class=3D""><br
class=3D""></div><div class=3D"">We =
are already doing this for block.*.rd.bytes etc.</div><div
class=3D""><br =
class=3D""></div><div class=3D"">Francesco, what do you
=
think?</div></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div><div class=3D"">I check this
in <a =
href=3D"https://gerrit.ovirt.org/64093" =
class=3D"">https://gerrit.ovirt.org/64093</a>.</div...
class=3D""><br =
class=3D""></div><div class=3D"">Unfortunately, we
cannot use it, since =
libvirt allocation value</div><div class=3D"">is not compatible with
=
truesize.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">truesize is:</div><div class=3D"">- file
storage: number of =
blocks * block size</div><div class=3D"">- block storage: size of =
lv</div><div class=3D""><br
class=3D""></div><div class=3D"">Also =
allocation is available only if qemu has written something</div><div =
class=3D"">to a volume. When starting a vm with a chain of volumes, =
all</div><div class=3D"">volumes have allocation=3D0 except the top
=
volume in the boot</div><div class=3D"">disk, not very
useful.</div><div =
class=3D""><br class=3D""></div><div
class=3D"">So we will have to use =
the storage apis that do the right thing</div><div class=3D"">for
the =
storage type, but run them in a way that cannot affect</div><div =
class=3D"">unrelated vms.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Nir</div><div
class=3D""> </div><blockquote =
class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; =
border-left-width: 1px; border-left-style: solid; border-left-color: =
rgb(204, 204, 204); padding-left: 1ex;"><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote"><span =
class=3D"gmail-"><div class=3D""><br
class=3D""></div><div =
class=3D""> </div><blockquote
class=3D"gmail_quote" style=3D"margin: =
0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; =
border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><span =
class=3D""><br class=3D"">><br
class=3D"">> Please update if these =
values are used in engine/dwh.<br class=3D"">><br
class=3D"">> =
[1]<span class=3D"Apple-converted-space"> </span><a =
href=3D"https://github.com/oVirt/vdsm/blob/master/lib/vdsm/virt/vmst...
#L364" rel=3D"noreferrer" target=3D"_blank" =
class=3D"">https://github.com/oVirt/vdsm/<wbr =
class=3D"">blob/master/lib/vdsm/virt/vmst<wbr =
class=3D"">ats.py#L364</a><br class=3D"">>
[2]<span =
class=3D"Apple-converted-space"> </span><a =
href=3D"https://gerrit.ovirt.org/59801" rel=3D"noreferrer" =
target=3D"_blank"
class=3D"">https://gerrit.ovirt.org/59801</a><br =
class=3D"">><br class=3D"">> Nir<br
class=3D""></span>> =
______________________________<wbr class=3D"">_________________<br =
class=3D"">> Devel mailing list<br
class=3D"">><span =
class=3D"Apple-converted-space"> </span><a =
href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel(a)ovirt.org</a><br
class=3D"">><span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel"
rel=3D"noreferrer" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr
=
class=3D"">/listinfo/devel</a></blockquote></span></div></div></div></bloc=
kquote></div></div></div></div></blockquote></div><br
=
class=3D""></body></html>=
--Apple-Mail=_FD55A6CD-BBAC-47AB-A607-BEABC954D30B--