11:40 a.m.
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com>
wrote:
=20
thank you for response.
I bold-ed the ones that are listed as "paused".
=20
=20
[root@ovirt1 test vdsm]# virsh -r list --all
Id Name State
----------------------------------------------------
2 puppet.test.j2noc.com running
4 sftp2.test.j2noc.com running
5 oct.test.j2noc.com running
6 sftp2.dev.j2noc.com running
10 darmaster1.test.j2noc.com running
14 api1.test.j2noc.com running
25 ftp1.frb.test.j2noc.com running
26 auto7.test.j2noc.com running
32 epaymv02.j2noc.com running
34 media2.frb.test.j2noc.com running
36 auto2.j2noc.com running
44 nfs.testhvy2.colo.j2noc.com running
53 billapp-zuma1.dev.j2noc.com running
54 billing-ci.dev.j2noc.com running
60 log2.test.j2noc.com running
63 log1.test.j2noc.com running
69 sonar.dev.j2noc.com running
73 billapp-ui1.dev.j2noc.com running
74 billappvm01.dev.j2noc.com running
75 db2.frb.test.j2noc.com running
83 billapp-ui1.test.j2noc.com running
84 epayvm01.test.j2noc.com running
87 billappvm01.test.j2noc.com running
89 etapi1.test.j2noc.com running
93 billapp-zuma2.test.j2noc.com running
94 git.dev.j2noc.com running
=20
Yes I did "systemctl restart libvirtd" which apparently also restart =
vdsm?
yes, it does.=20
=20
=20
Looks like problem started around 2016-04-17 20:19:34,822, based on =
engine.log
attached.
yes, that time looks correct. Any idea what might have been a trigger? =
Anything interesting happened at that time (power outage of some host, =
some maintenance action, anything)?=20
logs indicate a problem when vdsm talks to libvirt(all those "monitor =
become unresponsive=E2=80=9D)
It does seem that at that time you started to have some storage =
connectivity issues - first one at 2016-04-17 20:06:53,929. And it =
doesn=E2=80=99t look temporary because such errors are still there =
couple hours later(in your most recent file you attached I can see at =
23:00:54)
When I/O gets blocked the VMs may experience issues (then VM gets =
Paused), or their qemu process gets stuck(resulting in libvirt either =
reporting error or getting stuck as well -> resulting in what vdsm sees =
as =E2=80=9Cmonitor unresponsive=E2=80=9D)
Since you now bounced libvirtd - did it help? Do you still see wrong =
status for those VMs and still those "monitor unresponsive" errors in =
vdsm.log?
If not=E2=80=A6then I would suspect the =E2=80=9Cvm recovery=E2=80=9D =
code not working correctly. Milan is looking at that.
Thanks,
michal
There's a lot of vdsm logs!
=20
fyi, the storage domain for these Vms is a "local" nfs share, =
7e566f55-e060-47b7-bfa4-ac3c48d70dda.
=20
attached more logs.
=20
=20
On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
>> On 27 Apr 2016, at 19:16, Bill James <bill.james(a)j2.com> =
<mailto:bill.james@j2.com> wrote:
>>=20
>> virsh # list --all
>> error: failed to connect to the hypervisor
>> error: no valid connection
>> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': =
No such file or directory
>>=20
> you need to run virsh in read-only mode
> virsh -r list =E2=80=94all
>=20
>> [root@ovirt1 test vdsm]# systemctl status libvirtd
>> =E2=97=8F libvirtd.service - Virtualization daemon
>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; =
vendor preset: enabled)
>> Drop-In: /etc/systemd/system/libvirtd.service.d
>> =E2=94=94=E2=94=80unlimited-core.conf
>> Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days =
ago
>>=20
>>=20
>> tried systemctl restart libvirtd.
>> No change.
>>=20
>> Attached vdsm.log and supervdsm.log.
>>=20
>>=20
>> [root@ovirt1 test vdsm]# systemctl status vdsmd
>> =E2=97=8F vdsmd.service - Virtual Desktop Server Manager
>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; =
vendor
preset: enabled)
>> Active: active (running) since Wed 2016-04-27 10:09:14 PDT;
3min =
46s ago
>>=20
>>=20
>> vdsm-4.17.18-0.el7.centos.noarch
> the vdsm.log attach is good, but it=E2=80=99s too short interval, it =
only
shows recovery(vdsm restart) phase when the VMs are identified as =
paused=E2=80=A6.can you add earlier logs? Did you restart vdsm yourself =
or did it crash?
>=20
>=20
>> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
>>=20
>>=20
>> Thanks.
>>=20
>>=20
>> On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
>>>> On 27 Apr 2016, at 02:04, Nir Soffer <nsoffer(a)redhat.com> =
<mailto:nsoffer@redhat.com> wrote:
>>>>=20
>>>> jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <bill.james(a)j2.com> =
<mailto:bill.james@j2.com> wrote:
>>>>> I have a hardware node that has 26 VMs.
>>>>> 9 are listed as "running", 17 are listed as
"paused".
>>>>>=20
>>>>> In truth all VMs are up and running fine.
>>>>>=20
>>>>> I tried telling the db they are up:
>>>>>=20
>>>>> engine=3D> update vm_dynamic set status =3D 1 where vm_guid =
=3D(select
>>>>> vm_guid from vm_static where vm_name =3D
'api1.test.j2noc.com');
>>>>>=20
>>>>> GUI then shows it up for a short while,
>>>>>=20
>>>>> then puts it back in paused state.
>>>>>=20
>>>>> 2016-04-26 15:16:46,095 INFO =
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>>>> (DefaultQuartzScheduler_Worker-16) [157cc21e] VM
=
'242ca0af-4ab2-4dd6-b515-5
>>>>> d435e6452c4'(api1.test.j2noc.com) moved from
'Up' --> 'Paused'
>>>>> 2016-04-26 15:16:46,221 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditlogh
>>>>> andling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-16) =
[157cc21e] Cor
>>>>> relation ID: null, Call Stack: null, Custom Event
ID: -1, =
Message: VM api1.
>>>>> test.j2noc.com has been paused.
>>>>>=20
>>>>>=20
>>>>> Why does the engine think the VMs are paused?
>>>>> Attached engine.log.
>>>>>=20
>>>>> I can fix the problem by powering off the VM then starting it =
back up.
>>>>> But the VM is working fine! How do I get ovirt to
realize that?
>>>> If this is an issue in engine, restarting engine may fix this.
>>>> but having this problem only with one node, I don't think this is =
the issue.
>>>>=20
>>>> If this is an issue in vdsm, restarting vdsm may fix this.
>>>>=20
>>>> If this does not help, maybe this is libvirt issue? did you try to =
check vm
>>>> status using virsh?
>>> this looks more likely as it seems such status is being reported
>>> logs would help, vdsm.log at the very least.
>>>=20
>>>> If virsh thinks that the vms are paused, you can try to restart =
libvirtd.
>>>>=20
>>>> Please file a bug about this in any case with engine and vdsm =
logs.
>>>>=20
>>>> Adding Michal in case he has better idea how to proceed.
>>>>=20
>>>> Nir
>>=20
>> Cloud Services for Business www.j2.com <http://www.j2.com/>
>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>=20
>>=20
>> This email, its contents and attachments contain information from j2 =
Global, Inc. and/or its affiliates which may be privileged, confidential =
or otherwise protected from disclosure. The information is intended to =
be for the addressee(s) only. If you are not an addressee, any =
disclosure, copy, distribution, or use of the contents of this message =
is prohibited. If you have received this email in error please notify =
the sender by reply e-mail and delete the original message and any =
copies. (c) 2015 j2 Global, Inc. All rights reserved. eFax, eVoice, =
Campaigner, FuseMail, KeepItSafe, and Onebox are registered trademarks =
of j2 Global, Inc. and its affiliates.
>> =
<supervdsm.log.gz><vdsm.log.gz>___________________________________________=
____
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
<engine.log-20160421.gz><vdsm.logs.tar.gz>
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 28 Apr 2016, at 19:40, Bill James <<a =
href=3D"mailto:bill.james@j2.com"
class=3D"">bill.james(a)j2.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D"">
=20
<meta content=3D"text/html; charset=3DUTF-8" =
http-equiv=3D"Content-Type" class=3D"">
=20
<div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
thank you for response.<br class=3D"">
I bold-ed the ones that are listed as "paused".<br
class=3D"">
<br class=3D"">
<br class=3D"">
[root@ovirt1 test vdsm]# virsh -r list --all<br class=3D"">
Id =
Name &nbs=
p; =
State<br class=3D"">
----------------------------------------------------<br class=3D"">
<b class=3D""> 2
<a =
href=3D"http://puppet.test.j2noc.com" =
class=3D"">puppet.test.j2noc.com</a> &n=
bsp; running</b><br class=3D"">
<b class=3D""> 4
<a =
href=3D"http://sftp2.test.j2noc.com" =
class=3D"">sftp2.test.j2noc.com</a> &nb=
sp; running</b><br class=3D"">
<b class=3D""> 5
<a =
href=3D"http://oct.test.j2noc.com" =
class=3D"">oct.test.j2noc.com</a>  =
; running</b><br
class=3D"">
<b class=3D""> 6
<a =
href=3D"http://sftp2.dev.j2noc.com" =
class=3D"">sftp2.dev.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
<b class=3D""> 10 <a =
href=3D"http://darmaster1.test.j2noc.com" =
class=3D"">darmaster1.test.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 14 <a =
href=3D"http://api1.test.j2noc.com" =
class=3D"">api1.test.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
25 <a
href=3D"http://ftp1.frb.test.j2noc.com" =
class=3D"">ftp1.frb.test.j2noc.com</a> =
running<br class=3D"">
26 <a
href=3D"http://auto7.test.j2noc.com" =
class=3D"">auto7.test.j2noc.com</a> &nb=
sp; running<br class=3D"">
<b class=3D""> 32 <a =
href=3D"http://epaymv02.j2noc.com" =
class=3D"">epaymv02.j2noc.com</a>  =
; running</b><br
class=3D"">
34 <a =
href=3D"http://media2.frb.test.j2noc.com" =
class=3D"">media2.frb.test.j2noc.com</a>
=
running<br class=3D"">
36 <a
href=3D"http://auto2.j2noc.com" =
class=3D"">auto2.j2noc.com</a> &n=
bsp; running<br
class=3D"">
44 <a =
href=3D"http://nfs.testhvy2.colo.j2noc.com" =
class=3D"">nfs.testhvy2.colo.j2noc.com</a>
running<br =
class=3D"">
<b class=3D""> 53 <a =
href=3D"http://billapp-zuma1.dev.j2noc.com" =
class=3D"">billapp-zuma1.dev.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 54 <a =
href=3D"http://billing-ci.dev.j2noc.com" =
class=3D"">billing-ci.dev.j2noc.com</a>  =
; running</b><br class=3D"">
60 <a
href=3D"http://log2.test.j2noc.com" =
class=3D"">log2.test.j2noc.com</a> &nbs=
p; running<br class=3D"">
63 <a
href=3D"http://log1.test.j2noc.com" =
class=3D"">log1.test.j2noc.com</a> &nbs=
p; running<br class=3D"">
<b class=3D""> 69 <a =
href=3D"http://sonar.dev.j2noc.com" =
class=3D"">sonar.dev.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
<b class=3D""> 73 <a =
href=3D"http://billapp-ui1.dev.j2noc.com" =
class=3D"">billapp-ui1.dev.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 74 <a =
href=3D"http://billappvm01.dev.j2noc.com" =
class=3D"">billappvm01.dev.j2noc.com</a>
=
running</b><br class=3D"">
75 <a
href=3D"http://db2.frb.test.j2noc.com" =
class=3D"">db2.frb.test.j2noc.com</a> &=
nbsp; running<br class=3D"">
83 <a =
href=3D"http://billapp-ui1.test.j2noc.com" =
class=3D"">billapp-ui1.test.j2noc.com</a>
=
running<br class=3D"">
<b class=3D""> 84 <a =
href=3D"http://epayvm01.test.j2noc.com" =
class=3D"">epayvm01.test.j2noc.com</a> =
running</b><br class=3D"">
<b class=3D""> 87 <a =
href=3D"http://billappvm01.test.j2noc.com" =
class=3D"">billappvm01.test.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 89 <a =
href=3D"http://etapi1.test.j2noc.com" =
class=3D"">etapi1.test.j2noc.com</a> &n=
bsp; running</b><br class=3D"">
<b class=3D""> 93 <a =
href=3D"http://billapp-zuma2.test.j2noc.com" =
class=3D"">billapp-zuma2.test.j2noc.com</a>
running</b><br =
class=3D"">
<b class=3D""> 94 <a =
href=3D"http://git.dev.j2noc.com" =
class=3D"">git.dev.j2noc.com</a> =
running</b><br
class=3D"">
<br class=3D"">
Yes I did "systemctl restart libvirtd" which apparently also restart
vdsm?<br
class=3D""></div></div></blockquote><div><br =
class=3D""></div>yes, it does. </div><div><br
=
class=3D""></div><div><blockquote type=3D"cite"
class=3D""><div =
class=3D""><div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
<br class=3D"">
<br class=3D"">
Looks like problem started around 2016-04-17 20:19:34,822, based on
engine.log attached.<br
class=3D""></div></div></blockquote><div><br =
class=3D""></div><div>yes, that time looks correct. Any idea what
might =
have been a trigger? Anything interesting happened at that time (power =
outage of some host, some maintenance action, =
anything)? </div><div>logs indicate a problem when vdsm talks to =
libvirt(all those "monitor become unresponsive=E2=80=9D)</div><div><br
=
class=3D""></div><div>It does seem that at that time you started
to have =
some storage connectivity issues - first one at 2016-04-17 =
20:06:53,929. And it doesn=E2=80=99t look temporary because such errors =
are still there couple hours later(in your most recent file you attached =
I can see at 23:00:54)</div><div>When I/O gets blocked the VMs may =
experience issues (then VM gets Paused), or their qemu process gets =
stuck(resulting in libvirt either reporting error or getting stuck as =
well -> resulting in what vdsm sees as =E2=80=9Cmonitor =
unresponsive=E2=80=9D)</div><div><br
class=3D""></div><div>Since you now =
bounced libvirtd - did it help? Do you still see wrong status for those =
VMs and still those "monitor unresponsive" errors in =
vdsm.log?</div><div>If not=E2=80=A6then I would suspect the =E2=80=9Cvm =
recovery=E2=80=9D code not working correctly. Milan is looking at =
that.</div><div><br =
class=3D""></div><div>Thanks,</div><div>michal</div><div><div><br
=
class=3D""></div></div><div class=3D""><br
class=3D""></div><blockquote =
type=3D"cite" class=3D""><div class=3D""><div
bgcolor=3D"#FFFFFF" =
text=3D"#000000" class=3D"">
There's a lot of vdsm logs!<br class=3D"">
<br class=3D"">
fyi, the storage domain for these Vms is a "local" nfs share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br class=3D"">
<br class=3D"">
attached more logs.<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 04/28/2016 12:53 AM, Michal
Skrivanek wrote:<br class=3D"">
</div>
<blockquote =
cite=3D"mid:28BF55E6-3A90-4BB7-90B9-1EE0A82FC460@redhat.com"
type=3D"cite"=
class=3D"">
<pre wrap=3D"" class=3D""></pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">On 27 Apr 2016, at 19:16, Bill
James =
<a class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No =
such file or directory
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">you need to run virsh in
read-only mode
virsh -r list =E2=80=94all
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">[root@ovirt1 test vdsm]#
systemctl =
status libvirtd
=E2=97=8F libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; =
vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
=E2=94=94=E2=94=80unlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
=E2=97=8F vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor =
preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s =
ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">the vdsm.log attach is good, but
it=E2=80=99=
s too short interval, it only shows recovery(vdsm restart) phase when =
the VMs are identified as paused=E2=80=A6.can you add earlier logs? Did =
you restart vdsm yourself or did it crash?
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D""
class=3D"">libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">On 27 Apr 2016, at 02:04,
Nir =
Soffer <a class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a>
wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a =
class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">I have a hardware node
that has =
26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=3D> update vm_dynamic set status =3D 1 where vm_guid =3D(select
vm_guid from vm_static where vm_name =3D '<a =
href=3D"http://api1.test.j2noc.com"
class=3D"">api1.test.j2noc.com</a>');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO =
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM =
'242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(<a href=3D"http://api1.test.j2noc.com" =
class=3D"">api1.test.j2noc.com</a>) moved from 'Up' -->
'Paused'
2016-04-26 15:16:46,221 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] =
Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM =
api1.
<a href=3D"http://test.j2noc.com"
class=3D"">test.j2noc.com</a> has been =
paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">If this is an issue in
engine, =
restarting engine may fix this.
but having this problem only with one node, I don't think this is the =
issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check =
vm
status using virsh?
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">this looks more likely as it
seems =
such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">If virsh thinks that the
vms are =
paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre wrap=3D"" class=3D"">
Cloud Services for Business <a class=3D"moz-txt-link-abbreviated" =
href=3D"http://www.j2.com/">www.j2.com</a>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 =
Global, Inc. and/or its affiliates which may be privileged, confidential =
or otherwise protected from disclosure. The information is intended to =
be for the addressee(s) only. If you are not an addressee, any =
disclosure, copy, distribution, or use of the contents of this message =
is prohibited. If you have received this email in error please notify =
the sender by reply e-mail and delete the original message and any =
copies. (c) 2015 j2 Global, Inc. All rights reserved. eFax, eVoice, =
Campaigner, FuseMail, KeepItSafe, and Onebox are registered trademarks =
of j2 Global, Inc. and its affiliates.
=
<supervdsm.log.gz><vdsm.log.gz>_______________________________=
________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class=3D"moz-txt-link-freetext" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
org/mailman/listinfo/users</a>
</pre>
</blockquote>
<pre wrap=3D"" class=3D""></pre>
</blockquote>
<br class=3D"">
</div>
<span =
id=3D"cid:EB02B488-C070-46FA-9938-DC7D6DF5BEED@brq.redhat.com"><engine.=
log-20160421.gz></span><span =
id=3D"cid:52E27023-A602-4DB0-B69A-18237CC048A3@brq.redhat.com"><vdsm.lo=
gs.tar.gz></span></div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D--