11:40 a.m.
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com>
wrote:
=20
thank you for response.
I bold-ed the ones that are listed as "paused".
=20
=20
[root@ovirt1 test vdsm]# virsh -r list --all
Id Name State
----------------------------------------------------
2 puppet.test.j2noc.com running
4 sftp2.test.j2noc.com running
5 oct.test.j2noc.com running
6 sftp2.dev.j2noc.com running
10 darmaster1.test.j2noc.com running
14 api1.test.j2noc.com running
25 ftp1.frb.test.j2noc.com running
26 auto7.test.j2noc.com running
32 epaymv02.j2noc.com running
34 media2.frb.test.j2noc.com running
36 auto2.j2noc.com running
44 nfs.testhvy2.colo.j2noc.com running
53 billapp-zuma1.dev.j2noc.com running
54 billing-ci.dev.j2noc.com running
60 log2.test.j2noc.com running
63 log1.test.j2noc.com running
69 sonar.dev.j2noc.com running
73 billapp-ui1.dev.j2noc.com running
74 billappvm01.dev.j2noc.com running
75 db2.frb.test.j2noc.com running
83 billapp-ui1.test.j2noc.com running
84 epayvm01.test.j2noc.com running
87 billappvm01.test.j2noc.com running
89 etapi1.test.j2noc.com running
93 billapp-zuma2.test.j2noc.com running
94 git.dev.j2noc.com running
=20
Yes I did "systemctl restart libvirtd" which apparently also restart =
vdsm?
yes, it does.=20
=20
=20
Looks like problem started around 2016-04-17 20:19:34,822, based on =
engine.log
attached.
yes, that time looks correct. Any idea what might have been a trigger? =
Anything interesting happened at that time (power outage of some host, =
some maintenance action, anything)?=20
logs indicate a problem when vdsm talks to libvirt(all those "monitor =
become unresponsive=E2=80=9D)
It does seem that at that time you started to have some storage =
connectivity issues - first one at 2016-04-17 20:06:53,929. And it =
doesn=E2=80=99t look temporary because such errors are still there =
couple hours later(in your most recent file you attached I can see at =
23:00:54)
When I/O gets blocked the VMs may experience issues (then VM gets =
Paused), or their qemu process gets stuck(resulting in libvirt either =
reporting error or getting stuck as well -> resulting in what vdsm sees =
as =E2=80=9Cmonitor unresponsive=E2=80=9D)
Since you now bounced libvirtd - did it help? Do you still see wrong =
status for those VMs and still those "monitor unresponsive" errors in =
vdsm.log?
If not=E2=80=A6then I would suspect the =E2=80=9Cvm recovery=E2=80=9D =
code not working correctly. Milan is looking at that.
Thanks,
michal
There's a lot of vdsm logs!
=20
fyi, the storage domain for these Vms is a "local" nfs share, =
7e566f55-e060-47b7-bfa4-ac3c48d70dda.
=20
attached more logs.
=20
=20
On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
>> On 27 Apr 2016, at 19:16, Bill James <bill.james(a)j2.com> =
<mailto:bill.james@j2.com> wrote:
>>=20
>> virsh # list --all
>> error: failed to connect to the hypervisor
>> error: no valid connection
>> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': =
No such file or directory
>>=20
> you need to run virsh in read-only mode
> virsh -r list =E2=80=94all
>=20
>> [root@ovirt1 test vdsm]# systemctl status libvirtd
>> =E2=97=8F libvirtd.service - Virtualization daemon
>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; =
vendor preset: enabled)
>> Drop-In: /etc/systemd/system/libvirtd.service.d
>> =E2=94=94=E2=94=80unlimited-core.conf
>> Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days =
ago
>>=20
>>=20
>> tried systemctl restart libvirtd.
>> No change.
>>=20
>> Attached vdsm.log and supervdsm.log.
>>=20
>>=20
>> [root@ovirt1 test vdsm]# systemctl status vdsmd
>> =E2=97=8F vdsmd.service - Virtual Desktop Server Manager
>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; =
vendor
preset: enabled)
>> Active: active (running) since Wed 2016-04-27 10:09:14 PDT;
3min =
46s ago
>>=20
>>=20
>> vdsm-4.17.18-0.el7.centos.noarch
> the vdsm.log attach is good, but it=E2=80=99s too short interval, it =
only
shows recovery(vdsm restart) phase when the VMs are identified as =
paused=E2=80=A6.can you add earlier logs? Did you restart vdsm yourself =
or did it crash?
>=20
>=20
>> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
>>=20
>>=20
>> Thanks.
>>=20
>>=20
>> On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
>>>> On 27 Apr 2016, at 02:04, Nir Soffer <nsoffer(a)redhat.com> =
<mailto:nsoffer@redhat.com> wrote:
>>>>=20
>>>> jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <bill.james(a)j2.com> =
<mailto:bill.james@j2.com> wrote:
>>>>> I have a hardware node that has 26 VMs.
>>>>> 9 are listed as "running", 17 are listed as
"paused".
>>>>>=20
>>>>> In truth all VMs are up and running fine.
>>>>>=20
>>>>> I tried telling the db they are up:
>>>>>=20
>>>>> engine=3D> update vm_dynamic set status =3D 1 where vm_guid =
=3D(select
>>>>> vm_guid from vm_static where vm_name =3D
'api1.test.j2noc.com');
>>>>>=20
>>>>> GUI then shows it up for a short while,
>>>>>=20
>>>>> then puts it back in paused state.
>>>>>=20
>>>>> 2016-04-26 15:16:46,095 INFO =
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>>>> (DefaultQuartzScheduler_Worker-16) [157cc21e] VM
=
'242ca0af-4ab2-4dd6-b515-5
>>>>> d435e6452c4'(api1.test.j2noc.com) moved from
'Up' --> 'Paused'
>>>>> 2016-04-26 15:16:46,221 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditlogh
>>>>> andling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-16) =
[157cc21e] Cor
>>>>> relation ID: null, Call Stack: null, Custom Event
ID: -1, =
Message: VM api1.
>>>>> test.j2noc.com has been paused.
>>>>>=20
>>>>>=20
>>>>> Why does the engine think the VMs are paused?
>>>>> Attached engine.log.
>>>>>=20
>>>>> I can fix the problem by powering off the VM then starting it =
back up.
>>>>> But the VM is working fine! How do I get ovirt to
realize that?
>>>> If this is an issue in engine, restarting engine may fix this.
>>>> but having this problem only with one node, I don't think this is =
the issue.
>>>>=20
>>>> If this is an issue in vdsm, restarting vdsm may fix this.
>>>>=20
>>>> If this does not help, maybe this is libvirt issue? did you try to =
check vm
>>>> status using virsh?
>>> this looks more likely as it seems such status is being reported
>>> logs would help, vdsm.log at the very least.
>>>=20
>>>> If virsh thinks that the vms are paused, you can try to restart =
libvirtd.
>>>>=20
>>>> Please file a bug about this in any case with engine and vdsm =
logs.
>>>>=20
>>>> Adding Michal in case he has better idea how to proceed.
>>>>=20
>>>> Nir
>>=20
>> Cloud Services for Business www.j2.com <http://www.j2.com/>
>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>=20
>>=20
>> This email, its contents and attachments contain information from j2 =
Global, Inc. and/or its affiliates which may be privileged, confidential =
or otherwise protected from disclosure. The information is intended to =
be for the addressee(s) only. If you are not an addressee, any =
disclosure, copy, distribution, or use of the contents of this message =
is prohibited. If you have received this email in error please notify =
the sender by reply e-mail and delete the original message and any =
copies. (c) 2015 j2 Global, Inc. All rights reserved. eFax, eVoice, =
Campaigner, FuseMail, KeepItSafe, and Onebox are registered trademarks =
of j2 Global, Inc. and its affiliates.
>> =
<supervdsm.log.gz><vdsm.log.gz>___________________________________________=
____
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
<engine.log-20160421.gz><vdsm.logs.tar.gz>
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 28 Apr 2016, at 19:40, Bill James <<a =
href=3D"mailto:bill.james@j2.com"
class=3D"">bill.james(a)j2.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D"">
=20
<meta content=3D"text/html; charset=3DUTF-8" =
http-equiv=3D"Content-Type" class=3D"">
=20
<div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
thank you for response.<br class=3D"">
I bold-ed the ones that are listed as "paused".<br
class=3D"">
<br class=3D"">
<br class=3D"">
[root@ovirt1 test vdsm]# virsh -r list --all<br class=3D"">
Id =
Name &nbs=
p; =
State<br class=3D"">
----------------------------------------------------<br class=3D"">
<b class=3D""> 2
<a =
href=3D"http://puppet.test.j2noc.com" =
class=3D"">puppet.test.j2noc.com</a> &n=
bsp; running</b><br class=3D"">
<b class=3D""> 4
<a =
href=3D"http://sftp2.test.j2noc.com" =
class=3D"">sftp2.test.j2noc.com</a> &nb=
sp; running</b><br class=3D"">
<b class=3D""> 5
<a =
href=3D"http://oct.test.j2noc.com" =
class=3D"">oct.test.j2noc.com</a>  =
; running</b><br
class=3D"">
<b class=3D""> 6
<a =
href=3D"http://sftp2.dev.j2noc.com" =
class=3D"">sftp2.dev.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
<b class=3D""> 10 <a =
href=3D"http://darmaster1.test.j2noc.com" =
class=3D"">darmaster1.test.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 14 <a =
href=3D"http://api1.test.j2noc.com" =
class=3D"">api1.test.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
25 <a
href=3D"http://ftp1.frb.test.j2noc.com" =
class=3D"">ftp1.frb.test.j2noc.com</a> =
running<br class=3D"">
26 <a
href=3D"http://auto7.test.j2noc.com" =
class=3D"">auto7.test.j2noc.com</a> &nb=
sp; running<br class=3D"">
<b class=3D""> 32 <a =
href=3D"http://epaymv02.j2noc.com" =
class=3D"">epaymv02.j2noc.com</a>  =
; running</b><br
class=3D"">
34 <a =
href=3D"http://media2.frb.test.j2noc.com" =
class=3D"">media2.frb.test.j2noc.com</a>
=
running<br class=3D"">
36 <a
href=3D"http://auto2.j2noc.com" =
class=3D"">auto2.j2noc.com</a> &n=
bsp; running<br
class=3D"">
44 <a =
href=3D"http://nfs.testhvy2.colo.j2noc.com" =
class=3D"">nfs.testhvy2.colo.j2noc.com</a>
running<br =
class=3D"">
<b class=3D""> 53 <a =
href=3D"http://billapp-zuma1.dev.j2noc.com" =
class=3D"">billapp-zuma1.dev.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 54 <a =
href=3D"http://billing-ci.dev.j2noc.com" =
class=3D"">billing-ci.dev.j2noc.com</a>  =
; running</b><br class=3D"">
60 <a
href=3D"http://log2.test.j2noc.com" =
class=3D"">log2.test.j2noc.com</a> &nbs=
p; running<br class=3D"">
63 <a
href=3D"http://log1.test.j2noc.com" =
class=3D"">log1.test.j2noc.com</a> &nbs=
p; running<br class=3D"">
<b class=3D""> 69 <a =
href=3D"http://sonar.dev.j2noc.com" =
class=3D"">sonar.dev.j2noc.com</a> &nbs=
p; running</b><br
class=3D"">
<b class=3D""> 73 <a =
href=3D"http://billapp-ui1.dev.j2noc.com" =
class=3D"">billapp-ui1.dev.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 74 <a =
href=3D"http://billappvm01.dev.j2noc.com" =
class=3D"">billappvm01.dev.j2noc.com</a>
=
running</b><br class=3D"">
75 <a
href=3D"http://db2.frb.test.j2noc.com" =
class=3D"">db2.frb.test.j2noc.com</a> &=
nbsp; running<br class=3D"">
83 <a =
href=3D"http://billapp-ui1.test.j2noc.com" =
class=3D"">billapp-ui1.test.j2noc.com</a>
=
running<br class=3D"">
<b class=3D""> 84 <a =
href=3D"http://epayvm01.test.j2noc.com" =
class=3D"">epayvm01.test.j2noc.com</a> =
running</b><br class=3D"">
<b class=3D""> 87 <a =
href=3D"http://billappvm01.test.j2noc.com" =
class=3D"">billappvm01.test.j2noc.com</a>
=
running</b><br class=3D"">
<b class=3D""> 89 <a =
href=3D"http://etapi1.test.j2noc.com" =
class=3D"">etapi1.test.j2noc.com</a> &n=
bsp; running</b><br class=3D"">
<b class=3D""> 93 <a =
href=3D"http://billapp-zuma2.test.j2noc.com" =
class=3D"">billapp-zuma2.test.j2noc.com</a>
running</b><br =
class=3D"">
<b class=3D""> 94 <a =
href=3D"http://git.dev.j2noc.com" =
class=3D"">git.dev.j2noc.com</a> =
running</b><br
class=3D"">
<br class=3D"">
Yes I did "systemctl restart libvirtd" which apparently also restart
vdsm?<br
class=3D""></div></div></blockquote><div><br =
class=3D""></div>yes, it does. </div><div><br
=
class=3D""></div><div><blockquote type=3D"cite"
class=3D""><div =
class=3D""><div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
<br class=3D"">
<br class=3D"">
Looks like problem started around 2016-04-17 20:19:34,822, based on
engine.log attached.<br
class=3D""></div></div></blockquote><div><br =
class=3D""></div><div>yes, that time looks correct. Any idea what
might =
have been a trigger? Anything interesting happened at that time (power =
outage of some host, some maintenance action, =
anything)? </div><div>logs indicate a problem when vdsm talks to =
libvirt(all those "monitor become unresponsive=E2=80=9D)</div><div><br
=
class=3D""></div><div>It does seem that at that time you started
to have =
some storage connectivity issues - first one at 2016-04-17 =
20:06:53,929. And it doesn=E2=80=99t look temporary because such errors =
are still there couple hours later(in your most recent file you attached =
I can see at 23:00:54)</div><div>When I/O gets blocked the VMs may =
experience issues (then VM gets Paused), or their qemu process gets =
stuck(resulting in libvirt either reporting error or getting stuck as =
well -> resulting in what vdsm sees as =E2=80=9Cmonitor =
unresponsive=E2=80=9D)</div><div><br
class=3D""></div><div>Since you now =
bounced libvirtd - did it help? Do you still see wrong status for those =
VMs and still those "monitor unresponsive" errors in =
vdsm.log?</div><div>If not=E2=80=A6then I would suspect the =E2=80=9Cvm =
recovery=E2=80=9D code not working correctly. Milan is looking at =
that.</div><div><br =
class=3D""></div><div>Thanks,</div><div>michal</div><div><div><br
=
class=3D""></div></div><div class=3D""><br
class=3D""></div><blockquote =
type=3D"cite" class=3D""><div class=3D""><div
bgcolor=3D"#FFFFFF" =
text=3D"#000000" class=3D"">
There's a lot of vdsm logs!<br class=3D"">
<br class=3D"">
fyi, the storage domain for these Vms is a "local" nfs share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br class=3D"">
<br class=3D"">
attached more logs.<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 04/28/2016 12:53 AM, Michal
Skrivanek wrote:<br class=3D"">
</div>
<blockquote =
cite=3D"mid:28BF55E6-3A90-4BB7-90B9-1EE0A82FC460@redhat.com"
type=3D"cite"=
class=3D"">
<pre wrap=3D"" class=3D""></pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">On 27 Apr 2016, at 19:16, Bill
James =
<a class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No =
such file or directory
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">you need to run virsh in
read-only mode
virsh -r list =E2=80=94all
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">[root@ovirt1 test vdsm]#
systemctl =
status libvirtd
=E2=97=8F libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; =
vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
=E2=94=94=E2=94=80unlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
=E2=97=8F vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor =
preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s =
ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">the vdsm.log attach is good, but
it=E2=80=99=
s too short interval, it only shows recovery(vdsm restart) phase when =
the VMs are identified as paused=E2=80=A6.can you add earlier logs? Did =
you restart vdsm yourself or did it crash?
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D""
class=3D"">libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">On 27 Apr 2016, at 02:04,
Nir =
Soffer <a class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a>
wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a =
class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">I have a hardware node
that has =
26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=3D> update vm_dynamic set status =3D 1 where vm_guid =3D(select
vm_guid from vm_static where vm_name =3D '<a =
href=3D"http://api1.test.j2noc.com"
class=3D"">api1.test.j2noc.com</a>');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO =
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM =
'242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(<a href=3D"http://api1.test.j2noc.com" =
class=3D"">api1.test.j2noc.com</a>) moved from 'Up' -->
'Paused'
2016-04-26 15:16:46,221 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] =
Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM =
api1.
<a href=3D"http://test.j2noc.com"
class=3D"">test.j2noc.com</a> has been =
paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">If this is an issue in
engine, =
restarting engine may fix this.
but having this problem only with one node, I don't think this is the =
issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check =
vm
status using virsh?
</pre>
</blockquote>
<pre wrap=3D"" class=3D"">this looks more likely as it
seems =
such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type=3D"cite" class=3D"">
<pre wrap=3D"" class=3D"">If virsh thinks that the
vms are =
paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre wrap=3D"" class=3D"">
Cloud Services for Business <a class=3D"moz-txt-link-abbreviated" =
href=3D"http://www.j2.com/">www.j2.com</a>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 =
Global, Inc. and/or its affiliates which may be privileged, confidential =
or otherwise protected from disclosure. The information is intended to =
be for the addressee(s) only. If you are not an addressee, any =
disclosure, copy, distribution, or use of the contents of this message =
is prohibited. If you have received this email in error please notify =
the sender by reply e-mail and delete the original message and any =
copies. (c) 2015 j2 Global, Inc. All rights reserved. eFax, eVoice, =
Campaigner, FuseMail, KeepItSafe, and Onebox are registered trademarks =
of j2 Global, Inc. and its affiliates.
=
<supervdsm.log.gz><vdsm.log.gz>_______________________________=
________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class=3D"moz-txt-link-freetext" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
org/mailman/listinfo/users</a>
</pre>
</blockquote>
<pre wrap=3D"" class=3D""></pre>
</blockquote>
<br class=3D"">
</div>
<span =
id=3D"cid:EB02B488-C070-46FA-9938-DC7D6DF5BEED@brq.redhat.com"><engine.=
log-20160421.gz></span><span =
id=3D"cid:52E27023-A602-4DB0-B69A-18237CC048A3@brq.redhat.com"><vdsm.lo=
gs.tar.gz></span></div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_FC29514C-FC8B-44E8-A9D8-B78C0939837D--
7:26 p.m.
New subject: vms in paused state
--------------010504060305050806040805
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
yes they are still saying "paused" state.
No, bouncing libvirt didn't help.
I noticed the errors about the ISO domain. Didn't think that was related.
I have been migrating a lot of VMs to ovirt lately, and recently added
another node.
Also had some problems with /etc/exports for a while, but I think those
issues are all resolved.
Last "unresponsive" message in vdsm.log was:
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::*2016-04-21*
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor become unresponsive
(command timeout, age=310323.97)
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor become unresponsive
(command timeout, age=310323.97)
Thanks.
On 4/29/16 1:40 AM, Michal Skrivanek wrote:
> On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com
> <mailto:bill.james@j2.com>> wrote:
>
> thank you for response.
> I bold-ed the ones that are listed as "paused".
>
>
> [root@ovirt1 test vdsm]# virsh -r list --all
> Id Name State
> ----------------------------------------------------
>
>
> Looks like problem started around 2016-04-17 20:19:34,822, based on
> engine.log attached.
yes, that time looks correct. Any idea what might have been a trigger?
Anything interesting happened at that time (power outage of some host,
some maintenance action, anything)?
logs indicate a problem when vdsm talks to libvirt(all those "monitor
become unresponsiveâ)
It does seem that at that time you started to have some storage
connectivity issues - first one at 2016-04-17 20:06:53,929. And it
doesnât look temporary because such errors are still there couple
hours later(in your most recent file you attached I can see at 23:00:54)
When I/O gets blocked the VMs may experience issues (then VM gets
Paused), or their qemu process gets stuck(resulting in libvirt either
reporting error or getting stuck as well -> resulting in what vdsm
sees as âmonitor unresponsiveâ)
Since you now bounced libvirtd - did it help? Do you still see wrong
status for those VMs and still those "monitor unresponsive" errors in
vdsm.log?
If notâŠthen I would suspect the âvm recoveryâ code not working
correctly. Milan is looking at that.
Thanks,
michal
> There's a lot of vdsm logs!
>
> fyi, the storage domain for these Vms is a "local" nfs share,
> 7e566f55-e060-47b7-bfa4-ac3c48d70dda.
>
> attached more logs.
>
>
> On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
>>> On 27 Apr 2016, at 19:16, Bill James<bill.james(a)j2.com> wrote:
>>>
>>> virsh # list --all
>>> error: failed to connect to the hypervisor
>>> error: no valid connection
>>> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
>>>
>> you need to run virsh in read-only mode
>> virsh -r list âall
>>
>>> [root@ovirt1 test vdsm]# systemctl status libvirtd
>>> â libvirtd.service - Virtualization daemon
>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor
preset: enabled)
>>> Drop-In: /etc/systemd/system/libvirtd.service.d
>>> ââunlimited-core.conf
>>> Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
>>>
>>>
>>> tried systemctl restart libvirtd.
>>> No change.
>>>
>>> Attached vdsm.log and supervdsm.log.
>>>
>>>
>>> [root@ovirt1 test vdsm]# systemctl status vdsmd
>>> â vdsmd.service - Virtual Desktop Server Manager
>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: enabled)
>>> Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
>>>
>>>
>>> vdsm-4.17.18-0.el7.centos.noarch
>> the vdsm.log attach is good, but itâs too short interval, it only shows
recovery(vdsm restart) phase when the VMs are identified as pausedâŠ.can you add earlier
logs? Did you restart vdsm yourself or did it crash?
>>
>>
>>> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
>>>
>>>
>>> Thanks.
>>>
>>>
>>> On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
>>>>> On 27 Apr 2016, at 02:04, Nir Soffer<nsoffer(a)redhat.com>
wrote:
>>>>>
>>>>> jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill
James<bill.james(a)j2.com> wrote:
>>>>>> I have a hardware node that has 26 VMs.
>>>>>> 9 are listed as "running", 17 are listed as
"paused".
>>>>>>
>>>>>> In truth all VMs are up and running fine.
>>>>>>
>>>>>> I tried telling the db they are up:
>>>>>>
>>>>>> engine=> update vm_dynamic set status = 1 where vm_guid
=(select
>>>>>> vm_guid from vm_static where vm_name = 'api1.test.j2noc.com
<http://api1.test.j2noc.com>');
>>>>>>
>>>>>> GUI then shows it up for a short while,
>>>>>>
>>>>>> then puts it back in paused state.
>>>>>>
>>>>>> 2016-04-26 15:16:46,095 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>>>>> (DefaultQuartzScheduler_Worker-16) [157cc21e] VM
'242ca0af-4ab2-4dd6-b515-5
>>>>>> d435e6452c4'(api1.test.j2noc.com
<http://api1.test.j2noc.com>) moved from 'Up' --> 'Paused'
>>>>>> 2016-04-26 15:16:46,221 INFO
[org.ovirt.engine.core.dal.dbbroker.auditlogh
>>>>>> andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16)
[157cc21e] Cor
>>>>>> relation ID: null, Call Stack: null, Custom Event ID: -1,
Message: VM api1.
>>>>>> test.j2noc.com <http://test.j2noc.com> has been paused.
>>>>>>
>>>>>>
>>>>>> Why does the engine think the VMs are paused?
>>>>>> Attached engine.log.
>>>>>>
>>>>>> I can fix the problem by powering off the VM then starting it
back up.
>>>>>> But the VM is working fine! How do I get ovirt to realize that?
>>>>> If this is an issue in engine, restarting engine may fix this.
>>>>> but having this problem only with one node, I don't think this is
the issue.
>>>>>
>>>>> If this is an issue in vdsm, restarting vdsm may fix this.
>>>>>
>>>>> If this does not help, maybe this is libvirt issue? did you try to
check vm
>>>>> status using virsh?
>>>> this looks more likely as it seems such status is being reported
>>>> logs would help, vdsm.log at the very least.
>>>>
>>>>> If virsh thinks that the vms are paused, you can try to restart
libvirtd.
>>>>>
>>>>> Please file a bug about this in any case with engine and vdsm logs.
>>>>>
>>>>> Adding Michal in case he has better idea how to proceed.
>>>>>
>>>>> Nir
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>
> <engine.log-20160421.gz><vdsm.logs.tar.gz>
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------010504060305050806040805
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
yes they are still saying "paused" state.<br>
No, bouncing libvirt didn't help.<br>
<br>
I noticed the errors about the ISO domain. Didn't think that was
related.<br>
I have been migrating a lot of VMs to ovirt lately, and recently
added another node.<br>
Also had some problems with /etc/exports for a while, but I think
those issues are all resolved.<br>
<br>
<br>
Last "unresponsive" message in vdsm.log was:<br>
<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::<b>2016-04-21</b>
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor become
unresponsive (command timeout, age=310323.97)<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor become
unresponsive (command timeout, age=310323.97)<br>
<br>
<br>
<br>
Thanks.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 4/29/16 1:40 AM, Michal Skrivanek
wrote:<br>
</div>
<blockquote
cite="mid:656BFC5C-A6F5-4332-90AC-C039D4E9170E@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 28 Apr 2016, at 19:40, Bill James
<<a
moz-do-not-send="true" href="mailto:bill.james@j2.com"
class=""><a class="moz-txt-link-abbreviated"
href="mailto:bill.james@j2.com">bill.james@j2.com</a></a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class=""> thank you
for response.<br class="">
I bold-ed the ones that are listed as "paused".<br
class="">
<br class="">
<br class="">
[root@ovirt1 test vdsm]# virsh -r list --all<br class="">
 Id   Name                         Â
State<br class="">
----------------------------------------------------<br
class="">
</div>
</div>
</blockquote>
</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class=""> <br
class="">
<br class="">
Looks like problem started around 2016-04-17 20:19:34,822,
based on engine.log attached.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>yes, that time looks correct. Any idea what might have been
a trigger? Anything interesting happened at that time (power
outage of some host, some maintenance action, anything)? </div>
<div>logs indicate a problem when vdsm talks to libvirt(all
those "monitor become unresponsiveâ)</div>
<div><br class="">
</div>
<div>It does seem that at that time you started to have some
storage connectivity issues - first one at 2016-04-17
20:06:53,929. And it doesnât look temporary because such
errors are still there couple hours later(in your most recent
file you attached I can see at 23:00:54)</div>
<div>When I/O gets blocked the VMs may experience issues (then
VM gets Paused), or their qemu process gets stuck(resulting in
libvirt either reporting error or getting stuck as well ->
resulting in what vdsm sees as âmonitor unresponsiveâ)</div>
<div><br class="">
</div>
<div>Since you now bounced libvirtd - did it help? Do you still
see wrong status for those VMs and still those "monitor
unresponsive" errors in vdsm.log?</div>
<div>If notâŠthen I would suspect the âvm recoveryâ code not
working correctly. Milan is looking at that.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div>michal</div>
<div>
<div><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class=""> There's a
lot of vdsm logs!<br class="">
<br class="">
fyi, the storage domain for these Vms is a "local" nfs
share, 7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br class="">
<br class="">
attached more logs.<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/28/2016 12:53 AM,
Michal Skrivanek wrote:<br class="">
</div>
<blockquote
cite="mid:28BF55E6-3A90-4BB7-90B9-1EE0A82FC460@redhat.com"
type="cite" class="">
<blockquote type="cite" class="">
<pre class="" wrap="">On 27 Apr 2016, at
19:16, Bill James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file
or directory
</pre>
</blockquote>
<pre class="" wrap="">you need to run virsh in
read-only mode
virsh -r list âall
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">[root@ovirt1 test vdsm]#
systemctl status libvirtd
â libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset:
enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
ââunlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
â vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre class="" wrap="">the vdsm.log attach is
good, but itâs too short interval, it only shows recovery(vdsm restart) phase when the
VMs are identified as pausedâŠ.can you add earlier logs? Did you restart vdsm yourself or
did it crash?
</pre>
<blockquote type="cite" class="">
<pre class=""
wrap="">libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<pre class="" wrap="">On 27 Apr 2016, at
02:04, Nir Soffer <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a>
wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">I have a hardware
node that has 26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=> update vm_dynamic set status = 1 where vm_guid =(select
vm_guid from vm_static where vm_name = '<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
class="">api1.test.j2noc.com</a>');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM '242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
class="">api1.test.j2noc.com</a>) moved from 'Up' -->
'Paused'
2016-04-26 15:16:46,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM api1.
<a moz-do-not-send="true" href="http://test.j2noc.com"
class="">test.j2noc.com</a> has been paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre class="" wrap="">If this is an issue
in engine, restarting engine may fix this.
but having this problem only with one node, I don't think this is the issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check vm
status using virsh?
</pre>
</blockquote>
<pre class="" wrap="">this looks more likely
as it seems such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">If virsh thinks that
the vms are paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre class="" wrap="">
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
</blockquote>
<br class="">
</div>
<span
id="cid:EB02B488-C070-46FA-9938-DC7D6DF5BEED@brq.redhat.com"><engine.log-20160421.gz></span><span
id="cid:52E27023-A602-4DB0-B69A-18237CC048A3@brq.redhat.com"><vdsm.logs.tar.gz></span></div>
</blockquote>
</div>
<br class="">
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------010504060305050806040805--
![src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...](http://src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Footer.jpg%22){kind=link}
8:02 p.m.
New subject: vms in paused state
--Apple-Mail-E8504E27-508B-4D91-9E91-823A22F58109
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: base64
DQoNCj4gT24gMjkgQXByIDIwMTYsIGF0IDE4OjI2LCBCaWxsIEphbWVzIDxiaWxsLmphbWVzQGoy
LmNvbT4gd3JvdGU6DQo+IA0KPiB5ZXMgdGhleSBhcmUgc3RpbGwgc2F5aW5nICJwYXVzZWQiIHN0
YXRlLg0KPiBObywgYm91bmNpbmcgbGlidmlydCBkaWRuJ3QgaGVscC4NCg0KVGhlbiBteSBzdXNw
aWNpb24gb2Ygdm0gcmVjb3ZlcnkgZ2V0cyBjbG9zZXIgdG8gYSBjZXJ0YWludHk6KQ0KQ2FuIHlv
dSBnZXQgb25lIG9mIHRoZSBwYXVzZWQgdm0ncyAucmVjb3ZlcnkgZmlsZSBmcm9tIC92YXIvbGli
L3Zkc20gYW5kIGNoZWNrIGl0IHNheXMgUGF1c2VkIHRoZXJlPyBJdCdzIHdvcnRoIGEgc2hvdCB0
byB0cnkgdG8gcmVtb3ZlIHRoYXQgZmlsZSBhbmQgcmVzdGFydCB2ZHNtLCB0aGVuIGNoZWNrIGxv
Z3MgYW5kIHRoYXQgdm0gc3RhdHVzLi4uaXQgc2hvdWxkIHJlY292ZXIgImdvb2QgZW5vdWdoIiBm
cm9tIGxpYnZpcnQgb25seS4gDQpUcnkgaXQgd2l0aCBvbmUgZmlyc3QNCg0KPiBJIG5vdGljZWQg
dGhlIGVycm9ycyBhYm91dCB0aGUgSVNPIGRvbWFpbi4gRGlkbid0IHRoaW5rIHRoYXQgd2FzIHJl
bGF0ZWQuDQo+IEkgaGF2ZSBiZWVuIG1pZ3JhdGluZyBhIGxvdCBvZiBWTXMgdG8gb3ZpcnQgbGF0
ZWx5LCBhbmQgcmVjZW50bHkgYWRkZWQgYW5vdGhlciBub2RlLg0KPiBBbHNvIGhhZCBzb21lIHBy
b2JsZW1zIHdpdGggL2V0Yy9leHBvcnRzIGZvciBhIHdoaWxlLCBidXQgSSB0aGluayB0aG9zZSBp
c3N1ZXMgYXJlIGFsbCByZXNvbHZlZC4NCj4gDQo+IA0KPiBMYXN0ICJ1bnJlc3BvbnNpdmUiIG1l
c3NhZ2UgaW4gdmRzbS5sb2cgd2FzOg0KPiANCj4gdmRzbS5sb2cuNDkueHo6anNvbnJwYy5FeGVj
dXRvci8wOjpXQVJOSU5HOjoyMDE2LTA0LTIxIDExOjAwOjU0LDcwMzo6dm06OjUwNjc6OnZpcnQu
dm06Oihfc2V0VW5yZXNwb25zaXZlSWZUaW1lb3V0KSB2bUlkPWBiNmExMzgwOC05NTUyLTQwMWIt
ODQwYi00ZjcwMjJlODI5M2RgOjptb25pdG9yIGJlY29tZSB1bnJlc3BvbnNpdmUgKGNvbW1hbmQg
dGltZW91dCwgYWdlPTMxMDMyMy45NykNCj4gdmRzbS5sb2cuNDkueHo6anNvbnJwYy5FeGVjdXRv
ci8wOjpXQVJOSU5HOjoyMDE2LTA0LTIxIDExOjAwOjU0LDcwMzo6dm06OjUwNjc6OnZpcnQudm06
Oihfc2V0VW5yZXNwb25zaXZlSWZUaW1lb3V0KSB2bUlkPWA1YmZiMTQwYS1hOTcxLTRjOWMtODJj
Ni0yNzc5MjllYjQ1ZDRgOjptb25pdG9yIGJlY29tZSB1bnJlc3BvbnNpdmUgKGNvbW1hbmQgdGlt
ZW91dCwgYWdlPTMxMDMyMy45NykNCj4gDQo+IA0KPiANCj4gVGhhbmtzLg0KPiANCj4gDQo+IA0K
Pj4gT24gNC8yOS8xNiAxOjQwIEFNLCBNaWNoYWwgU2tyaXZhbmVrIHdyb3RlOg0KPj4gDQo+Pj4g
T24gMjggQXByIDIwMTYsIGF0IDE5OjQwLCBCaWxsIEphbWVzIDxiaWxsLmphbWVzQGoyLmNvbT4g
d3JvdGU6DQo+Pj4gDQo+Pj4gdGhhbmsgeW91IGZvciByZXNwb25zZS4NCj4+PiBJIGJvbGQtZWQg
dGhlIG9uZXMgdGhhdCBhcmUgbGlzdGVkIGFzICJwYXVzZWQiLg0KPj4+IA0KPj4+IA0KPj4+IFty
b290QG92aXJ0MSB0ZXN0IHZkc21dIyB2aXJzaCAtciBsaXN0IC0tYWxsDQo+Pj4gw4IgSWTDgiDD
giDDgiAgTmFtZcOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOC
IMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCIMOCICBTdGF0ZQ0KPj4+IC0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+IA0KPj4gDQo+Pj4gDQo+
Pj4gDQo+Pj4gTG9va3MgbGlrZSBwcm9ibGVtIHN0YXJ0ZWQgYXJvdW5kIDIwMTYtMDQtMTcgMjA6
MTk6MzQsODIyLCBiYXNlZCBvbiBlbmdpbmUubG9nIGF0dGFjaGVkLg0KPj4gDQo+PiB5ZXMsIHRo
YXQgdGltZSBsb29rcyBjb3JyZWN0LiBBbnkgaWRlYSB3aGF0IG1pZ2h0IGhhdmUgYmVlbiBhIHRy
aWdnZXI/IEFueXRoaW5nIGludGVyZXN0aW5nIGhhcHBlbmVkIGF0IHRoYXQgdGltZSAocG93ZXIg
b3V0YWdlIG9mIHNvbWUgaG9zdCwgc29tZSBtYWludGVuYW5jZSBhY3Rpb24sIGFueXRoaW5nKT/D
giANCj4+IGxvZ3MgaW5kaWNhdGUgYSBwcm9ibGVtIHdoZW4gdmRzbSB0YWxrcyB0byBsaWJ2aXJ0
KGFsbCB0aG9zZSAibW9uaXRvciBiZWNvbWUgdW5yZXNwb25zaXZlw6LigqzCnSkNCj4+IA0KPj4g
SXQgZG9lcyBzZWVtIHRoYXQgYXQgdGhhdCB0aW1lIHlvdSBzdGFydGVkIHRvIGhhdmUgc29tZSBz
dG9yYWdlIGNvbm5lY3Rpdml0eSBpc3N1ZXMgLSBmaXJzdCBvbmUgYXTDgiAyMDE2LTA0LTE3IDIw
OjA2OjUzLDkyOS4gQW5kIGl0IGRvZXNuw6LigqzihKJ0IGxvb2sgdGVtcG9yYXJ5IGJlY2F1c2Ug
c3VjaCBlcnJvcnMgYXJlIHN0aWxsIHRoZXJlIGNvdXBsZSBob3VycyBsYXRlcihpbiB5b3VyIG1v
c3QgcmVjZW50IGZpbGUgeW91IGF0dGFjaGVkIEkgY2FuIHNlZSBhdCAyMzowMDo1NCkNCj4+IFdo
ZW4gSS9PIGdldHMgYmxvY2tlZCB0aGUgVk1zIG1heSBleHBlcmllbmNlIGlzc3VlcyAodGhlbiBW
TSBnZXRzIFBhdXNlZCksIG9yIHRoZWlyIHFlbXUgcHJvY2VzcyBnZXRzIHN0dWNrKHJlc3VsdGlu
ZyBpbiBsaWJ2aXJ0IGVpdGhlciByZXBvcnRpbmcgZXJyb3Igb3IgZ2V0dGluZyBzdHVjayBhcyB3
ZWxsIC0+IHJlc3VsdGluZyBpbiB3aGF0IHZkc20gc2VlcyBhcyDDouKCrMWTbW9uaXRvciB1bnJl
c3BvbnNpdmXDouKCrMKdKQ0KPj4gDQo+PiBTaW5jZSB5b3Ugbm93IGJvdW5jZWQgbGlidmlydGQg
LSBkaWQgaXQgaGVscD8gRG8geW91IHN0aWxsIHNlZSB3cm9uZyBzdGF0dXMgZm9yIHRob3NlIFZN
cyBhbmQgc3RpbGwgdGhvc2UgIm1vbml0b3IgdW5yZXNwb25zaXZlIiBlcnJvcnMgaW4gdmRzbS5s
b2c/DQo+PiBJZiBub3TDouKCrMKmdGhlbiBJIHdvdWxkIHN1c3BlY3QgdGhlIMOi4oKsxZN2bSBy
ZWNvdmVyecOi4oKswp0gY29kZSBub3Qgd29ya2luZyBjb3JyZWN0bHkuIE1pbGFuIGlzIGxvb2tp
bmcgYXQgdGhhdC4NCj4+IA0KPj4gVGhhbmtzLA0KPj4gbWljaGFsDQo+PiANCj4+IA0KPj4+IFRo
ZXJlJ3MgYSBsb3Qgb2YgdmRzbSBsb2dzIQ0KPj4+IA0KPj4+IGZ5aSwgdGhlIHN0b3JhZ2UgZG9t
YWluIGZvciB0aGVzZSBWbXMgaXMgYSAibG9jYWwiIG5mcyBzaGFyZSwgN2U1NjZmNTUtZTA2MC00
N2I3LWJmYTQtYWMzYzQ4ZDcwZGRhLg0KPj4+IA0KPj4+IGF0dGFjaGVkIG1vcmUgbG9ncy4NCj4+
PiANCj4+PiANCj4+Pj4gT24gMDQvMjgvMjAxNiAxMjo1MyBBTSwgTWljaGFsIFNrcml2YW5layB3
cm90ZToNCj4+Pj4+PiBPbiAyNyBBcHIgMjAxNiwgYXQgMTk6MTYsIEJpbGwgSmFtZXMgPGJpbGwu
amFtZXNAajIuY29tPiB3cm90ZToNCj4+Pj4+PiANCj4+Pj4+PiB2aXJzaCAjIGxpc3QgLS1hbGwN
Cj4+Pj4+PiBlcnJvcjogZmFpbGVkIHRvIGNvbm5lY3QgdG8gdGhlIGh5cGVydmlzb3INCj4+Pj4+
PiBlcnJvcjogbm8gdmFsaWQgY29ubmVjdGlvbg0KPj4+Pj4+IGVycm9yOiBGYWlsZWQgdG8gY29u
bmVjdCBzb2NrZXQgdG8gJy92YXIvcnVuL2xpYnZpcnQvbGlidmlydC1zb2NrJzogTm8gc3VjaCBm
aWxlIG9yIGRpcmVjdG9yeQ0KPj4+Pj4+IA0KPj4+Pj4geW91IG5lZWQgdG8gcnVuIHZpcnNoIGlu
IHJlYWQtb25seSBtb2RlDQo+Pj4+PiB2aXJzaCAtciBsaXN0IMOi4oKs4oCdYWxsDQo+Pj4+PiAN
Cj4+Pj4+IFtyb290QG92aXJ0MSB0ZXN0IHZkc21dIyBzeXN0ZW1jdGwgc3RhdHVzIGxpYnZpcnRk
DQo+Pj4+PiDDouKAlMKPIGxpYnZpcnRkLnNlcnZpY2UgLSBWaXJ0dWFsaXphdGlvbiBkYWVtb24N
Cj4+Pj4+ICAgTG9hZGVkOiBsb2FkZWQgKC91c3IvbGliL3N5c3RlbWQvc3lzdGVtL2xpYnZpcnRk
LnNlcnZpY2U7IGVuYWJsZWQ7IHZlbmRvciBwcmVzZXQ6IGVuYWJsZWQpDQo+Pj4+PiAgRHJvcC1J
bjogL2V0Yy9zeXN0ZW1kL3N5c3RlbS9saWJ2aXJ0ZC5zZXJ2aWNlLmQNCj4+Pj4+ICAgICAgICAg
ICDDouKAneKAncOi4oCd4oKsdW5saW1pdGVkLWNvcmUuY29uZg0KPj4+Pj4gICBBY3RpdmU6IGFj
dGl2ZSAocnVubmluZykgc2luY2UgVGh1IDIwMTYtMDQtMjEgMTY6MDA6MDMgUERUOyA1IGRheXMg
YWdvDQo+Pj4+PiANCj4+Pj4+IA0KPj4+Pj4gdHJpZWQgc3lzdGVtY3RsIHJlc3RhcnQgbGlidmly
dGQuDQo+Pj4+PiBObyBjaGFuZ2UuDQo+Pj4+PiANCj4+Pj4+IEF0dGFjaGVkIHZkc20ubG9nIGFu
ZCBzdXBlcnZkc20ubG9nLg0KPj4+Pj4gDQo+Pj4+PiANCj4+Pj4+IFtyb290QG92aXJ0MSB0ZXN0
IHZkc21dIyBzeXN0ZW1jdGwgc3RhdHVzIHZkc21kDQo+Pj4+PiDDouKAlMKPIHZkc21kLnNlcnZp
Y2UgLSBWaXJ0dWFsIERlc2t0b3AgU2VydmVyIE1hbmFnZXINCj4+Pj4+ICAgTG9hZGVkOiBsb2Fk
ZWQgKC91c3IvbGliL3N5c3RlbWQvc3lzdGVtL3Zkc21kLnNlcnZpY2U7IGVuYWJsZWQ7IHZlbmRv
ciBwcmVzZXQ6IGVuYWJsZWQpDQo+Pj4+PiAgIEFjdGl2ZTogYWN0aXZlIChydW5uaW5nKSBzaW5j
ZSBXZWQgMjAxNi0wNC0yNyAxMDowOToxNCBQRFQ7IDNtaW4gNDZzIGFnbw0KPj4+Pj4gDQo+Pj4+
PiANCj4+Pj4+IHZkc20tNC4xNy4xOC0wLmVsNy5jZW50b3Mubm9hcmNoDQo+Pj4+IHRoZSB2ZHNt
LmxvZyBhdHRhY2ggaXMgZ29vZCwgYnV0IGl0w6LigqzihKJzIHRvbyBzaG9ydCBpbnRlcnZhbCwg
aXQgb25seSBzaG93cyByZWNvdmVyeSh2ZHNtIHJlc3RhcnQpIHBoYXNlIHdoZW4gdGhlIFZNcyBh
cmUgaWRlbnRpZmllZCBhcyBwYXVzZWTDouKCrMKmLmNhbiB5b3UgYWRkIGVhcmxpZXIgbG9ncz8g
RGlkIHlvdSByZXN0YXJ0IHZkc20geW91cnNlbGYgb3IgZGlkIGl0IGNyYXNoPw0KPj4+PiANCj4+
Pj4gDQo+Pj4+PiBsaWJ2aXJ0LWRhZW1vbi0xLjIuMTctMTMuZWw3XzIuNC54ODZfNjQNCj4+Pj4+
IA0KPj4+Pj4gDQo+Pj4+PiBUaGFua3MuDQo+Pj4+PiANCj4+Pj4+IA0KPj4+Pj4gT24gMDQvMjYv
MjAxNiAxMTozNSBQTSwgTWljaGFsIFNrcml2YW5layB3cm90ZToNCj4+Pj4+Pj4+IE9uIDI3IEFw
ciAyMDE2LCBhdCAwMjowNCwgTmlyIFNvZmZlciA8bnNvZmZlckByZWRoYXQuY29tPiB3cm90ZToN
Cj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gampPbiBXZWQsIEFwciAyNywgMjAxNiBhdCAyOjAzIEFNLCBC
aWxsIEphbWVzIDxiaWxsLmphbWVzQGoyLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+PiBJIGhhdmUgYSBo
YXJkd2FyZSBub2RlIHRoYXQgaGFzIDI2IFZNcy4NCj4+Pj4+Pj4+IDkgYXJlIGxpc3RlZCBhcyAi
cnVubmluZyIsIDE3IGFyZSBsaXN0ZWQgYXMgInBhdXNlZCIuDQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+
IEluIHRydXRoIGFsbCBWTXMgYXJlIHVwIGFuZCBydW5uaW5nIGZpbmUuDQo+Pj4+Pj4+PiANCj4+
Pj4+Pj4+IEkgdHJpZWQgdGVsbGluZyB0aGUgZGIgdGhleSBhcmUgdXA6DQo+Pj4+Pj4+PiANCj4+
Pj4+Pj4+IGVuZ2luZT0+IHVwZGF0ZSB2bV9keW5hbWljIHNldCBzdGF0dXMgPSAxIHdoZXJlIHZt
X2d1aWQgPShzZWxlY3QNCj4+Pj4+Pj4+IHZtX2d1aWQgZnJvbSB2bV9zdGF0aWMgd2hlcmUgdm1f
bmFtZSA9ICdhcGkxLnRlc3QuajJub2MuY29tJyk7DQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+IEdVSSB0
aGVuIHNob3dzIGl0IHVwIGZvciBhIHNob3J0IHdoaWxlLA0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiB0
aGVuIHB1dHMgaXQgYmFjayBpbiBwYXVzZWQgc3RhdGUuDQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+IDIw
MTYtMDQtMjYgMTU6MTY6NDYsMDk1IElORk8gW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9r
ZXIuVm1BbmFseXplcl0NCj4+Pj4+Pj4+IChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0x
NikgWzE1N2NjMjFlXSBWTSAnMjQyY2EwYWYtNGFiMi00ZGQ2LWI1MTUtNQ0KPj4+Pj4+Pj4gZDQz
NWU2NDUyYzQnKGFwaTEudGVzdC5qMm5vYy5jb20pIG1vdmVkIGZyb20gJ1VwJyAtLT4gJ1BhdXNl
ZCcNCj4+Pj4+Pj4+IDIwMTYtMDQtMjYgMTU6MTY6NDYsMjIxIElORk8gW29yZy5vdmlydC5lbmdp
bmUuY29yZS5kYWwuZGJicm9rZXIuYXVkaXRsb2doDQo+Pj4+Pj4+PiBhbmRsaW5nLkF1ZGl0TG9n
RGlyZWN0b3JdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0xNikgWzE1N2NjMjFlXSBD
b3INCj4+Pj4+Pj4+IHJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOiBudWxsLCBDdXN0b20g
RXZlbnQgSUQ6IC0xLCBNZXNzYWdlOiBWTSBhcGkxLg0KPj4+Pj4+Pj4gdGVzdC5qMm5vYy5jb20g
aGFzIGJlZW4gcGF1c2VkLg0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+IFdoeSBkb2Vz
IHRoZSBlbmdpbmUgdGhpbmsgdGhlIFZNcyBhcmUgcGF1c2VkPw0KPj4+Pj4+Pj4gQXR0YWNoZWQg
ZW5naW5lLmxvZy4NCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gSSBjYW4gZml4IHRoZSBwcm9ibGVtIGJ5
IHBvd2VyaW5nIG9mZiB0aGUgVk0gdGhlbiBzdGFydGluZyBpdCBiYWNrIHVwLg0KPj4+Pj4+Pj4g
QnV0IHRoZSBWTSBpcyB3b3JraW5nIGZpbmUhIEhvdyBkbyBJIGdldCBvdmlydCB0byByZWFsaXpl
IHRoYXQ/DQo+Pj4+Pj4+IElmIHRoaXMgaXMgYW4gaXNzdWUgaW4gZW5naW5lLCByZXN0YXJ0aW5n
IGVuZ2luZSBtYXkgZml4IHRoaXMuDQo+Pj4+Pj4+IGJ1dCBoYXZpbmcgdGhpcyBwcm9ibGVtIG9u
bHkgd2l0aCBvbmUgbm9kZSwgSSBkb24ndCB0aGluayB0aGlzIGlzIHRoZSBpc3N1ZS4NCj4+Pj4+
Pj4gDQo+Pj4+Pj4+IElmIHRoaXMgaXMgYW4gaXNzdWUgaW4gdmRzbSwgcmVzdGFydGluZyB2ZHNt
IG1heSBmaXggdGhpcy4NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IElmIHRoaXMgZG9lcyBub3QgaGVscCwg
bWF5YmUgdGhpcyBpcyBsaWJ2aXJ0IGlzc3VlPyBkaWQgeW91IHRyeSB0byBjaGVjayB2bQ0KPj4+
Pj4+PiBzdGF0dXMgdXNpbmcgdmlyc2g/DQo+Pj4+Pj4gdGhpcyBsb29rcyBtb3JlIGxpa2VseSBh
cyBpdCBzZWVtcyBzdWNoIHN0YXR1cyBpcyBiZWluZyByZXBvcnRlZA0KPj4+Pj4+IGxvZ3Mgd291
bGQgaGVscCwgdmRzbS5sb2cgYXQgdGhlIHZlcnkgbGVhc3QuDQo+Pj4+Pj4gDQo+Pj4+Pj4+IElm
IHZpcnNoIHRoaW5rcyB0aGF0IHRoZSB2bXMgYXJlIHBhdXNlZCwgeW91IGNhbiB0cnkgdG8gcmVz
dGFydCBsaWJ2aXJ0ZC4NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IFBsZWFzZSBmaWxlIGEgYnVnIGFib3V0
IHRoaXMgaW4gYW55IGNhc2Ugd2l0aCBlbmdpbmUgYW5kIHZkc20gbG9ncy4NCj4+Pj4+Pj4gDQo+
Pj4+Pj4+IEFkZGluZyBNaWNoYWwgaW4gY2FzZSBoZSBoYXMgYmV0dGVyIGlkZWEgaG93IHRvIHBy
b2NlZWQuDQo+Pj4+Pj4+IA0KPj4+Pj4+PiBOaXINCj4+Pj4+IFVzZXJzQG92aXJ0Lm9yZw0KPj4+
Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzDQo+Pj4gDQo+
Pj4gPGVuZ2luZS5sb2ctMjAxNjA0MjEuZ3o+PHZkc20ubG9ncy50YXIuZ3o+DQo+PiANCj4gDQo+
IA0KPiBUaGlzIGVtYWlsLCBpdHMgY29udGVudHMgYW5kIGF0dGFjaG1lbnRzIGNvbnRhaW4gaW5m
b3JtYXRpb24gZnJvbSBqMiBHbG9iYWwsIEluYy4gYW5kL29yIGl0cyBhZmZpbGlhdGVzIHdoaWNo
IG1heSBiZSBwcml2aWxlZ2VkLCBjb25maWRlbnRpYWwgb3Igb3RoZXJ3aXNlIHByb3RlY3RlZCBm
cm9tIGRpc2Nsb3N1cmUuIFRoZSBpbmZvcm1hdGlvbiBpcyBpbnRlbmRlZCB0byBiZSBmb3IgdGhl
IGFkZHJlc3NlZShzKSBvbmx5LiBJZiB5b3UgYXJlIG5vdCBhbiBhZGRyZXNzZWUsIGFueSBkaXNj
bG9zdXJlLCBjb3B5LCBkaXN0cmlidXRpb24sIG9yIHVzZSBvZiB0aGUgY29udGVudHMgb2YgdGhp
cyBtZXNzYWdlIGlzIHByb2hpYml0ZWQuIElmIHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgZW1haWwg
aW4gZXJyb3IgcGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGJ5IHJlcGx5IGUtbWFpbCBhbmQgZGVs
ZXRlIHRoZSBvcmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29waWVzLiDCqSAyMDE1IGoyIEdsb2Jh
bCwgSW5jLiBBbGwgcmlnaHRzIHJlc2VydmVkLiBlRmF4IMKuLCBlVm9pY2Ugwq4sIENhbXBhaWdu
ZXIgwq4sIEZ1c2VNYWlsIMKuLCBLZWVwSXRTYWZlIMKuIGFuZCBPbmVib3ggwq4gYXJlICEgcmVn
aXN0ZXJlIGQgdHJhZGVtYXJrcyBvZiBqMiBHbG9iYWwsIEluYy4gYW5kIGl0cyBhZmZpbGlhdGVz
Lg0K
--Apple-Mail-E8504E27-508B-4D91-9E91-823A22F58109
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0
L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPjwvaGVhZD48Ym9keSBkaXI9ImF1dG8iPjxkaXY+PC9kaXY+
PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+T24gMjkgQXByIDIwMTYsIGF0IDE4OjI2LCBCaWxsIEph
bWVzICZsdDs8YSBocmVmPSJtYWlsdG86YmlsbC5qYW1lc0BqMi5jb20iPmJpbGwuamFtZXNAajIu
Y29tPC9hPiZndDsgd3JvdGU6PGJyPjxicj48L2Rpdj48YmxvY2txdW90ZSB0eXBlPSJjaXRlIj48
ZGl2Pg0KICANCiAgICA8bWV0YSBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgiIGh0
dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSI+DQogIA0KICANCiAgICB5ZXMgdGhleSBhcmUgc3RpbGwg
c2F5aW5nICJwYXVzZWQiIHN0YXRlLjxicj4NCiAgICBObywgYm91bmNpbmcgbGlidmlydCBkaWRu
J3QgaGVscC48YnI+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PlRoZW4gbXkgc3Vz
cGljaW9uIG9mIHZtIHJlY292ZXJ5IGdldHMgY2xvc2VyIHRvIGEgY2VydGFpbnR5Oik8ZGl2PkNh
biB5b3UgZ2V0IG9uZSBvZiB0aGUgcGF1c2VkIHZtJ3MgLnJlY292ZXJ5IGZpbGUgZnJvbSAvdmFy
L2xpYi92ZHNtIGFuZCBjaGVjayBpdCBzYXlzIFBhdXNlZCB0aGVyZT8gSXQncyB3b3J0aCBhIHNo
b3QgdG8gdHJ5IHRvIHJlbW92ZSB0aGF0IGZpbGUgYW5kIHJlc3RhcnQgdmRzbSwgdGhlbiBjaGVj
ayBsb2dzIGFuZCB0aGF0IHZtIHN0YXR1cy4uLml0IHNob3VsZCByZWNvdmVyICJnb29kIGVub3Vn
aCIgZnJvbSBsaWJ2aXJ0IG9ubHkuJm5ic3A7PC9kaXY+PGRpdj5UcnkgaXQgd2l0aCBvbmUgZmly
c3Q8YnI+PGRpdj48ZGl2Pjxicj48YmxvY2txdW90ZSB0eXBlPSJjaXRlIj48ZGl2Pg0KICAgIA0K
ICAgIEkgbm90aWNlZCB0aGUgZXJyb3JzIGFib3V0IHRoZSBJU08gZG9tYWluLiBEaWRuJ3QgdGhp
bmsgdGhhdCB3YXMNCiAgICByZWxhdGVkLjxicj4NCiAgICBJIGhhdmUgYmVlbiBtaWdyYXRpbmcg
YSBsb3Qgb2YgVk1zIHRvIG92aXJ0IGxhdGVseSwgYW5kIHJlY2VudGx5DQogICAgYWRkZWQgYW5v
dGhlciBub2RlLjxicj4NCiAgICBBbHNvIGhhZCBzb21lIHByb2JsZW1zIHdpdGggL2V0Yy9leHBv
cnRzIGZvciBhIHdoaWxlLCBidXQgSSB0aGluaw0KICAgIHRob3NlIGlzc3VlcyBhcmUgYWxsIHJl
c29sdmVkLjxicj4NCiAgICA8YnI+DQogICAgPGJyPg0KICAgIExhc3QgInVucmVzcG9uc2l2ZSIg
bWVzc2FnZSBpbiB2ZHNtLmxvZyB3YXM6PGJyPg0KICAgIDxicj4NCiAgICB2ZHNtLmxvZy40OS54
ejpqc29ucnBjLkV4ZWN1dG9yLzA6OldBUk5JTkc6OjxiPjIwMTYtMDQtMjE8L2I+DQogICAgMTE6
MDA6NTQsNzAzOjp2bTo6NTA2Nzo6dmlydC52bTo6KF9zZXRVbnJlc3BvbnNpdmVJZlRpbWVvdXQp
DQogICAgdm1JZD1gYjZhMTM4MDgtOTU1Mi00MDFiLTg0MGItNGY3MDIyZTgyOTNkYDo6bW9uaXRv
ciBiZWNvbWUNCiAgICB1bnJlc3BvbnNpdmUgKGNvbW1hbmQgdGltZW91dCwgYWdlPTMxMDMyMy45
Nyk8YnI+DQogICAgdmRzbS5sb2cuNDkueHo6anNvbnJwYy5FeGVjdXRvci8wOjpXQVJOSU5HOjoy
MDE2LTA0LTIxDQogICAgMTE6MDA6NTQsNzAzOjp2bTo6NTA2Nzo6dmlydC52bTo6KF9zZXRVbnJl
c3BvbnNpdmVJZlRpbWVvdXQpDQogICAgdm1JZD1gNWJmYjE0MGEtYTk3MS00YzljLTgyYzYtMjc3
OTI5ZWI0NWQ0YDo6bW9uaXRvciBiZWNvbWUNCiAgICB1bnJlc3BvbnNpdmUgKGNvbW1hbmQgdGlt
ZW91dCwgYWdlPTMxMDMyMy45Nyk8YnI+DQogICAgPGJyPg0KICAgIDxicj4NCiAgICA8YnI+DQog
ICAgVGhhbmtzLjxicj4NCiAgICA8YnI+DQogICAgPGJyPg0KICAgIDxicj4NCiAgICA8ZGl2IGNs
YXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDQvMjkvMTYgMTo0MCBBTSwgTWljaGFsIFNrcml2YW5l
aw0KICAgICAgd3JvdGU6PGJyPg0KICAgIDwvZGl2Pg0KICAgIDxibG9ja3F1b3RlIGNpdGU9Im1p
ZDo2NTZCRkM1Qy1BNkY1LTQzMzItOTBBQy1DMDM5RDRFOTE3MEVAcmVkaGF0LmNvbSIgdHlwZT0i
Y2l0ZSI+DQogICAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl
eHQvaHRtbDsgY2hhcnNldD1VVEYtOCI+DQogICAgICA8YnIgY2xhc3M9IiI+DQogICAgICA8ZGl2
Pg0KICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj4NCiAgICAgICAgICA8
ZGl2IGNsYXNzPSIiPk9uIDI4IEFwciAyMDE2LCBhdCAxOTo0MCwgQmlsbCBKYW1lcyAmbHQ7PGEg
bW96LWRvLW5vdC1zZW5kPSJ0cnVlIiBocmVmPSJtYWlsdG86YmlsbC5qYW1lc0BqMi5jb20iIGNs
YXNzPSIiPjwvYT48YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWls
dG86YmlsbC5qYW1lc0BqMi5jb20iPmJpbGwuamFtZXNAajIuY29tPC9hPiZndDsgd3JvdGU6PC9k
aXY+DQogICAgICAgICAgPGJyIGNsYXNzPSJBcHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj4NCiAg
ICAgICAgICA8ZGl2IGNsYXNzPSIiPg0KICAgICAgICAgICAgPGRpdiBiZ2NvbG9yPSIjRkZGRkZG
IiB0ZXh0PSIjMDAwMDAwIiBjbGFzcz0iIj4gdGhhbmsgeW91DQogICAgICAgICAgICAgIGZvciBy
ZXNwb25zZS48YnIgY2xhc3M9IiI+DQogICAgICAgICAgICAgIEkgYm9sZC1lZCB0aGUgb25lcyB0
aGF0IGFyZSBsaXN0ZWQgYXMgInBhdXNlZCIuPGJyIGNsYXNzPSIiPg0KICAgICAgICAgICAgICA8
YnIgY2xhc3M9IiI+DQogICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4NCiAgICAgICAgICAgICAg
W3Jvb3RAb3ZpcnQxIHRlc3QgdmRzbV0jIHZpcnNoIC1yIGxpc3QgLS1hbGw8YnIgY2xhc3M9IiI+
DQogICAgICAgICAgICAgIMOCJm5ic3A7SWTDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDsgTmFtZcOC
Jm5ic3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5ic3A7w4Im
bmJzcDvDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDvDgiZu
YnNwO8OCJm5ic3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5i
c3A7w4ImbmJzcDvDgiZuYnNwO8OCJm5ic3A7w4ImbmJzcDsgU3RhdGU8YnIgY2xhc3M9IiI+DQog
ICAgICAgICAgICAgIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS08YnIgY2xhc3M9IiI+DQogICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICA8L2Rp
dj4NCiAgICAgICAgPC9ibG9ja3F1b3RlPg0KICAgICAgPC9kaXY+DQogICAgICA8ZGl2PjxiciBj
bGFzcz0iIj4NCiAgICAgIDwvZGl2Pg0KICAgICAgPGRpdj4NCiAgICAgICAgPGJsb2NrcXVvdGUg
dHlwZT0iY2l0ZSIgY2xhc3M9IiI+DQogICAgICAgICAgPGRpdiBjbGFzcz0iIj4NCiAgICAgICAg
ICAgIDxkaXYgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCIgY2xhc3M9IiI+IDxiciBj
bGFzcz0iIj4NCiAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPg0KICAgICAgICAgICAgICBMb29r
cyBsaWtlIHByb2JsZW0gc3RhcnRlZCBhcm91bmQgMjAxNi0wNC0xNyAyMDoxOTozNCw4MjIsDQog
ICAgICAgICAgICAgIGJhc2VkIG9uIGVuZ2luZS5sb2cgYXR0YWNoZWQuPGJyIGNsYXNzPSIiPg0K
ICAgICAgICAgICAgPC9kaXY+DQogICAgICAgICAgPC9kaXY+DQogICAgICAgIDwvYmxvY2txdW90
ZT4NCiAgICAgICAgPGRpdj48YnIgY2xhc3M9IiI+DQogICAgICAgIDwvZGl2Pg0KICAgICAgICA8
ZGl2PnllcywgdGhhdCB0aW1lIGxvb2tzIGNvcnJlY3QuIEFueSBpZGVhIHdoYXQgbWlnaHQgaGF2
ZSBiZWVuDQogICAgICAgICAgYSB0cmlnZ2VyPyBBbnl0aGluZyBpbnRlcmVzdGluZyBoYXBwZW5l
ZCBhdCB0aGF0IHRpbWUgKHBvd2VyDQogICAgICAgICAgb3V0YWdlIG9mIHNvbWUgaG9zdCwgc29t
ZSBtYWludGVuYW5jZSBhY3Rpb24sIGFueXRoaW5nKT/DgiZuYnNwOzwvZGl2Pg0KICAgICAgICA8
ZGl2PmxvZ3MgaW5kaWNhdGUgYSBwcm9ibGVtIHdoZW4gdmRzbSB0YWxrcyB0byBsaWJ2aXJ0KGFs
bA0KICAgICAgICAgIHRob3NlICJtb25pdG9yIGJlY29tZSB1bnJlc3BvbnNpdmXDouKCrMKdKTwv
ZGl2Pg0KICAgICAgICA8ZGl2PjxiciBjbGFzcz0iIj4NCiAgICAgICAgPC9kaXY+DQogICAgICAg
IDxkaXY+SXQgZG9lcyBzZWVtIHRoYXQgYXQgdGhhdCB0aW1lIHlvdSBzdGFydGVkIHRvIGhhdmUg
c29tZQ0KICAgICAgICAgIHN0b3JhZ2UgY29ubmVjdGl2aXR5IGlzc3VlcyAtIGZpcnN0IG9uZSBh
dMOCJm5ic3A7MjAxNi0wNC0xNw0KICAgICAgICAgIDIwOjA2OjUzLDkyOS4gQW5kIGl0IGRvZXNu
w6LigqzihKJ0IGxvb2sgdGVtcG9yYXJ5IGJlY2F1c2Ugc3VjaA0KICAgICAgICAgIGVycm9ycyBh
cmUgc3RpbGwgdGhlcmUgY291cGxlIGhvdXJzIGxhdGVyKGluIHlvdXIgbW9zdCByZWNlbnQNCiAg
ICAgICAgICBmaWxlIHlvdSBhdHRhY2hlZCBJIGNhbiBzZWUgYXQgMjM6MDA6NTQpPC9kaXY+DQog
ICAgICAgIDxkaXY+V2hlbiBJL08gZ2V0cyBibG9ja2VkIHRoZSBWTXMgbWF5IGV4cGVyaWVuY2Ug
aXNzdWVzICh0aGVuDQogICAgICAgICAgVk0gZ2V0cyBQYXVzZWQpLCBvciB0aGVpciBxZW11IHBy
b2Nlc3MgZ2V0cyBzdHVjayhyZXN1bHRpbmcgaW4NCiAgICAgICAgICBsaWJ2aXJ0IGVpdGhlciBy
ZXBvcnRpbmcgZXJyb3Igb3IgZ2V0dGluZyBzdHVjayBhcyB3ZWxsIC0mZ3Q7DQogICAgICAgICAg
cmVzdWx0aW5nIGluIHdoYXQgdmRzbSBzZWVzIGFzIMOi4oKsxZNtb25pdG9yIHVucmVzcG9uc2l2
ZcOi4oKswp0pPC9kaXY+DQogICAgICAgIDxkaXY+PGJyIGNsYXNzPSIiPg0KICAgICAgICA8L2Rp
dj4NCiAgICAgICAgPGRpdj5TaW5jZSB5b3Ugbm93IGJvdW5jZWQgbGlidmlydGQgLSBkaWQgaXQg
aGVscD8gRG8geW91IHN0aWxsDQogICAgICAgICAgc2VlIHdyb25nIHN0YXR1cyBmb3IgdGhvc2Ug
Vk1zIGFuZCBzdGlsbCB0aG9zZSAibW9uaXRvcg0KICAgICAgICAgIHVucmVzcG9uc2l2ZSIgZXJy
b3JzIGluIHZkc20ubG9nPzwvZGl2Pg0KICAgICAgICA8ZGl2PklmIG5vdMOi4oKswqZ0aGVuIEkg
d291bGQgc3VzcGVjdCB0aGUgw6LigqzFk3ZtIHJlY292ZXJ5w6LigqzCnSBjb2RlIG5vdA0KICAg
ICAgICAgIHdvcmtpbmcgY29ycmVjdGx5LiBNaWxhbiBpcyBsb29raW5nIGF0IHRoYXQuPC9kaXY+
DQogICAgICAgIDxkaXY+PGJyIGNsYXNzPSIiPg0KICAgICAgICA8L2Rpdj4NCiAgICAgICAgPGRp
dj5UaGFua3MsPC9kaXY+DQogICAgICAgIDxkaXY+bWljaGFsPC9kaXY+DQogICAgICAgIDxkaXY+
DQogICAgICAgICAgPGRpdj48YnIgY2xhc3M9IiI+DQogICAgICAgICAgPC9kaXY+DQogICAgICAg
IDwvZGl2Pg0KICAgICAgICA8ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj4NCiAgICAgICAgPC9k
aXY+DQogICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPg0KICAgICAgICAg
IDxkaXYgY2xhc3M9IiI+DQogICAgICAgICAgICA8ZGl2IGJnY29sb3I9IiNGRkZGRkYiIHRleHQ9
IiMwMDAwMDAiIGNsYXNzPSIiPiBUaGVyZSdzIGENCiAgICAgICAgICAgICAgbG90IG9mIHZkc20g
bG9ncyE8YnIgY2xhc3M9IiI+DQogICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4NCiAgICAgICAg
ICAgICAgZnlpLCB0aGUgc3RvcmFnZSBkb21haW4gZm9yIHRoZXNlIFZtcyBpcyBhICJsb2NhbCIg
bmZzDQogICAgICAgICAgICAgIHNoYXJlLCA3ZTU2NmY1NS1lMDYwLTQ3YjctYmZhNC1hYzNjNDhk
NzBkZGEuPGJyIGNsYXNzPSIiPg0KICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+DQogICAgICAg
ICAgICAgIGF0dGFjaGVkIG1vcmUgbG9ncy48YnIgY2xhc3M9IiI+DQogICAgICAgICAgICAgIDxi
ciBjbGFzcz0iIj4NCiAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPg0KICAgICAgICAgICAgICA8
ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA0LzI4LzIwMTYgMTI6NTMgQU0sDQogICAg
ICAgICAgICAgICAgTWljaGFsIFNrcml2YW5layB3cm90ZTo8YnIgY2xhc3M9IiI+DQogICAgICAg
ICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICA8YmxvY2txdW90ZSBjaXRlPSJtaWQ6MjhCRjU1
RTYtM0E5MC00QkI3LTkwQjktMUVFMEE4MkZDNDYwQHJlZGhhdC5jb20iIHR5cGU9ImNpdGUiIGNs
YXNzPSIiPg0KICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIi
Pg0KICAgICAgICAgICAgICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIiPk9uIDI3IEFwciAyMDE2
LCBhdCAxOToxNiwgQmlsbCBKYW1lcyA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGNsYXNzPSJt
b3otdHh0LWxpbmstcmZjMjM5NkUiIGhyZWY9Im1haWx0bzpiaWxsLmphbWVzQGoyLmNvbSI+Jmx0
O2JpbGwuamFtZXNAajIuY29tJmd0OzwvYT4gd3JvdGU6DQoNCnZpcnNoICMgbGlzdCAtLWFsbA0K
ZXJyb3I6IGZhaWxlZCB0byBjb25uZWN0IHRvIHRoZSBoeXBlcnZpc29yDQplcnJvcjogbm8gdmFs
aWQgY29ubmVjdGlvbg0KZXJyb3I6IEZhaWxlZCB0byBjb25uZWN0IHNvY2tldCB0byAnL3Zhci9y
dW4vbGlidmlydC9saWJ2aXJ0LXNvY2snOiBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5DQoNCjwv
cHJlPg0KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4NCiAgICAgICAgICAgICAgICA8cHJl
IGNsYXNzPSIiIHdyYXA9IiI+eW91IG5lZWQgdG8gcnVuIHZpcnNoIGluIHJlYWQtb25seSBtb2Rl
DQp2aXJzaCAtciBsaXN0IMOi4oKs4oCdYWxsDQoNCjwvcHJlPg0KICAgICAgICAgICAgICAgIDxi
bG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPg0KICAgICAgICAgICAgICAgICAgPHByZSBj
bGFzcz0iIiB3cmFwPSIiPltyb290QG92aXJ0MSB0ZXN0IHZkc21dIyBzeXN0ZW1jdGwgc3RhdHVz
IGxpYnZpcnRkDQrDouKAlMKPIGxpYnZpcnRkLnNlcnZpY2UgLSBWaXJ0dWFsaXphdGlvbiBkYWVt
b24NCiAgTG9hZGVkOiBsb2FkZWQgKC91c3IvbGliL3N5c3RlbWQvc3lzdGVtL2xpYnZpcnRkLnNl
cnZpY2U7IGVuYWJsZWQ7IHZlbmRvciBwcmVzZXQ6IGVuYWJsZWQpDQogRHJvcC1JbjogL2V0Yy9z
eXN0ZW1kL3N5c3RlbS9saWJ2aXJ0ZC5zZXJ2aWNlLmQNCiAgICAgICAgICDDouKAneKAncOi4oCd
4oKsdW5saW1pdGVkLWNvcmUuY29uZg0KICBBY3RpdmU6IGFjdGl2ZSAocnVubmluZykgc2luY2Ug
VGh1IDIwMTYtMDQtMjEgMTY6MDA6MDMgUERUOyA1IGRheXMgYWdvDQoNCg0KdHJpZWQgc3lzdGVt
Y3RsIHJlc3RhcnQgbGlidmlydGQuDQpObyBjaGFuZ2UuDQoNCkF0dGFjaGVkIHZkc20ubG9nIGFu
ZCBzdXBlcnZkc20ubG9nLg0KDQoNCltyb290QG92aXJ0MSB0ZXN0IHZkc21dIyBzeXN0ZW1jdGwg
c3RhdHVzIHZkc21kDQrDouKAlMKPIHZkc21kLnNlcnZpY2UgLSBWaXJ0dWFsIERlc2t0b3AgU2Vy
dmVyIE1hbmFnZXINCiAgTG9hZGVkOiBsb2FkZWQgKC91c3IvbGliL3N5c3RlbWQvc3lzdGVtL3Zk
c21kLnNlcnZpY2U7IGVuYWJsZWQ7IHZlbmRvciBwcmVzZXQ6IGVuYWJsZWQpDQogIEFjdGl2ZTog
YWN0aXZlIChydW5uaW5nKSBzaW5jZSBXZWQgMjAxNi0wNC0yNyAxMDowOToxNCBQRFQ7IDNtaW4g
NDZzIGFnbw0KDQoNCnZkc20tNC4xNy4xOC0wLmVsNy5jZW50b3Mubm9hcmNoDQo8L3ByZT4NCiAg
ICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+DQogICAgICAgICAgICAgICAgPHByZSBjbGFzcz0i
IiB3cmFwPSIiPnRoZSB2ZHNtLmxvZyBhdHRhY2ggaXMgZ29vZCwgYnV0IGl0w6LigqzihKJzIHRv
byBzaG9ydCBpbnRlcnZhbCwgaXQgb25seSBzaG93cyByZWNvdmVyeSh2ZHNtIHJlc3RhcnQpIHBo
YXNlIHdoZW4gdGhlIFZNcyBhcmUgaWRlbnRpZmllZCBhcyBwYXVzZWTDouKCrMKmLmNhbiB5b3Ug
YWRkIGVhcmxpZXIgbG9ncz8gRGlkIHlvdSByZXN0YXJ0IHZkc20geW91cnNlbGYgb3IgZGlkIGl0
IGNyYXNoPw0KDQoNCjwvcHJlPg0KICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNp
dGUiIGNsYXNzPSIiPg0KICAgICAgICAgICAgICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIiPmxp
YnZpcnQtZGFlbW9uLTEuMi4xNy0xMy5lbDdfMi40Lng4Nl82NA0KDQoNClRoYW5rcy4NCg0KDQpP
biAwNC8yNi8yMDE2IDExOjM1IFBNLCBNaWNoYWwgU2tyaXZhbmVrIHdyb3RlOg0KPC9wcmU+DQog
ICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj4NCiAgICAg
ICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9IiI+DQogICAgICAg
ICAgICAgICAgICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIiPk9uIDI3IEFwciAyMDE2LCBhdCAw
MjowNCwgTmlyIFNvZmZlciA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGNsYXNzPSJtb3otdHh0
LWxpbmstcmZjMjM5NkUiIGhyZWY9Im1haWx0bzpuc29mZmVyQHJlZGhhdC5jb20iPiZsdDtuc29m
ZmVyQHJlZGhhdC5jb20mZ3Q7PC9hPiB3cm90ZToNCg0KampPbiBXZWQsIEFwciAyNywgMjAxNiBh
dCAyOjAzIEFNLCBCaWxsIEphbWVzIDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIgY2xhc3M9Im1v
ei10eHQtbGluay1yZmMyMzk2RSIgaHJlZj0ibWFpbHRvOmJpbGwuamFtZXNAajIuY29tIj4mbHQ7
YmlsbC5qYW1lc0BqMi5jb20mZ3Q7PC9hPiB3cm90ZToNCjwvcHJlPg0KICAgICAgICAgICAgICAg
ICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPg0KICAgICAgICAgICAgICAg
ICAgICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIiPkkgaGF2ZSBhIGhhcmR3YXJlIG5vZGUgdGhh
dCBoYXMgMjYgVk1zLg0KOSBhcmUgbGlzdGVkIGFzICJydW5uaW5nIiwgMTcgYXJlIGxpc3RlZCBh
cyAicGF1c2VkIi4NCg0KSW4gdHJ1dGggYWxsIFZNcyBhcmUgdXAgYW5kIHJ1bm5pbmcgZmluZS4N
Cg0KSSB0cmllZCB0ZWxsaW5nIHRoZSBkYiB0aGV5IGFyZSB1cDoNCg0KZW5naW5lPSZndDsgdXBk
YXRlIHZtX2R5bmFtaWMgc2V0IHN0YXR1cyA9IDEgd2hlcmUgdm1fZ3VpZCA9KHNlbGVjdA0Kdm1f
Z3VpZCBmcm9tIHZtX3N0YXRpYyB3aGVyZSB2bV9uYW1lID0gJzxhIG1vei1kby1ub3Qtc2VuZD0i
dHJ1ZSIgaHJlZj0iaHR0cDovL2FwaTEudGVzdC5qMm5vYy5jb20iIGNsYXNzPSIiPmFwaTEudGVz
dC5qMm5vYy5jb208L2E+Jyk7DQoNCkdVSSB0aGVuIHNob3dzIGl0IHVwIGZvciBhIHNob3J0IHdo
aWxlLA0KDQp0aGVuIHB1dHMgaXQgYmFjayBpbiBwYXVzZWQgc3RhdGUuDQoNCjIwMTYtMDQtMjYg
MTU6MTY6NDYsMDk1IElORk8gW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuVm1BbmFs
eXplcl0NCihEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0xNikgWzE1N2NjMjFlXSBWTSAn
MjQyY2EwYWYtNGFiMi00ZGQ2LWI1MTUtNQ0KZDQzNWU2NDUyYzQnKDxhIG1vei1kby1ub3Qtc2Vu
ZD0idHJ1ZSIgaHJlZj0iaHR0cDovL2FwaTEudGVzdC5qMm5vYy5jb20iIGNsYXNzPSIiPmFwaTEu
dGVzdC5qMm5vYy5jb208L2E+KSBtb3ZlZCBmcm9tICdVcCcgLS0mZ3Q7ICdQYXVzZWQnDQoyMDE2
LTA0LTI2IDE1OjE2OjQ2LDIyMSBJTkZPIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJv
a2VyLmF1ZGl0bG9naA0KYW5kbGluZy5BdWRpdExvZ0RpcmVjdG9yXSAoRGVmYXVsdFF1YXJ0elNj
aGVkdWxlcl9Xb3JrZXItMTYpIFsxNTdjYzIxZV0gQ29yDQpyZWxhdGlvbiBJRDogbnVsbCwgQ2Fs
bCBTdGFjazogbnVsbCwgQ3VzdG9tIEV2ZW50IElEOiAtMSwgTWVzc2FnZTogVk0gYXBpMS4NCjxh
IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIgaHJlZj0iaHR0cDovL3Rlc3QuajJub2MuY29tIiBjbGFz
cz0iIj50ZXN0Lmoybm9jLmNvbTwvYT4gaGFzIGJlZW4gcGF1c2VkLg0KDQoNCldoeSBkb2VzIHRo
ZSBlbmdpbmUgdGhpbmsgdGhlIFZNcyBhcmUgcGF1c2VkPw0KQXR0YWNoZWQgZW5naW5lLmxvZy4N
Cg0KSSBjYW4gZml4IHRoZSBwcm9ibGVtIGJ5IHBvd2VyaW5nIG9mZiB0aGUgVk0gdGhlbiBzdGFy
dGluZyBpdCBiYWNrIHVwLg0KQnV0IHRoZSBWTSBpcyB3b3JraW5nIGZpbmUhIEhvdyBkbyBJIGdl
dCBvdmlydCB0byByZWFsaXplIHRoYXQ/DQo8L3ByZT4NCiAgICAgICAgICAgICAgICAgICAgICA8
L2Jsb2NrcXVvdGU+DQogICAgICAgICAgICAgICAgICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIi
PklmIHRoaXMgaXMgYW4gaXNzdWUgaW4gZW5naW5lLCByZXN0YXJ0aW5nIGVuZ2luZSBtYXkgZml4
IHRoaXMuDQpidXQgaGF2aW5nIHRoaXMgcHJvYmxlbSBvbmx5IHdpdGggb25lIG5vZGUsIEkgZG9u
J3QgdGhpbmsgdGhpcyBpcyB0aGUgaXNzdWUuDQoNCklmIHRoaXMgaXMgYW4gaXNzdWUgaW4gdmRz
bSwgcmVzdGFydGluZyB2ZHNtIG1heSBmaXggdGhpcy4NCg0KSWYgdGhpcyBkb2VzIG5vdCBoZWxw
LCBtYXliZSB0aGlzIGlzIGxpYnZpcnQgaXNzdWU/IGRpZCB5b3UgdHJ5IHRvIGNoZWNrIHZtDQpz
dGF0dXMgdXNpbmcgdmlyc2g/DQo8L3ByZT4NCiAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1
b3RlPg0KICAgICAgICAgICAgICAgICAgICA8cHJlIGNsYXNzPSIiIHdyYXA9IiI+dGhpcyBsb29r
cyBtb3JlIGxpa2VseSBhcyBpdCBzZWVtcyBzdWNoIHN0YXR1cyBpcyBiZWluZyByZXBvcnRlZA0K
bG9ncyB3b3VsZCBoZWxwLCB2ZHNtLmxvZyBhdCB0aGUgdmVyeSBsZWFzdC4NCg0KPC9wcmU+DQog
ICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPg0KICAg
ICAgICAgICAgICAgICAgICAgIDxwcmUgY2xhc3M9IiIgd3JhcD0iIj5JZiB2aXJzaCB0aGlua3Mg
dGhhdCB0aGUgdm1zIGFyZSBwYXVzZWQsIHlvdSBjYW4gdHJ5IHRvIHJlc3RhcnQgbGlidmlydGQu
DQoNClBsZWFzZSBmaWxlIGEgYnVnIGFib3V0IHRoaXMgaW4gYW55IGNhc2Ugd2l0aCBlbmdpbmUg
YW5kIHZkc20gbG9ncy4NCg0KQWRkaW5nIE1pY2hhbCBpbiBjYXNlIGhlIGhhcyBiZXR0ZXIgaWRl
YSBob3cgdG8gcHJvY2VlZC4NCg0KTmlyDQo8L3ByZT4NCiAgICAgICAgICAgICAgICAgICAgPC9i
bG9ja3F1b3RlPg0KICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPg0KICAgICAgICAgICAg
ICAgICAgPHByZSBjbGFzcz0iIiB3cmFwPSIiPjxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIgY2xh
c3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9y
ZyI+VXNlcnNAb3ZpcnQub3JnPC9hPg0KPGEgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIiBjbGFzcz0i
bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxt
YW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m
by91c2VyczwvYT4NCjwvcHJlPg0KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4NCiAgICAg
ICAgICAgICAgPC9ibG9ja3F1b3RlPg0KICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+DQogICAg
ICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgIDxzcGFuIGlkPSJjaWQ6RUIwMkI0ODgtQzA3MC00
NkZBLTk5MzgtREM3RDZERjVCRUVEQGJycS5yZWRoYXQuY29tIj4mbHQ7ZW5naW5lLmxvZy0yMDE2
MDQyMS5neiZndDs8L3NwYW4+PHNwYW4gaWQ9ImNpZDo1MkUyNzAyMy1BNjAyLTREQjAtQjY5QS0x
ODIzN0NDMDQ4QTNAYnJxLnJlZGhhdC5jb20iPiZsdDt2ZHNtLmxvZ3MudGFyLmd6Jmd0Ozwvc3Bh
bj48L2Rpdj4NCiAgICAgICAgPC9ibG9ja3F1b3RlPg0KICAgICAgPC9kaXY+DQogICAgICA8YnIg
Y2xhc3M9IiI+DQogICAgPC9ibG9ja3F1b3RlPg0KICAgIDxicj4NCiAgDQo8cD48YSBocmVmPSJo
dHRwOi8vd3d3LmoyLmNvbS8/dXRtX3NvdXJjZT1qMmdsb2JhbCZhbXA7dXRtX21lZGl1bT14c2Vs
bC1yZWZlcnJhbCZhbXA7dXRtX2NhbXBhaWduPWVtcGxveWVlZW1haWwiPjxzcGFuIHN0eWxlPSJj
b2xvcjp3aW5kb3d0ZXh0Ow0KdGV4dC1kZWNvcmF0aW9uOm5vbmUiPjxpbWcgYm9yZGVyPSIwIiB3
aWR0aD0iMzkxIiBoZWlnaHQ9IjQ2IiBzcmM9Imh0dHA6Ly9ob21lLmoyLmNvbS9qMl9HbG9iYWxf
Q2xvdWRfU2VydmljZXMvajJfR2xvYmFsX0VtYWlsX0Zvb3Rlci5qcGciIGFsdD0id3d3LmoyLmNv
bSI+PC9zcGFuPjwvYT48L3A+DQoNCjxwPjxzcGFuIHN0eWxlPSJmb250LXNpemU6OC4wcHQ7Zm9u
dC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OzsNCmNvbG9y
OmdyYXkiPlRoaXMgZW1haWwsIGl0cyBjb250ZW50cyBhbmQgYXR0YWNobWVudHMgY29udGFpbiBp
bmZvcm1hdGlvbiBmcm9tIDxhIGhyZWY9Imh0dHA6Ly93d3cuajIuY29tLz91dG1fc291cmNlPWoy
Z2xvYmFsJmFtcDt1dG1fbWVkaXVtPXhzZWxsLXJlZmVycmFsJmFtcDt1dG1fY2FtcGFpZ249ZW1w
bG95ZW1haWwiPmoyIEdsb2JhbCwgSW5jPC9hPi4gYW5kL29yIGl0cyBhZmZpbGlhdGVzIHdoaWNo
IG1heSBiZSBwcml2aWxlZ2VkLCBjb25maWRlbnRpYWwgb3Igb3RoZXJ3aXNlIHByb3RlY3RlZCBm
cm9tIGRpc2Nsb3N1cmUuIFRoZSBpbmZvcm1hdGlvbiBpcyBpbnRlbmRlZCB0byBiZSBmb3IgdGhl
IGFkZHJlc3NlZShzKSBvbmx5LiBJZiB5b3UgYXJlIG5vdCBhbiBhZGRyZXNzZWUsIGFueSBkaXNj
bG9zdXJlLCBjb3B5LCBkaXN0cmlidXRpb24sIG9yIHVzZSBvZiB0aGUgY29udGVudHMgb2YgdGhp
cyBtZXNzYWdlIGlzIHByb2hpYml0ZWQuIElmIHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgZW1haWwg
aW4gZXJyb3IgcGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGJ5IHJlcGx5IGUtbWFpbCBhbmQgZGVs
ZXRlIHRoZSBvcmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29waWVzLiDCqSAyMDE1IDxhIGhyZWY9
Imh0dHA6Ly93d3cuajIuY29tLyI+ajIgR2xvYmFsLCBJbmM8L2E+LiBBbGwgcmlnaHRzIHJlc2Vy
dmVkLiA8YSBocmVmPSJodHRwOi8vd3d3LmVmYXguY29tLyI+ZUZheCDCrjwvYT4sIDxhIGhyZWY9
Imh0dHA6Ly93d3cuZXZvaWNlLmNvbS8iPmVWb2ljZSDCrjwvYT4sIDxhIGhyZWY9Imh0dHA6Ly93
d3cuY2FtcGFpZ25lci5jb20vIj5DYW1wYWlnbmVyIMKuPC9hPiwgPGEgaHJlZj0iaHR0cDovL3d3
dy5mdXNlbWFpbC5jb20vIj5GdXNlTWFpbCDCrjwvYT4sIDxhIGhyZWY9Imh0dHA6Ly93d3cua2Vl
cGl0c2FmZS5jb20vIj5LZWVwSXRTYWZlIMKuPC9hPiBhbmQgPGEgaHJlZj0iaHR0cDovL3d3dy5v
bmVib3guY29tLyI+T25lYm94IMKuPC9hPiBhcmUgIQ0KIHJlZ2lzdGVyZQ0KIGQgdHJhZGVtYXJr
cyBvZiA8YSBocmVmPSJodHRwOi8vd3d3LmoyLmNvbS8iPmoyIEdsb2JhbCwgSW5jPC9hPi4gYW5k
IGl0cyBhZmZpbGlhdGVzLjwvc3Bhbj48L3A+DQoNCjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48
L2Rpdj48L2Rpdj48L2JvZHk+PC9odG1sPg==
--Apple-Mail-E8504E27-508B-4D91-9E91-823A22F58109--
10:59 p.m.
New subject: vms in paused state
--------------090600060303000004030204
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit
where do I find the recovery files?
[root@ovirt1 test vdsm]# pwd
/var/lib/vdsm
[root@ovirt1 test vdsm]# ls -la
total 16
drwxr-xr-x 6 vdsm kvm 100 Mar 17 16:33 .
drwxr-xr-x. 45 root root 4096 Apr 29 12:01 ..
-rw-r--r-- 1 vdsm kvm 10170 Jan 19 05:04 bonding-defaults.json
drwxr-xr-x 2 vdsm root 6 Apr 19 11:34 netconfback
drwxr-xr-x 3 vdsm kvm 54 Apr 19 11:35 persistence
drwxr-x---. 2 vdsm kvm 6 Mar 17 16:33 transient
drwxr-xr-x 2 vdsm kvm 40 Mar 17 16:33 upgrade
[root@ovirt1 test vdsm]# locate recovery
/opt/hp/hpdiags/en/tcstorage.ldinterimrecovery.htm
/opt/hp/hpdiags/en/tcstorage.ldrecoveryready.htm
/usr/share/doc/postgresql-9.2.15/html/archive-recovery-settings.html
/usr/share/doc/postgresql-9.2.15/html/recovery-config.html
/usr/share/doc/postgresql-9.2.15/html/recovery-target-settings.html
/usr/share/pgsql/recovery.conf.sample
/var/lib/nfs/v4recovery
[root@ovirt1 test vdsm]# locate 757a5 (disk id)
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.lease
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.meta
[root@ovirt1 test vdsm]# locate 5bfb140 (vm id)
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.com.redhat.rhevm.vdsm
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.org.qemu.guest_agent.0
On 4/29/16 10:02 AM, Michal Skrivanek wrote:
On 29 Apr 2016, at 18:26, Bill James <bill.james(a)j2.com
<mailto:bill.james@j2.com>> wrote:
> yes they are still saying "paused" state.
> No, bouncing libvirt didn't help.
Then my suspicion of vm recovery gets closer to a certainty:)
Can you get one of the paused vm's .recovery file from /var/lib/vdsm
and check it says Paused there? It's worth a shot to try to remove
that file and restart vdsm, then check logs and that vm status...it
should recover "good enough" from libvirt only.
Try it with one first
> I noticed the errors about the ISO domain. Didn't think that was related.
> I have been migrating a lot of VMs to ovirt lately, and recently
> added another node.
> Also had some problems with /etc/exports for a while, but I think
> those issues are all resolved.
>
>
> Last "unresponsive" message in vdsm.log was:
>
> vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::*2016-04-21*
> 11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
> vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor become
> unresponsive (command timeout, age=310323.97)
> vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
> 11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
> vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor become
> unresponsive (command timeout, age=310323.97)
>
>
>
> Thanks.
>
>
>
> On 4/29/16 1:40 AM, Michal Skrivanek wrote:
>>
>>> On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com> wrote:
>>>
>>> thank you for response.
>>> I bold-ed the ones that are listed as "paused".
>>>
>>>
>>> [root@ovirt1 test vdsm]# virsh -r list --all
>>> à Idà à à Nameà à à à à à à à à à à à à à à à à à à Ã
à à à à à à State
>>> ----------------------------------------------------
>>
>>>
>>>
>>> Looks like problem started around 2016-04-17 20:19:34,822, based on
>>> engine.log attached.
>>
>> yes, that time looks correct. Any idea what might have been a
>> trigger? Anything interesting happened at that time (power outage of
>> some host, some maintenance action, anything)?Ã
>> logs indicate a problem when vdsm talks to libvirt(all those
>> "monitor become unresponsiveââ¬Â)
>>
>> It does seem that at that time you started to have some storage
>> connectivity issues - first one atà 2016-04-17 20:06:53,929. And it
>> doesnââ¬â¢t look temporary because such errors are still there couple
>> hours later(in your most recent file you attached I can see at 23:00:54)
>> When I/O gets blocked the VMs may experience issues (then VM gets
>> Paused), or their qemu process gets stuck(resulting in libvirt
>> either reporting error or getting stuck as well -> resulting in what
>> vdsm sees as ââ¬Åmonitor unresponsiveââ¬Â)
>>
>> Since you now bounced libvirtd - did it help? Do you still see wrong
>> status for those VMs and still those "monitor unresponsive" errors
>> in vdsm.log?
>> If notââ¬ÂŠthen I would suspect the ââ¬Åvm recoveryââ¬Â code not working
>> correctly. Milan is looking at that.
>>
>> Thanks,
>> michal
>>
>>
>>> There's a lot of vdsm logs!
>>>
>>> fyi, the storage domain for these Vms is a "local" nfs share,
>>> 7e566f55-e060-47b7-bfa4-ac3c48d70dda.
>>>
>>> attached more logs.
>>>
>>>
>>> On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
>>>>> On 27 Apr 2016, at 19:16, Bill James<bill.james(a)j2.com>
wrote:
>>>>>
>>>>> virsh # list --all
>>>>> error: failed to connect to the hypervisor
>>>>> error: no valid connection
>>>>> error: Failed to connect socket to
'/var/run/libvirt/libvirt-sock': No such file or directory
>>>>>
>>>> you need to run virsh in read-only mode
>>>> virsh -r list ââ¬âall
>>>>
>>>>> [root@ovirt1 test vdsm]# systemctl status libvirtd
>>>>> ââ libvirtd.service - Virtualization daemon
>>>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
>>>>> Drop-In: /etc/systemd/system/libvirtd.service.d
>>>>> ââââââ¬unlimited-core.conf
>>>>> Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days
ago
>>>>>
>>>>>
>>>>> tried systemctl restart libvirtd.
>>>>> No change.
>>>>>
>>>>> Attached vdsm.log and supervdsm.log.
>>>>>
>>>>>
>>>>> [root@ovirt1 test vdsm]# systemctl status vdsmd
>>>>> ââ vdsmd.service - Virtual Desktop Server Manager
>>>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled;
vendor preset: enabled)
>>>>> Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min
46s ago
>>>>>
>>>>>
>>>>> vdsm-4.17.18-0.el7.centos.noarch
>>>> the vdsm.log attach is good, but itââ¬â¢s too short interval, it only
shows recovery(vdsm restart) phase when the VMs are identified as pausedââ¬ÂŠ.can you
add earlier logs? Did you restart vdsm yourself or did it crash?
>>>>
>>>>
>>>>> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
>>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
>>>>>>> On 27 Apr 2016, at 02:04, Nir
Soffer<nsoffer(a)redhat.com> wrote:
>>>>>>>
>>>>>>> jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill
James<bill.james(a)j2.com> wrote:
>>>>>>>> I have a hardware node that has 26 VMs.
>>>>>>>> 9 are listed as "running", 17 are listed as
"paused".
>>>>>>>>
>>>>>>>> In truth all VMs are up and running fine.
>>>>>>>>
>>>>>>>> I tried telling the db they are up:
>>>>>>>>
>>>>>>>> engine=> update vm_dynamic set status = 1 where
vm_guid =(select
>>>>>>>> vm_guid from vm_static where vm_name =
'api1.test.j2noc.com <http://api1.test.j2noc.com>');
>>>>>>>>
>>>>>>>> GUI then shows it up for a short while,
>>>>>>>>
>>>>>>>> then puts it back in paused state.
>>>>>>>>
>>>>>>>> 2016-04-26 15:16:46,095 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>>>>>>> (DefaultQuartzScheduler_Worker-16) [157cc21e] VM
'242ca0af-4ab2-4dd6-b515-5
>>>>>>>> d435e6452c4'(api1.test.j2noc.com
<http://api1.test.j2noc.com>) moved from 'Up' --> 'Paused'
>>>>>>>> 2016-04-26 15:16:46,221 INFO
[org.ovirt.engine.core.dal.dbbroker.auditlogh
>>>>>>>> andling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
>>>>>>>> relation ID: null, Call Stack: null, Custom Event ID: -1,
Message: VM api1.
>>>>>>>> test.j2noc.com <http://test.j2noc.com> has been
paused.
>>>>>>>>
>>>>>>>>
>>>>>>>> Why does the engine think the VMs are paused?
>>>>>>>> Attached engine.log.
>>>>>>>>
>>>>>>>> I can fix the problem by powering off the VM then
starting it back up.
>>>>>>>> But the VM is working fine! How do I get ovirt to realize
that?
>>>>>>> If this is an issue in engine, restarting engine may fix
this.
>>>>>>> but having this problem only with one node, I don't think
this is the issue.
>>>>>>>
>>>>>>> If this is an issue in vdsm, restarting vdsm may fix this.
>>>>>>>
>>>>>>> If this does not help, maybe this is libvirt issue? did you
try to check vm
>>>>>>> status using virsh?
>>>>>> this looks more likely as it seems such status is being reported
>>>>>> logs would help, vdsm.log at the very least.
>>>>>>
>>>>>>> If virsh thinks that the vms are paused, you can try to
restart libvirtd.
>>>>>>>
>>>>>>> Please file a bug about this in any case with engine and vdsm
logs.
>>>>>>>
>>>>>>> Adding Michal in case he has better idea how to proceed.
>>>>>>>
>>>>>>> Nir
>>>>> Users(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>> <engine.log-20160421.gz><vdsm.logs.tar.gz>
>>
>
> www.j2.com
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>
> This email, its contents and attachments contain information from j2
> Global, Inc
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
> and/or its affiliates which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to
> be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this
> message is prohibited. If you have received this email in error
> please notify the sender by reply e-mail and delete the original
> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
> are ! registere d trademarks of j2 Global, Inc <http://www.j2.com/>;.
> and its affiliates.
>
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------090600060303000004030204
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
where do I find the recovery files?<br>
<br>
[root@ovirt1 test vdsm]# pwd<br>
/var/lib/vdsm<br>
[root@ovirt1 test vdsm]# ls -la<br>
total 16<br>
drwxr-xr-x  6 vdsm kvm   100 Mar 17 16:33 .<br>
drwxr-xr-x. 45 root root 4096 Apr 29 12:01 ..<br>
-rw-r--r--  1 vdsm kvm 10170 Jan 19 05:04 bonding-defaults.json<br>
drwxr-xr-x  2 vdsm root    6 Apr 19 11:34 netconfback<br>
drwxr-xr-x  3 vdsm kvm    54 Apr 19 11:35 persistence<br>
drwxr-x---. 2 vdsm kvm     6 Mar 17 16:33 transient<br>
drwxr-xr-x  2 vdsm kvm    40 Mar 17 16:33 upgrade<br>
[root@ovirt1 test vdsm]# locate recovery<br>
/opt/hp/hpdiags/en/tcstorage.ldinterimrecovery.htm<br>
/opt/hp/hpdiags/en/tcstorage.ldrecoveryready.htm<br>
/usr/share/doc/postgresql-9.2.15/html/archive-recovery-settings.html<br>
/usr/share/doc/postgresql-9.2.15/html/recovery-config.html<br>
/usr/share/doc/postgresql-9.2.15/html/recovery-target-settings.html<br>
/usr/share/pgsql/recovery.conf.sample<br>
/var/lib/nfs/v4recovery<br>
<br>
<br>
[root@ovirt1 test vdsm]# locate 757a5Â (disk id)<br>
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118<br>
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2<br>
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.lease<br>
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.meta<br>
[root@ovirt1 test vdsm]# locate 5bfb140 (vm id)<br>
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.com.redhat.rhevm.vdsm<br>
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.org.qemu.guest_agent.0<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 4/29/16 10:02 AM, Michal Skrivanek
wrote:<br>
</div>
<blockquote
cite="mid:034AC28F-A06A-43B4-95E3-BB4EAD615E04@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div><br>
</div>
<div><br>
On 29 Apr 2016, at 18:26, Bill James <<a
moz-do-not-send="true"
href="mailto:bill.james@j2.com"><a
class="moz-txt-link-abbreviated"
href="mailto:bill.james@j2.com">bill.james@j2.com</a></a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div> yes they are still saying "paused" state.<br>
No, bouncing libvirt didn't help.<br>
</div>
</blockquote>
<div><br>
</div>
Then my suspicion of vm recovery gets closer to a certainty:)
<div>Can you get one of the paused vm's .recovery file from
/var/lib/vdsm and check it says Paused there? It's worth a shot
to try to remove that file and restart vdsm, then check logs and
that vm status...it should recover "good enough" from libvirt
only. </div>
<div>Try it with one first<br>
<div>
<div><br>
<blockquote type="cite">
<div> I noticed the errors about the ISO domain. Didn't
think that was related.<br>
I have been migrating a lot of VMs to ovirt lately, and
recently added another node.<br>
Also had some problems with /etc/exports for a while,
but I think those issues are all resolved.<br>
<br>
<br>
Last "unresponsive" message in vdsm.log was:<br>
<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::<b>2016-04-21</b>
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor
become unresponsive (command timeout, age=310323.97)<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor
become unresponsive (command timeout, age=310323.97)<br>
<br>
<br>
<br>
Thanks.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 4/29/16 1:40 AM, Michal
Skrivanek wrote:<br>
</div>
<blockquote
cite="mid:656BFC5C-A6F5-4332-90AC-C039D4E9170E@redhat.com"
type="cite"> <br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 28 Apr 2016, at 19:40, Bill James
<<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:bill.james@j2.com">bill.james@j2.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class="">
thank you for response.<br class="">
I bold-ed the ones that are listed as
"paused".<br class="">
<br class="">
<br class="">
[root@ovirt1 test vdsm]# virsh -r list --all<br
class="">
ÃÂ IdÃÂ ÃÂ ÃÂ
NameÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ ÃÂ
State<br class="">
----------------------------------------------------<br class="">
</div>
</div>
</blockquote>
</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class="">
<br class="">
<br class="">
Looks like problem started around 2016-04-17
20:19:34,822, based on engine.log attached.<br
class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>yes, that time looks correct. Any idea what
might have been a trigger? Anything interesting
happened at that time (power outage of some host,
some maintenance action, anything)?ÃÂ </div>
<div>logs indicate a problem when vdsm talks to
libvirt(all those "monitor become
unresponsiveââ¬Â)</div>
<div><br class="">
</div>
<div>It does seem that at that time you started to
have some storage connectivity issues - first one
atà2016-04-17 20:06:53,929. And it doesnââ¬â¢t look
temporary because such errors are still there
couple hours later(in your most recent file you
attached I can see at 23:00:54)</div>
<div>When I/O gets blocked the VMs may experience
issues (then VM gets Paused), or their qemu
process gets stuck(resulting in libvirt either
reporting error or getting stuck as well ->
resulting in what vdsm sees as ââ¬Åmonitor
unresponsiveââ¬Â)</div>
<div><br class="">
</div>
<div>Since you now bounced libvirtd - did it help?
Do you still see wrong status for those VMs and
still those "monitor unresponsive" errors in
vdsm.log?</div>
<div>If notââ¬ÂŠthen I would suspect the ââ¬Åvm
recoveryââ¬Â code not working correctly. Milan is
looking at that.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div>michal</div>
<div>
<div><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class="">
There's a lot of vdsm logs!<br class="">
<br class="">
fyi, the storage domain for these Vms is a
"local" nfs share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br
class="">
<br class="">
attached more logs.<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/28/2016
12:53 AM, Michal Skrivanek wrote:<br
class="">
</div>
<blockquote
cite="mid:28BF55E6-3A90-4BB7-90B9-1EE0A82FC460@redhat.com"
type="cite" class="">
<blockquote type="cite" class="">
<pre class="" wrap="">On 27 Apr
2016, at 19:16, Bill James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file
or directory
</pre>
</blockquote>
<pre class="" wrap="">you need to
run virsh in read-only mode
virsh -r list ââ¬âall
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">[root@ovirt1
test vdsm]# systemctl status libvirtd
ââ libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset:
enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
ââââââ¬unlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
ââ vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre class="" wrap="">the vdsm.log
attach is good, but itââ¬â¢s too short interval, it only shows recovery(vdsm restart)
phase when the VMs are identified as pausedââ¬ÂŠ.can you add earlier logs? Did you
restart vdsm yourself or did it crash?
</pre>
<blockquote type="cite" class="">
<pre class=""
wrap="">libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type="cite" class="">
<blockquote type="cite"
class="">
<pre class="" wrap="">On 27
Apr 2016, at 02:04, Nir Soffer <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a>
wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
</pre>
<blockquote type="cite"
class="">
<pre class="" wrap="">I have
a hardware node that has 26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=> update vm_dynamic set status = 1 where vm_guid =(select
vm_guid from vm_static where vm_name = '<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
class="">api1.test.j2noc.com</a>');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM '242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
class="">api1.test.j2noc.com</a>) moved from 'Up' -->
'Paused'
2016-04-26 15:16:46,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM api1.
<a moz-do-not-send="true" href="http://test.j2noc.com"
class="">test.j2noc.com</a> has been paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre class="" wrap="">If this
is an issue in engine, restarting engine may fix this.
but having this problem only with one node, I don't think this is the issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check vm
status using virsh?
</pre>
</blockquote>
<pre class="" wrap="">this looks
more likely as it seems such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type="cite"
class="">
<pre class="" wrap="">If virsh
thinks that the vms are paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre class="" wrap=""><a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
</blockquote>
<br class="">
</div>
<span
id="cid:EB02B488-C070-46FA-9938-DC7D6DF5BEED@brq.redhat.com"><engine.log-20160421.gz></span><span
id="cid:52E27023-A602-4DB0-B69A-18237CC048A3@brq.redhat.com"><vdsm.logs.tar.gz></span></div>
</blockquote>
</div>
<br class="">
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext;
text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" height="46"
border="0"
width="391"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain
information from <a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may
be privileged, confidential or otherwise protected
from disclosure. The information is intended to be
for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or
use of the contents of this message is prohibited.
If you have received this email in error please
notify the sender by reply e-mail and delete the
original message and any copies. © 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/">eFax
®</a>, <a moz-do-not-send="true"
href="http://www.evoice.com/">eVoice ®</a>,
<a
moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner
®</a>,
<a moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail ®</a>,
<a
moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe
®</a>
and <a moz-do-not-send="true"
href="http://www.onebox.com/">Onebox ®</a> are
!
registere d trademarks of <a moz-do-not-send="true"
href="http://www.j2.com/">j2 Global, Inc</a>.
and
its affiliates.</span></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------090600060303000004030204--
12:36 a.m.
New subject: vms in paused state
/run/vdsm/<vmid>.recovery
On Fri, Apr 29, 2016 at 10:59 PM, Bill James <bill.james(a)j2.com> wrote:
where do I find the recovery files?
[root@ovirt1 test vdsm]# pwd
/var/lib/vdsm
[root@ovirt1 test vdsm]# ls -la
total 16
drwxr-xr-x 6 vdsm kvm 100 Mar 17 16:33 .
drwxr-xr-x. 45 root root 4096 Apr 29 12:01 ..
-rw-r--r-- 1 vdsm kvm 10170 Jan 19 05:04 bonding-defaults.json
drwxr-xr-x 2 vdsm root 6 Apr 19 11:34 netconfback
drwxr-xr-x 3 vdsm kvm 54 Apr 19 11:35 persistence
drwxr-x---. 2 vdsm kvm 6 Mar 17 16:33 transient
drwxr-xr-x 2 vdsm kvm 40 Mar 17 16:33 upgrade
[root@ovirt1 test vdsm]# locate recovery
/opt/hp/hpdiags/en/tcstorage.ldinterimrecovery.htm
/opt/hp/hpdiags/en/tcstorage.ldrecoveryready.htm
/usr/share/doc/postgresql-9.2.15/html/archive-recovery-settings.html
/usr/share/doc/postgresql-9.2.15/html/recovery-config.html
/usr/share/doc/postgresql-9.2.15/html/recovery-target-settings.html
/usr/share/pgsql/recovery.conf.sample
/var/lib/nfs/v4recovery
[root@ovirt1 test vdsm]# locate 757a5 (disk id)
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.lease
/ovirt-store/nfs1/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/757a5e69-a791-4391-9d7d-9516bf7f2118/211581dc-fa98-41be-a0b9-ace236149bc2.meta
[root@ovirt1 test vdsm]# locate 5bfb140 (vm id)
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.com.redhat.rhevm.vdsm
/var/lib/libvirt/qemu/channels/5bfb140a-a971-4c9c-82c6-277929eb45d4.org.qemu.guest_agent.0
On 4/29/16 10:02 AM, Michal Skrivanek wrote:
On 29 Apr 2016, at 18:26, Bill James < <bill.james(a)j2.com>
bill.james(a)j2.com> wrote:
yes they are still saying "paused" state.
No, bouncing libvirt didn't help.
Then my suspicion of vm recovery gets closer to a certainty:)
Can you get one of the paused vm's .recovery file from /var/lib/vdsm and
check it says Paused there? It's worth a shot to try to remove that file
and restart vdsm, then check logs and that vm status...it should recover
"good enough" from libvirt only.
Try it with one first
I noticed the errors about the ISO domain. Didn't think that was related.
I have been migrating a lot of VMs to ovirt lately, and recently added
another node.
Also had some problems with /etc/exports for a while, but I think those
issues are all resolved.
Last "unresponsive" message in vdsm.log was:
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::*2016-04-21*
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor become unresponsive
(command timeout, age=310323.97)
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor become unresponsive
(command timeout, age=310323.97)
Thanks.
On 4/29/16 1:40 AM, Michal Skrivanek wrote:
On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com> wrote:
thank you for response.
I bold-ed the ones that are listed as "paused".
[root@ovirt1 test vdsm]# virsh -r list --all
 Id   Name                          State
----------------------------------------------------
Looks like problem started around 2016-04-17 20:19:34,822, based on
engine.log attached.
yes, that time looks correct. Any idea what might have been a trigger?
Anything interesting happened at that time (power outage of some host, some
maintenance action, anything)?Â
logs indicate a problem when vdsm talks to libvirt(all those "monitor
become unresponsive†)
It does seem that at that time you started to have some storage
connectivity issues - first one at 2016-04-17 20:06:53,929. And it
doesn’t look temporary because such errors are still there couple hours
later(in your most recent file you attached I can see at 23:00:54)
When I/O gets blocked the VMs may experience issues (then VM gets Paused),
or their qemu process gets stuck(resulting in libvirt either reporting
error or getting stuck as well -> resulting in what vdsm sees as “monitor
unresponsive†)
Since you now bounced libvirtd - did it help? Do you still see wrong
status for those VMs and still those "monitor unresponsive" errors in
vdsm.log?
If not…then I would suspect the “vm recovery†code not working
correctly. Milan is looking at that.
Thanks,
michal
There's a lot of vdsm logs!
fyi, the storage domain for these Vms is a "local" nfs share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.
attached more logs.
On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
On 27 Apr 2016, at 19:16, Bill James <bill.james(a)j2.com> <bill.james(a)j2.com>
wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file
or directory
you need to run virsh in read-only mode
virsh -r list —all
[root@ovirt1 test vdsm]# systemctl status libvirtd
â— libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset:
enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
└─unlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
â— vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset:
enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
vdsm-4.17.18-0.el7.centos.noarch
the vdsm.log attach is good, but it’s too short interval, it only shows recovery(vdsm
restart) phase when the VMs are identified as paused….can you add earlier logs? Did you
restart vdsm yourself or did it crash?
libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
On 27 Apr 2016, at 02:04, Nir Soffer <nsoffer(a)redhat.com>
<nsoffer(a)redhat.com> wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <bill.james(a)j2.com>
<bill.james(a)j2.com> wrote:
I have a hardware node that has 26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=> update vm_dynamic set status = 1 where vm_guid =(select
vm_guid from vm_static where vm_name = 'api1.test.j2noc.com');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM '242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(api1.test.j2noc.com) moved from 'Up' --> 'Paused'
2016-04-26 15:16:46,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM api1.test.j2noc.com
has been paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
If this is an issue in engine, restarting engine may fix this.
but having this problem only with one node, I don't think this is the issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check vm
status using virsh?
this looks more likely as it seems such status is being reported
logs would help, vdsm.log at the very least.
If virsh thinks that the vms are paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
Users@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
<engine.log-20160421.gz><vdsm.logs.tar.gz>
[image: www.j2.com]
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
This email, its contents and attachments contain information from j2
Global, Inc
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
and/or its affiliates which may be privileged, confidential or otherwise
protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy,
distribution, or use of the contents of this message is prohibited. If you
have received this email in error please notify the sender by reply e-mail
and delete the original message and any copies. © 2015 j2 Global, Inc
<http://www.j2.com/>;. All rights reserved. eFax ® <http://www.efax.com/>;,
eVoice
® <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
FuseMail
® <http://www.fusemail.com/>;, KeepItSafe ® <http://www.keepitsafe.com/>
and Onebox ® <http://www.onebox.com/> are ! registere d trademarks of j2
Global, Inc <http://www.j2.com/>;. and its affiliates.
[image: www.j2.com]
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
This email, its contents and attachments contain information from j2
Global, Inc
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
and/or its affiliates which may be privileged, confidential or otherwise
protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy,
distribution, or use of the contents of this message is prohibited. If you
have received this email in error please notify the sender by reply e-mail
and delete the original message and any copies. © 2015 j2 Global, Inc
<http://www.j2.com/>;. All rights reserved. eFax ® <http://www.efax.com/>;,
eVoice
® <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
FuseMail
® <http://www.fusemail.com/>;, KeepItSafe ® <http://www.keepitsafe.com/>
and Onebox ® <http://www.onebox.com/> are r egistered trademarks of j2
Global, Inc <http://www.j2.com/>;. and its affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:02 a.m.
New subject: vms in paused state
--------------070605020707010208090600
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 8bit
.recovery setting before removing:
p298
sS'status'
p299
S'Paused'
p300
After removing .recovery file and shutdown and restart:
V0
sS'status'
p51
S'Up'
p52
So far looks good, GUI show's VM as Up.
another host was:
p318
sS'status'
p319
S'Paused'
p320
after moving .recovery file and restarting:
V0
sS'status'
p51
S'Up'
Thanks.
On 04/29/2016 02:36 PM, Nir Soffer wrote:
/run/vdsm/<vmid>.recovery
On Fri, Apr 29, 2016 at 10:59 PM, Bill James <bill.james(a)j2.com
<mailto:bill.james@j2.com>> wrote:
where do I find the recovery files?
[root@ovirt1 test vdsm]# pwd
/var/lib/vdsm
[root@ovirt1 test vdsm]# ls -la
total 16
drwxr-xr-x 6 vdsm kvm 100 Mar 17 16:33 .
drwxr-xr-x. 45 root root 4096 Apr 29 12:01 ..
-rw-r--r-- 1 vdsm kvm 10170 Jan 19 05:04 bonding-defaults.json
drwxr-xr-x 2 vdsm root 6 Apr 19 11:34 netconfback
drwxr-xr-x 3 vdsm kvm 54 Apr 19 11:35 persistence
drwxr-x---. 2 vdsm kvm 6 Mar 17 16:33 transient
drwxr-xr-x 2 vdsm kvm 40 Mar 17 16:33 upgrade
On 4/29/16 10:02 AM, Michal Skrivanek wrote:
>
>
> On 29 Apr 2016, at 18:26, Bill James <bill.james(a)j2.com
> <mailto:bill.james@j2.com>> wrote:
>
>> yes they are still saying "paused" state.
>> No, bouncing libvirt didn't help.
>
> Then my suspicion of vm recovery gets closer to a certainty:)
> Can you get one of the paused vm's .recovery file from
> /var/lib/vdsm and check it says Paused there? It's worth a shot
> to try to remove that file and restart vdsm, then check logs and
> that vm status...it should recover "good enough" from libvirt only.
> Try it with one first
>
>> I noticed the errors about the ISO domain. Didn't think that was
>> related.
>> I have been migrating a lot of VMs to ovirt lately, and recently
>> added another node.
>> Also had some problems with /etc/exports for a while, but I
>> think those issues are all resolved.
>>
>>
>> Last "unresponsive" message in vdsm.log was:
>>
>> vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::*2016-04-21*
>> 11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
>> vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor become
>> unresponsive (command timeout, age=310323.97)
>> vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
>> 11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
>> vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor become
>> unresponsive (command timeout, age=310323.97)
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On 4/29/16 1:40 AM, Michal Skrivanek wrote:
>>>
>>>> On 28 Apr 2016, at 19:40, Bill James <bill.james(a)j2.com
>>>> <mailto:bill.james@j2.com>> wrote:
>>>>
>>>> thank you for response.
>>>> I bold-ed the ones that are listed as "paused".
>>>>
>>>>
>>>> [root@ovirt1 test vdsm]# virsh -r list --all
>>>>  Id  Â
>>>> Name                          State
>>>> ----------------------------------------------------
>>>
>>>>
>>>>
>>>> Looks like problem started around 2016-04-17 20:19:34,822,
>>>> based on engine.log attached.
>>>
>>> yes, that time looks correct. Any idea what might have been a
>>> trigger? Anything interesting happened at that time (power
>>> outage of some host, some maintenance action, anything)?Â
>>> logs indicate a problem when vdsm talks to libvirt(all those
>>> "monitor become unresponsiveâ )
>>>
>>> It does seem that at that time you started to have some storage
>>> connectivity issues - first one at 2016-04-17 20:06:53,929.
>>> And it doesnât look temporary because such errors are still
>>> there couple hours later(in your most recent file you attached
>>> I can see at 23:00:54)
>>> When I/O gets blocked the VMs may experience issues (then VM
>>> gets Paused), or their qemu process gets stuck(resulting in
>>> libvirt either reporting error or getting stuck as well ->
>>> resulting in what vdsm sees as âmonitor unresponsiveâ )
>>>
>>> Since you now bounced libvirtd - did it help? Do you still see
>>> wrong status for those VMs and still those "monitor
>>> unresponsive" errors in vdsm.log?
>>> If notâŠthen I would suspect the âvm recoveryâ code not
>>> working correctly. Milan is looking at that.
>>>
>>> Thanks,
>>> michal
>>>
>>>
>>>> There's a lot of vdsm logs!
>>>>
>>>> fyi, the storage domain for these Vms is a "local" nfs
share,
>>>> 7e566f55-e060-47b7-bfa4-ac3c48d70dda.
>>>>
>>>> attached more logs.
>>>>
>>>>
>>>> On 04/28/2016 12:53 AM, Michal Skrivanek wrote:
>>>>>> On 27 Apr 2016, at 19:16, Bill James<bill.james(a)j2.com>
<mailto:bill.james@j2.com> wrote:
>>>>>>
>>>>>> virsh # list --all
>>>>>> error: failed to connect to the hypervisor
>>>>>> error: no valid connection
>>>>>> error: Failed to connect socket to
'/var/run/libvirt/libvirt-sock': No such file or directory
>>>>>>
>>>>> you need to run virsh in read-only mode
>>>>> virsh -r list âall
>>>>>
>>>>>> [root@ovirt1 test vdsm]# systemctl status libvirtd
>>>>>> â libvirtd.service - Virtualization daemon
>>>>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
enabled; vendor preset: enabled)
>>>>>> Drop-In: /etc/systemd/system/libvirtd.service.d
>>>>>> ââunlimited-core.conf
>>>>>> Active: active (running) since Thu 2016-04-21 16:00:03
PDT; 5 days ago
>>>>>>
>>>>>>
>>>>>> tried systemctl restart libvirtd.
>>>>>> No change.
>>>>>>
>>>>>> Attached vdsm.log and supervdsm.log.
>>>>>>
>>>>>>
>>>>>> [root@ovirt1 test vdsm]# systemctl status vdsmd
>>>>>> â vdsmd.service - Virtual Desktop Server Manager
>>>>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service;
enabled; vendor preset: enabled)
>>>>>> Active: active (running) since Wed 2016-04-27 10:09:14
PDT; 3min 46s ago
>>>>>>
>>>>>>
>>>>>> vdsm-4.17.18-0.el7.centos.noarch
>>>>> the vdsm.log attach is good, but itâs too short interval, it
only shows recovery(vdsm restart) phase when the VMs are identified as pausedâŠ.can you
add earlier logs? Did you restart vdsm yourself or did it crash?
>>>>>
>>>>>
>>>>>> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
>>>>>>>> On 27 Apr 2016, at 02:04, Nir
Soffer<nsoffer(a)redhat.com> <mailto:nsoffer@redhat.com> wrote:
>>>>>>>>
>>>>>>>> jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill
James<bill.james(a)j2.com> <mailto:bill.james@j2.com> wrote:
>>>>>>>>> I have a hardware node that has 26 VMs.
>>>>>>>>> 9 are listed as "running", 17 are
listed as "paused".
>>>>>>>>>
>>>>>>>>> In truth all VMs are up and running fine.
>>>>>>>>>
>>>>>>>>> I tried telling the db they are up:
>>>>>>>>>
>>>>>>>>> engine=> update vm_dynamic set status = 1
where vm_guid =(select
>>>>>>>>> vm_guid from vm_static where vm_name =
'api1.test.j2noc.com <http://api1.test.j2noc.com>');
>>>>>>>>>
>>>>>>>>> GUI then shows it up for a short while,
>>>>>>>>>
>>>>>>>>> then puts it back in paused state.
>>>>>>>>>
>>>>>>>>> 2016-04-26 15:16:46,095 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>>>>>>>> (DefaultQuartzScheduler_Worker-16) [157cc21e] VM
'242ca0af-4ab2-4dd6-b515-5
>>>>>>>>> d435e6452c4'(api1.test.j2noc.com
<http://api1.test.j2noc.com>) moved from 'Up' --> 'Paused'
>>>>>>>>> 2016-04-26 15:16:46,221 INFO
[org.ovirt.engine.core.dal.dbbroker.auditlogh
>>>>>>>>> andling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
>>>>>>>>> relation ID: null, Call Stack: null, Custom Event
ID: -1, Message: VM api1.
>>>>>>>>> test.j2noc.com <http://test.j2noc.com> has
been paused.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Why does the engine think the VMs are paused?
>>>>>>>>> Attached engine.log.
>>>>>>>>>
>>>>>>>>> I can fix the problem by powering off the VM then
starting it back up.
>>>>>>>>> But the VM is working fine! How do I get ovirt to
realize that?
>>>>>>>> If this is an issue in engine, restarting engine may
fix this.
>>>>>>>> but having this problem only with one node, I
don't think this is the issue.
>>>>>>>>
>>>>>>>> If this is an issue in vdsm, restarting vdsm may fix
this.
>>>>>>>>
>>>>>>>> If this does not help, maybe this is libvirt issue?
did you try to check vm
>>>>>>>> status using virsh?
>>>>>>> this looks more likely as it seems such status is being
reported
>>>>>>> logs would help, vdsm.log at the very least.
>>>>>>>
>>>>>>>> If virsh thinks that the vms are paused, you can try
to restart libvirtd.
>>>>>>>>
>>>>>>>> Please file a bug about this in any case with engine
and vdsm logs.
>>>>>>>>
>>>>>>>> Adding Michal in case he has better idea how to
proceed.
>>>>>>>>
>>>>>>>> Nir
>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>> <engine.log-20160421.gz><vdsm.logs.tar.gz>
>>>
>>
>> www.j2.com
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>
>> This email, its contents and attachments contain information
>> from j2 Global, Inc
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>> and/or its affiliates which may be privileged, confidential or
>> otherwise protected from disclosure. The information is intended
>> to be for the addressee(s) only. If you are not an addressee,
>> any disclosure, copy, distribution, or use of the contents of
>> this message is prohibited. If you have received this email in
>> error please notify the sender by reply e-mail and delete the
>> original message and any copies. © 2015 j2 Global, Inc
>> <http://www.j2.com/>;. All rights reserved. eFax ®
>> <http://www.efax.com/>;, eVoice ® <http://www.evoice.com/>;,
>> Campaigner ® <http://www.campaigner.com/>;, FuseMail ®
>> <http://www.fusemail.com/>;, KeepItSafe ®
>> <http://www.keepitsafe.com/> and Onebox ®
>> <http://www.onebox.com/> are ! registere d trademarks of j2
>> Global, Inc <http://www.j2.com/>;. and its affiliates.
>>
www.j2.com
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
This email, its contents and attachments contain information from
j2 Global, Inc
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended
to be for the addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this
message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original
message and any copies. © 2015 j2 Global, Inc
<http://www.j2.com/>;. All rights reserved. eFax ®
<http://www.efax.com/>;, eVoice ® <http://www.evoice.com/>;,
Campaigner ® <http://www.campaigner.com/>;, FuseMail ®
<http://www.fusemail.com/>;, KeepItSafe ®
<http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
and its affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------070605020707010208090600
Content-Type: text/html; charset="windows-1252"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
.recovery setting before removing:<br>
p298<br>
sS'status'<br>
p299<br>
S'Paused'<br>
p300<br>
<br>
<br>
<br>
After removing .recovery file and shutdown and restart:<br>
V0<br>
sS'status'<br>
p51<br>
S'Up'<br>
p52<br>
<br>
<br>
So far looks good, GUI show's VM as Up.<br>
<br>
<br>
another host was:<br>
p318<br>
sS'status'<br>
p319<br>
S'Paused'<br>
p320<br>
<br>
after moving .recovery file and restarting:<br>
V0<br>
sS'status'<br>
p51<br>
S'Up'<br>
<br>
<br>
Thanks.<br>
<br>
<div class="moz-cite-prefix">On 04/29/2016 02:36 PM, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyyt8v2KnW74a066bBdtkhPd6TeaeoPop7t9oNqjyJi4efA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div dir="ltr">/run/vdsm/<vmid>.recovery</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Apr 29, 2016 at 10:59 PM, Bill
James <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bill.james@j2.com"
target="_blank">bill.james(a)j2.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> where do I
find the
recovery files?<br>
<br>
[root@ovirt1 test vdsm]# pwd<br>
/var/lib/vdsm<br>
[root@ovirt1 test vdsm]# ls -la<br>
total 16<br>
drwxr-xr-x 6 vdsm kvm 100 Mar 17 16:33 .<br>
drwxr-xr-x. 45 root root 4096 Apr 29 12:01 ..<br>
-rw-r--r-- 1 vdsm kvm 10170 Jan 19 05:04
bonding-defaults.json<br>
drwxr-xr-x 2 vdsm root 6 Apr 19 11:34 netconfback<br>
drwxr-xr-x 3 vdsm kvm 54 Apr 19 11:35 persistence<br>
drwxr-x---. 2 vdsm kvm 6 Mar 17 16:33 transient<br>
drwxr-xr-x 2 vdsm kvm 40 Mar 17 16:33 upgrade<br>
<br>
<div>
<div class="h5"> <br>
<br>
<div>On 4/29/16 10:02 AM, Michal Skrivanek wrote:<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div><br>
On 29 Apr 2016, at 18:26, Bill James <<a
moz-do-not-send="true"
href="mailto:bill.james@j2.com"
target="_blank"><a class="moz-txt-link-abbreviated"
href="mailto:bill.james@j2.com">bill.james@j2.com</a></a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div> yes they are still saying "paused"
state.<br>
No, bouncing libvirt didn't help.<br>
</div>
</blockquote>
<div><br>
</div>
Then my suspicion of vm recovery gets closer to a
certainty:)
<div>Can you get one of the paused vm's .recovery
file from /var/lib/vdsm and check it says Paused
there? It's worth a shot to try to remove that
file and restart vdsm, then check logs and that vm
status...it should recover "good enough" from
libvirt only. </div>
<div>Try it with one first<br>
<div>
<div><br>
<blockquote type="cite">
<div> I noticed the errors about the ISO
domain. Didn't think that was related.<br>
I have been migrating a lot of VMs to
ovirt lately, and recently added another
node.<br>
Also had some problems with /etc/exports
for a while, but I think those issues are
all resolved.<br>
<br>
<br>
Last "unresponsive" message in vdsm.log
was:<br>
<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::<b>2016-04-21</b>
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`b6a13808-9552-401b-840b-4f7022e8293d`::monitor
become unresponsive (command timeout,
age=310323.97)<br>
vdsm.log.49.xz:jsonrpc.Executor/0::WARNING::2016-04-21
11:00:54,703::vm::5067::virt.vm::(_setUnresponsiveIfTimeout)
vmId=`5bfb140a-a971-4c9c-82c6-277929eb45d4`::monitor
become unresponsive (command timeout,
age=310323.97)<br>
<br>
<br>
<br>
Thanks.<br>
<br>
<br>
<br>
<div>On 4/29/16 1:40 AM, Michal Skrivanek
wrote:<br>
</div>
<blockquote type="cite"> <br>
<div>
<blockquote type="cite">
<div>On 28 Apr 2016, at 19:40, Bill
James <<a
moz-do-not-send="true"
href="mailto:bill.james@j2.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:bill.james@j2.com">bill.james@j2.com</a></a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF"
text="#000000"> thank you for
response.<br>
I bold-ed the ones that are
listed as "paused".<br>
<br>
<br>
[root@ovirt1 test vdsm]# virsh
-r list --all<br>
 Id  Â
Name                         Â
State<br>
----------------------------------------------------<br>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF"
text="#000000"> <br>
<br>
Looks like problem started
around 2016-04-17 20:19:34,822,
based on engine.log attached.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>yes, that time looks correct. Any
idea what might have been a trigger?
Anything interesting happened at
that time (power outage of some
host, some maintenance action,
anything)? </div>
<div>logs indicate a problem when vdsm
talks to libvirt(all those "monitor
become unresponsiveâ )</div>
<div><br>
</div>
<div>It does seem that at that time
you started to have some storage
connectivity issues - first one
at 2016-04-17 20:06:53,929. And it
doesnât look temporary because
such errors are still there couple
hours later(in your most recent file
you attached I can see at 23:00:54)</div>
<div>When I/O gets blocked the VMs may
experience issues (then VM gets
Paused), or their qemu process gets
stuck(resulting in libvirt either
reporting error or getting stuck as
well -> resulting in what vdsm
sees as âmonitor unresponsiveâ )</div>
<div><br>
</div>
<div>Since you now bounced libvirtd -
did it help? Do you still see wrong
status for those VMs and still those
"monitor unresponsive" errors in
vdsm.log?</div>
<div>If notâŠthen I would suspect the
âvm recoveryâ code not working
correctly. Milan is looking at that.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>michal</div>
<div>
<div><br>
</div>
</div>
<div><br>
</div>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF"
text="#000000"> There's a lot of
vdsm logs!<br>
<br>
fyi, the storage domain for
these Vms is a "local" nfs
share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br>
<br>
attached more logs.<br>
<br>
<br>
<div>On 04/28/2016 12:53 AM,
Michal Skrivanek wrote:<br>
</div>
<blockquote type="cite">
<blockquote type="cite">
<pre>On 27 Apr 2016, at 19:16, Bill
James <a moz-do-not-send="true" href="mailto:bill.james@j2.com"
target="_blank">&lt;bill.james(a)j2.com&gt;</a> wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file
or directory
</pre>
</blockquote>
<pre>you need to run virsh in read-only
mode
virsh -r list âall
</pre>
<blockquote type="cite">
<pre>[root@ovirt1 test vdsm]# systemctl
status libvirtd
â libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset:
enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
ââunlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
â vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre>the vdsm.log attach is good, but
itâs too short interval, it only shows recovery(vdsm restart) phase when the VMs are
identified as pausedâŠ.can you add earlier logs? Did you restart vdsm yourself or did it
crash?
</pre>
<blockquote type="cite">
<pre>libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre>On 27 Apr 2016, at 02:04, Nir
Soffer <a moz-do-not-send="true" href="mailto:nsoffer@redhat.com"
target="_blank">&lt;nsoffer(a)redhat.com&gt;</a> wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a moz-do-not-send="true"
href="mailto:bill.james@j2.com"
target="_blank">&lt;bill.james(a)j2.com&gt;</a> wrote:
</pre>
<blockquote type="cite">
<pre>I have a hardware node that
has 26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=> update vm_dynamic set status = 1 where vm_guid =(select
vm_guid from vm_static where vm_name = '<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
target="_blank">api1.test.j2noc.com</a>');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM '242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(<a moz-do-not-send="true"
href="http://api1.test.j2noc.com"
target="_blank">api1.test.j2noc.com</a>) moved from 'Up'
--> 'Paused'
2016-04-26 15:16:46,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM api1.
<a moz-do-not-send="true" href="http://test.j2noc.com"
target="_blank">test.j2noc.com</a> has been paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre>If this is an issue in engine,
restarting engine may fix this.
but having this problem only with one node, I don't think this is the issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check vm
status using virsh?
</pre>
</blockquote>
<pre>this looks more likely as it
seems such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type="cite">
<pre>If virsh thinks that the vms
are paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre><a
moz-do-not-send="true" href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
</pre>
</blockquote>
</blockquote>
<br>
</div>
<span><engine.log-20160421.gz></span><span><vdsm.logs.tar.gz></span></div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
target="_blank"><span
style="color:windowtext;text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" border="0"
height="46"
width="391"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments
contain information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
target="_blank">j2 Global,
Inc</a>.
and/or its affiliates which may be
privileged, confidential or otherwise
protected from disclosure. The
information is intended to be for the
addressee(s) only. If you are not an
addressee, any disclosure, copy,
distribution, or use of the contents
of this message is prohibited. If you
have received this email in error
please notify the sender by reply
e-mail and delete the original message
and any copies. © 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/"
target="_blank">j2 Global,
Inc</a>.
All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/"
target="_blank">eFax ®</a>, <a
moz-do-not-send="true"
href="http://www.evoice.com/"
target="_blank">eVoice ®</a>,
<a
moz-do-not-send="true"
href="http://www.campaigner.com/"
target="_blank">Campaigner ®</a>,
<a
moz-do-not-send="true"
href="http://www.fusemail.com/"
target="_blank">FuseMail ®</a>,
<a
moz-do-not-send="true"
href="http://www.keepitsafe.com/"
target="_blank">KeepItSafe ®</a>
and
<a moz-do-not-send="true"
href="http://www.onebox.com/"
target="_blank">Onebox ®</a> are !
registere d trademarks of <a
moz-do-not-send="true"
href="http://www.j2.com/"
target="_blank">j2 Global,
Inc</a>.
and its affiliates.</span></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
target="_blank"><span
style="color:windowtext;text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" border="0"
height="46"
width="391"></span></a></p>
</div>
</div>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain
information from <a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
target="_blank">j2 Global, Inc</a>. and/or its
affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information
is intended to be for the addressee(s) only. If you
are not an addressee, any disclosure, copy,
distribution, or use of the contents of this message
is prohibited. If you have received this email in
error please notify the sender by reply e-mail and
delete the original message and any copies. © 2015 <a
moz-do-not-send="true" href="http://www.j2.com/"
target="_blank">j2 Global, Inc</a>. All rights
reserved. <a moz-do-not-send="true"
href="http://www.efax.com/"
target="_blank">eFax ®</a>,
<a moz-do-not-send="true"
href="http://www.evoice.com/"
target="_blank">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/"
target="_blank">Campaigner
®</a>, <a moz-do-not-send="true"
href="http://www.fusemail.com/"
target="_blank">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/"
target="_blank">KeepItSafe
®</a> and <a moz-do-not-send="true"
href="http://www.onebox.com/"
target="_blank">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true" href="http://www.j2.com/"
target="_blank">j2 Global, Inc</a>. and its
affiliates.</span></p>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------070605020707010208090600--
5:34 p.m.
New subject: vms in paused state
Bill James <bill.james(a)j2.com> writes:
.recovery setting before removing:
p298
sS'status'
p299
S'Paused'
p300
After removing .recovery file and shutdown and restart:
V0
sS'status'
p51
S'Up'
p52
Thank you for the information. I was able to reproduce the problem with
mistakenly reported paused state when Vdsm receives unexpected data from
libvirt. I'll try to look at it.
Restarting Vdsm (4.17.18 and some newer versions) afterwards remedies
the problem for me, even without removing the recovery file.
Milan
10:44 a.m.
New subject: vms in paused state
We've found out that if libvirtd got restarted then VMs with disabled
memory balloon device are wrongly reported as being in the paused state.
It's a bug and we're working on a fix.
3115
days inactive
3129
days old
7 comments
4 participants
participants (4)
-
Bill James -
Michal Skrivanek -
Milan Zamazal -
Nir Soffer