5:25 a.m.
--Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 14 Dec 2017, at 13:22, Francesco Romani <fromani(a)redhat.com>
wrote:
=20
=20
On 12/14/2017 01:12 PM, Michal Skrivanek wrote:
>=20
>> On 14 Dec 2017, at 13:00, Dafna Ron <dron(a)redhat.com =
<mailto:dron@redhat.com>> wrote:
>>=20
>> Hi,=20
>> We have a failure on basic suite on test: =
004_basic_sanity.disk_operations
>> I think that we query a snapshot that was already deleted =
successfully and report the snapshot as gone. and that is because of a =
different error in update vm query which happens before.=20
>>=20
>> Link and headline of suspected patches:=20
>> https://gerrit.ovirt.org/#/c/85168/ =
<https://gerrit.ovirt.org/#/c/85168/> - core: Prevent retry lease =
hotplag in case of failure.
>>=20
>> Link to Job:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393 =
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393>
>>=20
>> Link to all logs:
>> =
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifac=
t/ =
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifa=
ct/>
>>=20
>> (Relevant) error snippet from the log:=20
>> <error>
>>=20
>>=20
>> vdsm:=20
>>=20
>> 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) =
[jsonrpc.JsonRpcServer]
Internal server error (__init__:611)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", =
line 606, in _handle_request
>> res =3D method(**params)
>> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line =
201, in _dynamicMethod
>> result =3D fn(*methodArgs)
>> File "<string>", line 2, in getAllVmStats
>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line =
48, in method
>> ret =3D func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in =
getAllVmStats
>> statsList =3D self._cif.getAllVmStats()
>> File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line =
518, in getAllVmStats
>> return [v.getStats() for v in self.vmContainer.values()]
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1699, in getStats
>> oga_stats =3D self._getGuestStats()
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1895, in _getGuestStats
>> self._update_guest_disk_mapping()
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1909, in _update_guest_disk_mapping
>> self._sync_metadata()
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
4995, in _sync_metadata
>> self._md_desc.dump(self._dom)
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", =
line 477, in dump
>> dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", =
line 47, in __getattr__
>> % self.vmid)
>> NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not =
defined yet or was undefined
>=20
> it doesn=E2=80=99t seem to be relevant to this failure, but it =
deserves a fix
nevertheless
> Francesco?
>=20
=20
Smells like a race on shutdown.
and suspend/resume
looks harmless, but we want to clear up as many tracebacks as we can:)
=20
--=20
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
--Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 14 Dec 2017, at 13:22, Francesco Romani <<a =
href=3D"mailto:fromani@redhat.com"
class=3D"">fromani(a)redhat.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D"">
=20
<meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Dutf-8" class=3D"">
=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 12/14/2017 01:12 PM, Michal
Skrivanek wrote:<br class=3D"">
</div>
<blockquote type=3D"cite" =
cite=3D"mid:E8C90687-357B-46C4-902D-2FB5077F49B9@redhat.com"
class=3D"">
<meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Dutf-8" class=3D"">
<br class=3D"">
<div class=3D"">
<blockquote type=3D"cite" class=3D"">
<div class=3D"">On 14 Dec 2017, at 13:00, Dafna Ron
<<a =
href=3D"mailto:dron@redhat.com" class=3D"" =
moz-do-not-send=3D"true">dron(a)redhat.com</a>&gt;
wrote:</div>
<br class=3D"Apple-interchange-newline">
<div class=3D"">
<meta http-equiv=3D"content-type" content=3D"text/html;
charset=3Dutf-8" class=3D"">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D""><p =
class=3D"">Hi, <br class=3D"">
</p><p class=3D"">We have a failure on basic suite on
=
test:
004_basic_sanity.disk_operations<br class=3D"">
</p>
I think that we query a snapshot that was already deleted
successfully and report the snapshot as gone. and that is
because of a different error in update vm query which
happens before. <br class=3D"">
<br class=3D"">
<b style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868"
class=3D"">=
<div style=3D"line-height: 1.38; margin-top: 0pt;
margin-bottom: 0pt;" class=3D""><span =
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link and headline of suspected patches: </span></div>
<a class=3D"moz-txt-link-freetext" =
href=3D"https://gerrit.ovirt.org/#/c/85168/" =
moz-do-not-send=3D"true">https://gerrit.ovirt.org/#/c/85168/...
- core: Prevent retry lease hotplag in case of =
failure.<br class=3D"">
<br class=3D"">
<div style=3D"line-height: 1.38; margin-top: 0pt;
margin-bottom: 0pt;" class=3D""><span =
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link to Job:</span></div>
<a class=3D"moz-txt-link-freetext" =
href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste...
" =
moz-do-not-send=3D"true">http://jenkins.ovirt.org/job/ovirt-...
queue-tester/4393</a><br class=3D"">
<br class=3D"">
<div style=3D"line-height: 1.38; margin-top: 0pt;
margin-bottom: 0pt;" class=3D""><span =
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link to all logs:</span></div>
<tt class=3D""><a
class=3D"moz-txt-link-freetext" =
href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste...
/artifact/" =
moz-do-not-send=3D"true">http://jenkins.ovirt.org/job/ovirt-...
queue-tester/4393/artifact/</a></tt><br class=3D"">
<br class=3D"">
<div style=3D"line-height: 1.38; margin-top: 0pt;
margin-bottom: 0pt;" class=3D""><span =
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">(Relevant) error snippet from the log: </span></div>
<div style=3D"line-height: 1.38; margin-top: 0pt;
margin-bottom: 0pt;" class=3D""><span =
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D""><error></span></div>
<b class=3D""><br class=3D"">
<br class=3D"">
vdsm: </b></b></div>
</div>
</blockquote>
<blockquote type=3D"cite" class=3D"">
<div class=3D"">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D""><b =
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" =
class=3D""><b class=3D""> <br class=3D"">
2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7)
[jsonrpc.JsonRpcServer] Internal server error
(__init__:611)<br class=3D"">
Traceback (most recent call last):<br class=3D"">
File
=
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 606, in _handle_request<br class=3D"">
res =3D method(**params)<br =
class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
line 201, in _dynamicMethod<br class=3D"">
result =3D fn(*methodArgs)<br =
class=3D"">
File "<string>", line 2, in =
getAllVmStats<br class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line 48, in method<br class=3D"">
ret =3D func(*args, **kwargs)<br =
class=3D"">
File =
"/usr/lib/python2.7/site-packages/vdsm/API.py",
line 1342, in getAllVmStats<br class=3D"">
statsList =3D =
self._cif.getAllVmStats()<br class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
line 518, in getAllVmStats<br class=3D"">
return [v.getStats() for v in
self.vmContainer.values()]<br class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 1699, in getStats<br class=3D"">
oga_stats =3D =
self._getGuestStats()<br class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 1895, in _getGuestStats<br class=3D"">
=
self._update_guest_disk_mapping()<br class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 1909, in _update_guest_disk_mapping<br class=3D"">
self._sync_metadata()<br
class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 4995, in _sync_metadata<br class=3D"">
self._md_desc.dump(self._dom)<br =
class=3D"">
File
=
"/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py",
line 477, in dump<br class=3D"">
=
dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,<br class=3D"">
File
=
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line 47, in __getattr__<br class=3D"">
% self.vmid)<br
class=3D"">
NotConnectedError: VM
'7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined
yet or was undefined<br class=3D"">
</b></b></div>
</div>
</blockquote>
<div class=3D""><br class=3D"">
</div>
it doesn=E2=80=99t seem to be relevant to this failure, but it =
deserves
a fix nevertheless</div>
<div class=3D"">Francesco?</div>
<div class=3D""><br class=3D"">
</div>
</blockquote>
<br class=3D"">
Smells like a race on shutdown.<br =
class=3D""></div></div></blockquote><div><br
class=3D""></div>and =
suspend/resume</div><div>looks harmless, but we want to clear up as many =
tracebacks as we can:)</div><div><br class=3D""><blockquote
type=3D"cite" =
class=3D""><div class=3D""><div text=3D"#000000"
bgcolor=3D"#FFFFFF" =
class=3D"">
<br class=3D"">
<pre class=3D"moz-signature" cols=3D"72">--=20
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh</pre>
</div>
</div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF--