On 14 Dec 2017, at 13:22, Francesco Romani <fromani@redhat.com> wrote: =20 =20 On 12/14/2017 01:12 PM, Michal Skrivanek wrote:
=20
On 14 Dec 2017, at 13:00, Dafna Ron <dron@redhat.com = <mailto:dron@redhat.com>> wrote: =20 Hi,=20 We have a failure on basic suite on test: = 004_basic_sanity.disk_operations I think that we query a snapshot that was already deleted = successfully and report the snapshot as gone. and that is because of a = different error in update vm query which happens before.=20 =20 Link and headline of suspected patches:=20 https://gerrit.ovirt.org/#/c/85168/ = <https://gerrit.ovirt.org/#/c/85168/> - core: Prevent retry lease = hotplag in case of failure. =20 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393 = <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393> =20 Link to all logs: = http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifac= t/ = <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifa= ct/> =20 (Relevant) error snippet from the log:=20 <error> =20 =20 vdsm:=20 =20 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) = [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", =
res =3D method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line =
201, in _dynamicMethod
result =3D fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line =
48, in method
ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in =
getAllVmStats
statsList =3D self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line =
518, in getAllVmStats
return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1699, in getStats
oga_stats =3D self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1895, in _getGuestStats
self._update_guest_disk_mapping() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
1909, in _update_guest_disk_mapping
self._sync_metadata() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line =
4995, in _sync_metadata
self._md_desc.dump(self._dom) File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", =
dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT, File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", =
--Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 line 606, in _handle_request line 477, in dump line 47, in __getattr__
% self.vmid) NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not =
defined yet or was undefined =20 it doesn=E2=80=99t seem to be relevant to this failure, but it = deserves a fix nevertheless Francesco? =20 =20 Smells like a race on shutdown.
and suspend/resume looks harmless, but we want to clear up as many tracebacks as we can:)
=20 --=20 Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
--Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 14 Dec 2017, at 13:22, Francesco Romani <<a = href=3D"mailto:fromani@redhat.com" class=3D"">fromani@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""> =20 <meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8" class=3D""> =20 <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <br class=3D""> <div class=3D"moz-cite-prefix">On 12/14/2017 01:12 PM, Michal Skrivanek wrote:<br class=3D""> </div> <blockquote type=3D"cite" = cite=3D"mid:E8C90687-357B-46C4-902D-2FB5077F49B9@redhat.com" class=3D""> <meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8" class=3D""> <br class=3D""> <div class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 14 Dec 2017, at 13:00, Dafna Ron <<a = href=3D"mailto:dron@redhat.com" class=3D"" = moz-do-not-send=3D"true">dron@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8" class=3D""> <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><p = class=3D"">Hi, <br class=3D""> </p><p class=3D"">We have a failure on basic suite on = test: 004_basic_sanity.disk_operations<br class=3D""> </p> I think that we query a snapshot that was already deleted successfully and report the snapshot as gone. and that is because of a different error in update vm query which happens before. <br class=3D""> <br class=3D""> <b style=3D"font-weight:normal;" = id=3D"docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=3D"">= <div style=3D"line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><span = style=3D"font-size: 11pt; font-family: Arial; background-color: = transparent; font-weight: 400; font-style: normal; = font-variant-ligatures: normal; font-variant-caps: normal; = font-variant-east-asian: normal; font-variant-position: normal; = text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" = class=3D"">Link and headline of suspected patches: </span></div> <a class=3D"moz-txt-link-freetext" = href=3D"https://gerrit.ovirt.org/#/c/85168/" = moz-do-not-send=3D"true">https://gerrit.ovirt.org/#/c/85168/</a> - core: Prevent retry lease hotplag in case of = failure.<br class=3D""> <br class=3D""> <div style=3D"line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><span = style=3D"font-size: 11pt; font-family: Arial; background-color: = transparent; font-weight: 400; font-style: normal; = font-variant-ligatures: normal; font-variant-caps: normal; = font-variant-east-asian: normal; font-variant-position: normal; = text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" = class=3D"">Link to Job:</span></div> <a class=3D"moz-txt-link-freetext" = href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393= " = moz-do-not-send=3D"true">http://jenkins.ovirt.org/job/ovirt-master_change-= queue-tester/4393</a><br class=3D""> <br class=3D""> <div style=3D"line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><span = style=3D"font-size: 11pt; font-family: Arial; background-color: = transparent; font-weight: 400; font-style: normal; = font-variant-ligatures: normal; font-variant-caps: normal; = font-variant-east-asian: normal; font-variant-position: normal; = text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" = class=3D"">Link to all logs:</span></div> <tt class=3D""><a class=3D"moz-txt-link-freetext" = href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393= /artifact/" = moz-do-not-send=3D"true">http://jenkins.ovirt.org/job/ovirt-master_change-= queue-tester/4393/artifact/</a></tt><br class=3D""> <br class=3D""> <div style=3D"line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><span = style=3D"font-size: 11pt; font-family: Arial; background-color: = transparent; font-weight: 400; font-style: normal; = font-variant-ligatures: normal; font-variant-caps: normal; = font-variant-east-asian: normal; font-variant-position: normal; = text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" = class=3D"">(Relevant) error snippet from the log: </span></div> <div style=3D"line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><span = style=3D"font-size: 11pt; font-family: Arial; background-color: = transparent; font-weight: 400; font-style: normal; = font-variant-ligatures: normal; font-variant-caps: normal; = font-variant-east-asian: normal; font-variant-position: normal; = text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" = class=3D""><error></span></div> <b class=3D""><br class=3D""> <br class=3D""> vdsm: </b></b></div> </div> </blockquote> <blockquote type=3D"cite" class=3D""> <div class=3D""> <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><b = style=3D"font-weight:normal;" = id=3D"docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" = class=3D""><b class=3D""> <br class=3D""> 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611)<br class=3D""> Traceback (most recent call last):<br class=3D""> File = "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request<br class=3D""> res =3D method(**params)<br = class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod<br class=3D""> result =3D fn(*methodArgs)<br = class=3D""> File "<string>", line 2, in = getAllVmStats<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method<br class=3D""> ret =3D func(*args, **kwargs)<br = class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in getAllVmStats<br class=3D""> statsList =3D = self._cif.getAllVmStats()<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 518, in getAllVmStats<br class=3D""> return [v.getStats() for v in self.vmContainer.values()]<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1699, in getStats<br class=3D""> oga_stats =3D = self._getGuestStats()<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1895, in _getGuestStats<br class=3D""> = self._update_guest_disk_mapping()<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1909, in _update_guest_disk_mapping<br class=3D""> self._sync_metadata()<br class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4995, in _sync_metadata<br class=3D""> self._md_desc.dump(self._dom)<br = class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 477, in dump<br class=3D""> = dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,<br class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__<br class=3D""> % self.vmid)<br class=3D""> NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or was undefined<br class=3D""> </b></b></div> </div> </blockquote> <div class=3D""><br class=3D""> </div> it doesn=E2=80=99t seem to be relevant to this failure, but it = deserves a fix nevertheless</div> <div class=3D"">Francesco?</div> <div class=3D""><br class=3D""> </div> </blockquote> <br class=3D""> Smells like a race on shutdown.<br = class=3D""></div></div></blockquote><div><br class=3D""></div>and = suspend/resume</div><div>looks harmless, but we want to clear up as many = tracebacks as we can:)</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div text=3D"#000000" bgcolor=3D"#FFFFFF" = class=3D""> <br class=3D""> <pre class=3D"moz-signature" cols=3D"72">--=20 Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh</pre> </div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_2483837C-6135-491B-8C53-0A451D1F04CF--