This is a multi-part message in MIME format. --------------7C4A69F5EC47C0DE9F67DB09 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 12/14/2017 01:12 PM, Michal Skrivanek wrote:
On 14 Dec 2017, at 13:00, Dafna Ron <dron@redhat.com <mailto:dron@redhat.com>> wrote:
Hi,
We have a failure on basic suite on test: 004_basic_sanity.disk_operations
I think that we query a snapshot that was already deleted successfully and report the snapshot as gone. and that is because of a different error in update vm query which happens before.
* Link and headline of suspected patches: https://gerrit.ovirt.org/#/c/85168/ - core: Prevent retry lease hotplag in case of failure.
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/
(Relevant) error snippet from the log: <error> *
vdsm: ** ** 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 518, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1699, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1895, in _getGuestStats self._update_guest_disk_mapping() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1909, in _update_guest_disk_mapping self._sync_metadata() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4995, in _sync_metadata self._md_desc.dump(self._dom) File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 477, in dump dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT, File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or was undefined **
it doesn’t seem to be relevant to this failure, but it deserves a fix nevertheless Francesco?
Smells like a race on shutdown. -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh --------------7C4A69F5EC47C0DE9F67DB09 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 12/14/2017 01:12 PM, Michal Skrivanek wrote:<br> </div> <blockquote type="cite" cite="mid:E8C90687-357B-46C4-902D-2FB5077F49B9@redhat.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <br class=""> <div> <blockquote type="cite" class=""> <div class="">On 14 Dec 2017, at 13:00, Dafna Ron <<a href="mailto:dron@redhat.com" class="" moz-do-not-send="true">dron@redhat.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <meta http-equiv="content-type" content="text/html; charset=utf-8" class=""> <div text="#000000" bgcolor="#FFFFFF" class=""> <p class="">Hi, <br class=""> </p> <p class="">We have a failure on basic suite on test: 004_basic_sanity.disk_operations<br class=""> </p> I think that we query a snapshot that was already deleted successfully and report the snapshot as gone. and that is because of a different error in update vm query which happens before. <br class=""> <br class=""> <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=""> <div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link and headline of suspected patches: </span></div> <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/85168/" moz-do-not-send="true">https://gerrit.ovirt.org/#/c/85168/</a> - core: Prevent retry lease hotplag in case of failure.<br class=""> <br class=""> <div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to Job:</span></div> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393</a><br class=""> <br class=""> <div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to all logs:</span></div> <tt><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/</a></tt><br class=""> <br class=""> <div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">(Relevant) error snippet from the log: </span></div> <div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""><error></span></div> <b class=""><br class=""> <br class=""> vdsm: </b></b></div> </div> </blockquote> <blockquote type="cite" class=""> <div class=""> <div text="#000000" bgcolor="#FFFFFF" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=""><b class=""> <br class=""> 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611)<br class=""> Traceback (most recent call last):<br class=""> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request<br class=""> res = method(**params)<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod<br class=""> result = fn(*methodArgs)<br class=""> File "<string>", line 2, in getAllVmStats<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method<br class=""> ret = func(*args, **kwargs)<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in getAllVmStats<br class=""> statsList = self._cif.getAllVmStats()<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 518, in getAllVmStats<br class=""> return [v.getStats() for v in self.vmContainer.values()]<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1699, in getStats<br class=""> oga_stats = self._getGuestStats()<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1895, in _getGuestStats<br class=""> self._update_guest_disk_mapping()<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1909, in _update_guest_disk_mapping<br class=""> self._sync_metadata()<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4995, in _sync_metadata<br class=""> self._md_desc.dump(self._dom)<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 477, in dump<br class=""> dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,<br class=""> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__<br class=""> % self.vmid)<br class=""> NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or was undefined<br class=""> </b></b></div> </div> </blockquote> <div><br class=""> </div> it doesn’t seem to be relevant to this failure, but it deserves a fix nevertheless</div> <div>Francesco?</div> <div><br class=""> </div> </blockquote> <br> Smells like a race on shutdown.<br> <br> <pre class="moz-signature" cols="72">-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh</pre> </body> </html> --------------7C4A69F5EC47C0DE9F67DB09--