<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 14 Dec 2017, at 13:00, Dafna Ron <<a href="mailto:dron@redhat.com" class="">dron@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="content-type" content="text/html; charset=utf-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi, <br class="">
</p><p class="">We have a failure on basic suite on test:
004_basic_sanity.disk_operations<br class="">
</p>
I think that we query a snapshot that was already deleted
successfully and report the snapshot as gone. and that is because of
a different error in update vm query which happens before. <br class="">
<br class="">
<b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link and headline of suspected patches: </span></div>
<a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/85168/">https://gerrit.ovirt.org/#/c/85168/</a> - core: Prevent retry lease
hotplag in case of failure.<br class="">
<br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to Job:</span></div>
<a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393</a><br class="">
<br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to all logs:</span></div>
<a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/</a><br class="">
<br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">(Relevant) error snippet from the log: </span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""><error></span></div>
<br class="">
Engine: <br class="">
<br class="">
<br class="">
<br class="">
<b class="">2017-12-14 02:38:01,470-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [] Command
'GetAllVmStatsVDSCommand(HostName =
lago-basic-suite-master-host-1, VdsIdVD<br class="">
SCommandParametersBase:{hostId='f1c2314f-5617-4c9e-84e0-e822160fbde9'})'
execution failed: VDSGenericException: VDSErrorException: Failed
to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason':
"VM '7cab7e5a-cb12-4977-ac4f-652185<br class="">
32df7e' was not defined yet or was undefined"}, code = -32603<br class="">
2017-12-14 02:38:01,470-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericE<br class="">
xception: VDSErrorException: Failed to GetAllVmStatsVDS, error =
Internal JSON-RPC error: {'reason': "VM
'7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or
was undefined"}, code = -32603<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:224)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:194)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown
Source) [vdsbroker.jar:]<br class="">
at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown
Source) [:1.8.0_151]<br class="">
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_151]<br class="">
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_151]<br class="">
at
org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
[common.jar:]<br class="">
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown
Source) [:1.8.0_151]<br class="">
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_151]<br class="">
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_151]<br class="">
at
org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br class="">
at
org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown
Source) [vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57)
[vdsbroker.jar:]<br class="">
at
org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42)
[vdsbroker.jar:]<br class="">
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_151]<br class="">
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[rt.jar:1.8.0_151]<br class="">
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383)
[javax.enterprise.concurrent-1.0.jar:]<br class="">
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534)
[javax.enterprise.concurrent-1.0.jar:]<br class="">
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_151]<br class="">
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_151]<br class="">
at java.lang.Thread.run(Thread.java:748)
[rt.jar:1.8.0_151]<br class="">
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
[javax.enterprise.concurrent-1.0.jar:]<br class="">
at
org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)<br class="">
<br class="">
</b><br class="">
<br class="">
<b class="">2017-12-14 02:42:15,525-05 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] Ending command
'org.ovirt.engine.core.bll.snapshots.Re<br class="">
moveSnapshotCommand' successfully.<br class="">
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter]
(default task-16) [] Entered SsoRestApiAuthFilter<br class="">
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter]
(default task-16) [] SsoRestApiAuthFilter authenticating with
sso<br class="">
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter]
(default task-16) [] SsoRestApiAuthFilter authenticating using
BEARER header<br class="">
2017-12-14 02:42:15,531-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter]
(default task-16) [] SsoRestApiAuthFilter successfully
authenticated using BEARER header<br class="">
2017-12-14 02:42:15,531-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter]
(default task-16) [] Entered SsoRestApiNegotiationFilter<br class="">
2017-12-14 02:42:15,533-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter]
(default task-16) [] SsoRestApiNegotiationFilter Not performing
Negotiate Auth<br class="">
2017-12-14 02:42:15,540-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] method: get, params:
[cc9ae472-ef79-456f-9e4<br class="">
7-01fd1617e189], timeElapsed: 3ms<br class="">
2017-12-14 02:42:15,543-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2] method:
runQuery, params: [GetAllVmSnapshotsByVmId,
IdQueryParameters:{refresh<br class="">
='false', filtered='false'}], timeElapsed: 6ms<br class="">
2017-12-14 02:42:15,548-05 WARN
[org.ovirt.engine.core.bll.GetVmConfigurationBySnapshotQuery]
(default task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2]
Snapshot '20b155eb-c15d-42cb-a03f-c45813a045b8' does not exist<br class="">
2017-12-14 02:42:15,548-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2] method:
runQuery, params: [GetVmConfigurationBySnapshot,
IdQueryParameters:{re<br class="">
fresh='false', filtered='false'}], timeElapsed: 5ms<br class="">
2017-12-14 02:42:15,548-05 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource]
(default task-16) [] Operation Failed: Entity not found: null<br class="">
2017-12-14 02:42:15,554-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] EVENT_ID:
USER_REMOVE_SNAPSHOT_FINISHED_SUC<br class="">
CESS(356), Snapshot 'dead_snap1' deletion for VM 'vm1' has been
completed.<br class="">
2017-12-14 02:42:16,660-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] START,
GetAllVmStatsVDSCommand(HostName =
lago-basic-suite-master-host-0, VdsIdVDS<br class="">
CommandParametersBase:{hostId='3c60017b-0bb3-4a3e-bc24-f1779100fe21'}),
log id: f9a1165<br class="">
2017-12-14 02:42:16,660-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] SEND<br class="">
destination:jms.topic.vdsm_requests<br class="">
reply-to:jms.topic.vdsm_responses<br class="">
content-length:103<br class="">
<br class="">
vdsm: </b></b></div></div></blockquote><blockquote type="cite" class=""><div class=""><div text="#000000" bgcolor="#FFFFFF" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=""><b class="">
<br class="">
2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7)
[jsonrpc.JsonRpcServer] Internal server error (__init__:611)<br class="">
Traceback (most recent call last):<br class="">
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 606, in _handle_request<br class="">
res = method(**params)<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
line 201, in _dynamicMethod<br class="">
result = fn(*methodArgs)<br class="">
File "<string>", line 2, in getAllVmStats<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line 48, in method<br class="">
ret = func(*args, **kwargs)<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line
1342, in getAllVmStats<br class="">
statsList = self._cif.getAllVmStats()<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line
518, in getAllVmStats<br class="">
return [v.getStats() for v in self.vmContainer.values()]<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
1699, in getStats<br class="">
oga_stats = self._getGuestStats()<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
1895, in _getGuestStats<br class="">
self._update_guest_disk_mapping()<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
1909, in _update_guest_disk_mapping<br class="">
self._sync_metadata()<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
4995, in _sync_metadata<br class="">
self._md_desc.dump(self._dom)<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py",
line 477, in dump<br class="">
dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,<br class="">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
47, in __getattr__<br class="">
% self.vmid)<br class="">
NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was
not defined yet or was undefined<br class=""></b></b></div></div></blockquote><div><br class=""></div>it doesn’t seem to be relevant to this failure, but it deserves a fix nevertheless</div><div>Francesco?</div><div><br class=""><blockquote type="cite" class=""><div class=""><div text="#000000" bgcolor="#FFFFFF" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-54dd-82fe-8fc8-8fab5e981868" class=""><b class="">
<br class="">
</b><br class="">
<br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""></error></span></div>
</b><br class="Apple-interchange-newline">
</div>
_______________________________________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" class="">Devel@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote></div><br class=""></body></html>