[ OST Failure Report ] [ oVirt Master ] [ 14-12-2017 ] [ 004_basic_sanity.disk_operations ]
Dafna Ron
dron at redhat.com
Thu Dec 14 12:00:06 UTC 2017
Hi,
We have a failure on basic suite on test: 004_basic_sanity.disk_operations
I think that we query a snapshot that was already deleted successfully
and report the snapshot as gone. and that is because of a different
error in update vm query which happens before.
*
Link and headline of suspected patches:
https://gerrit.ovirt.org/#/c/85168/ - core: Prevent retry lease hotplag
in case of failure.
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artifact/
(Relevant) error snippet from the log:
<error>
Engine:
*2017-12-14 02:38:01,470-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [] Command
'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-1, VdsIdVD
SCommandParametersBase:{hostId='f1c2314f-5617-4c9e-84e0-e822160fbde9'})'
execution failed: VDSGenericException: VDSErrorException: Failed to
GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "VM
'7cab7e5a-cb12-4977-ac4f-652185
32df7e' was not defined yet or was undefined"}, code = -32603
2017-12-14 02:38:01,470-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericE
xception: VDSErrorException: Failed to GetAllVmStatsVDS, error =
Internal JSON-RPC error: {'reason': "VM
'7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or was
undefined"}, code = -32603
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:224)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:194)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown
Source) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
[:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_151]
at
org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
[common.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_151]
at
org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown
Source) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42)
[vdsbroker.jar:]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_151]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[rt.jar:1.8.0_151]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383)
[javax.enterprise.concurrent-1.0.jar:]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534)
[javax.enterprise.concurrent-1.0.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151]
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
[javax.enterprise.concurrent-1.0.jar:]
at
org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
*
*2017-12-14 02:42:15,525-05 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] Ending command
'org.ovirt.engine.core.bll.snapshots.Re
moveSnapshotCommand' successfully.
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default
task-16) [] Entered SsoRestApiAuthFilter
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default
task-16) [] SsoRestApiAuthFilter authenticating with sso
2017-12-14 02:42:15,528-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default
task-16) [] SsoRestApiAuthFilter authenticating using BEARER header
2017-12-14 02:42:15,531-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default
task-16) [] SsoRestApiAuthFilter successfully authenticated using BEARER
header
2017-12-14 02:42:15,531-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default
task-16) [] Entered SsoRestApiNegotiationFilter
2017-12-14 02:42:15,533-05 DEBUG
[org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default
task-16) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth
2017-12-14 02:42:15,540-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] method: get, params:
[cc9ae472-ef79-456f-9e4
7-01fd1617e189], timeElapsed: 3ms
2017-12-14 02:42:15,543-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2] method:
runQuery, params: [GetAllVmSnapshotsByVmId, IdQueryParameters:{refresh
='false', filtered='false'}], timeElapsed: 6ms
2017-12-14 02:42:15,548-05 WARN
[org.ovirt.engine.core.bll.GetVmConfigurationBySnapshotQuery] (default
task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2] Snapshot
'20b155eb-c15d-42cb-a03f-c45813a045b8' does not exist
2017-12-14 02:42:15,548-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-16) [2c5d880a-5c36-41e7-8020-bb2007cf9be2] method:
runQuery, params: [GetVmConfigurationBySnapshot, IdQueryParameters:{re
fresh='false', filtered='false'}], timeElapsed: 5ms
2017-12-14 02:42:15,548-05 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-16) [] Operation Failed: Entity not found: null
2017-12-14 02:42:15,554-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-26)
[c5f1846d-d042-4271-86f1-2b1787684554] EVENT_ID:
USER_REMOVE_SNAPSHOT_FINISHED_SUC
CESS(356), Snapshot 'dead_snap1' deletion for VM 'vm1' has been completed.
2017-12-14 02:42:16,660-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] START,
GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdVDS
CommandParametersBase:{hostId='3c60017b-0bb3-4a3e-bc24-f1779100fe21'}),
log id: f9a1165
2017-12-14 02:42:16,660-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] SEND
destination:jms.topic.vdsm_requests
reply-to:jms.topic.vdsm_responses
content-length:103
vdsm:
2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
Internal server error (__init__:611)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
606, in _handle_request
res = method(**params)
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201,
in _dynamicMethod
result = fn(*methodArgs)
File "<string>", line 2, in getAllVmStats
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48,
in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in
getAllVmStats
statsList = self._cif.getAllVmStats()
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 518, in
getAllVmStats
return [v.getStats() for v in self.vmContainer.values()]
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1699, in
getStats
oga_stats = self._getGuestStats()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1895, in
_getGuestStats
self._update_guest_disk_mapping()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1909, in
_update_guest_disk_mapping
self._sync_metadata()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4995, in
_sync_metadata
self._md_desc.dump(self._dom)
File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line
477, in dump
dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
47, in __getattr__
% self.vmid)
NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not
defined yet or was undefined
*
</error>
*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20171214/c2cb515e/attachment-0001.html>
More information about the Infra
mailing list