8:38 p.m.
Hi,
There is a new failure on on master in experimental flow, the failing test
is 'add_secondary_storage_domain', the engine.log has few exceptions:
2017-01-09 10:07:24,943-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
(org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Command
'PollVDSCommand(HostName = lago-basic-suite-master-host1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed:
VDSGenericException: VDSNetworkException: Timeout during rpc call
2017-01-09 10:07:24,943-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
(org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Timeout during rpc call
at
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73)
[vdsbroker.jar:]
...
2017-01-09 10:10:23,323-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler10) [7cad9211] Command
'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed:
VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-09 10:10:23,323-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler10) [7cad9211] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Heartbeat exceeded
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
[vdsbroker.jar:]
...
2017-01-09 10:10:43,704-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) []
Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using
backslash to be included in name
at [Source: [B@6a84a0d0; line: 1, column: 889]:
org.codehaus.jackson.JsonParseException: Illegal unquoted character
((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
name
at [Source: [B@6a84a0d0; line: 1, column: 889]
at
org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
[jackson-core-asl-1.9.13.jar:1.9.13]
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
[jackson-core-asl-1.9.13.jar:1.9.13]
at
org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
[jackson-core-asl-1.9.13.jar:1.9.13]
at
org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName2(ReaderBasedParser.java:1042)
[jackson-core-asl-1.9.13.jar:1.9.13]
at
org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName(ReaderBasedParser.java:1008)
[jackson-core-asl-1.9.13.jar:1.9.13]
....
<JsonRpcRequest id: "7711f770-dbef-44be-9f9e-2d8a2bfae937", method:
Host.getAllVmStats, params: {}>
2017-01-09 10:11:33,336-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler7) [7cad9211] Command
'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1,
VdsIdVDSCommandParameters
Base:{runAsync='true',
hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})'
execution failed: VDSGenericException: VDSNetworkException: Unrecognized
message received
2017-01-09 10:11:33,336-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler7) [7cad9211] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNe
tworkException: Unrecognized message received
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23)
[vdsbroker.jar:]
VDSM logs on host1:
2017-01-09 10:11:27,120 ERROR (jsonrpc/4) [storage.StorageDomainCache]
domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'80985016-bdd8-4778-abd9-becc8fedcab4',)
2017-01-09 10:11:27,452 ERROR (jsonrpc/4) [storage.StorageDomainCache]
looking for unfetched domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:151)
2017-01-09 10:11:27,453 ERROR (jsonrpc/4) [storage.StorageDomainCache]
looking for domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:168)
2017-01-09 10:11:27,552 WARN (jsonrpc/4) [storage.LVM] lvm vgs failed: 5
[] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.',
' WARNING: To avoid corruption, rescan devices to make changes visible
(pvscan --cache).'
, ' Volume group "80985016-bdd8-4778-abd9-becc8fedcab4" not found',
'
Cannot process volume group 80985016-bdd8-4778-abd9-becc8fedcab4'] (lvm:377)
2017-01-09 10:11:27,559 ERROR (jsonrpc/4) [storage.StorageDomainCache]
domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'80985016-bdd8-4778-abd9-becc8fedcab4',)
2017-01-09 10:11:27,560 ERROR (jsonrpc/4) [storage.TaskManager.Task]
(Task='e2381f1f-eee5-4922-a56d-f6ca40d76eec') Unexpected error (task:870)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 877, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in
wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1159, in attachStorageDomain
pool.attachSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
79, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 924, in attachSD
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 112, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'80985016-bdd8-4778-abd9-becc8fedcab4',)
....
2017-01-09 10:19:31,467 ERROR (jsonrpc/6) [storage.TaskManager.Task]
(Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') Unexpected error (task:870)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 877, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in
wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2212, in getAllTasksInfo
raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
2017-01-09 10:19:31,471 INFO (jsonrpc/6) [storage.TaskManager.Task]
(Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') aborting: Task is aborted:
'Not SPM' - code 654 (task:1175)
2017-01-09 10:19:31,471 ERROR (jsonrpc/6) [storage.Dispatcher] {'status':
{'message': 'Not SPM: ()', 'code': 654}} (dispatcher:77)
2017-01-09 10:19:31,472 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
Host.getAllTasksInfo failed (error 654) in 0.01 seconds (__init__:515)
2017-01-09 10:19:31,479 INFO (jsonrpc/7) [dispatcher] Run and protect:
getAllTasksStatuses(spUUID=None, options=None) (logUtils:49)
2017-01-09 10:19:31,479 ERROR (jsonrpc/7) [storage.TaskManager.Task]
(Task='2841da07-b3b4-4573-ae38-b1500f793221') Unexpected error (task:870)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 877, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in
wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2172, in getAllTasksStatuses
raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
2017-01-09 10:19:31,480 INFO (jsonrpc/7) [storage.TaskManager.Task]
(Task='2841da07-b3b4-4573-ae38-b1500f793221') aborting: Task is aborted:
'Not SPM' - code 654 (task:1175)
Full engine logs can be found here:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art...
VDSM host1 logs:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art...
Rest of the logs:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art...
Could someone take a look?
Thanks,
Nadav.