Hi, There is a new failure on on master in experimental flow, the failing test is 'add_secondary_storage_domain', the engine.log has few exceptions: 2017-01-09 10:07:24,943-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Command 'PollVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call 2017-01-09 10:07:24,943-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] ... 2017-01-09 10:10:23,323-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-09 10:10:23,323-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] ... 2017-01-09 10:10:43,704-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name at [Source: [B@6a84a0d0; line: 1, column: 889]: org.codehaus.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name at [Source: [B@6a84a0d0; line: 1, column: 889] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName2(ReaderBasedParser.java:1042) [jackson-core-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName(ReaderBasedParser.java:1008) [jackson-core-asl-1.9.13.jar:1.9.13] .... <JsonRpcRequest id: "7711f770-dbef-44be-9f9e-2d8a2bfae937", method: Host.getAllVmStats, params: {}> 2017-01-09 10:11:33,336-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParameters Base:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Unrecognized message received 2017-01-09 10:11:33,336-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNe tworkException: Unrecognized message received at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] VDSM logs on host1: 2017-01-09 10:11:27,120 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) 2017-01-09 10:11:27,452 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for unfetched domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:151) 2017-01-09 10:11:27,453 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:168) 2017-01-09 10:11:27,552 WARN (jsonrpc/4) [storage.LVM] lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).' , ' Volume group "80985016-bdd8-4778-abd9-becc8fedcab4" not found', ' Cannot process volume group 80985016-bdd8-4778-abd9-becc8fedcab4'] (lvm:377) 2017-01-09 10:11:27,559 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) 2017-01-09 10:11:27,560 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='e2381f1f-eee5-4922-a56d-f6ca40d76eec') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1159, in attachStorageDomain pool.attachSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 924, in attachSD dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 112, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',) .... 2017-01-09 10:19:31,467 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2212, in getAllTasksInfo raise se.SpmStatusError() SpmStatusError: Not SPM: () 2017-01-09 10:19:31,471 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175) 2017-01-09 10:19:31,471 ERROR (jsonrpc/6) [storage.Dispatcher] {'status': {'message': 'Not SPM: ()', 'code': 654}} (dispatcher:77) 2017-01-09 10:19:31,472 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksInfo failed (error 654) in 0.01 seconds (__init__:515) 2017-01-09 10:19:31,479 INFO (jsonrpc/7) [dispatcher] Run and protect: getAllTasksStatuses(spUUID=None, options=None) (logUtils:49) 2017-01-09 10:19:31,479 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2172, in getAllTasksStatuses raise se.SpmStatusError() SpmStatusError: Not SPM: () 2017-01-09 10:19:31,480 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175) Full engine logs can be found here: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art... VDSM host1 logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art... Rest of the logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4643/art... Could someone take a look? Thanks, Nadav.