There is a new failure on on master in experimental flow, the failing test is 'add_secondary_storage_domain', the engine.log has few exceptions:

2017-01-09 10:07:24,943-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Command 'PollVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call
2017-01-09 10:07:24,943-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-2) [e9e4e3b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call
        at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:]
...
2017-01-09 10:10:23,323-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-09 10:10:23,323-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded
        at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:]
...
2017-01-09 10:10:43,704-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name
 at [Source: [B@6a84a0d0; line: 1, column: 889]: org.codehaus.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name
 at [Source: [B@6a84a0d0; line: 1, column: 889]
        at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) [jackson-core-asl-1.9.13.jar:1.9.13]
        at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521) [jackson-core-asl-1.9.13.jar:1.9.13]
        at org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482) [jackson-core-asl-1.9.13.jar:1.9.13]
        at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName2(ReaderBasedParser.java:1042) [jackson-core-asl-1.9.13.jar:1.9.13]
        at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName(ReaderBasedParser.java:1008) [jackson-core-asl-1.9.13.jar:1.9.13]
....


<JsonRpcRequest id: "7711f770-dbef-44be-9f9e-2d8a2bfae937", method: Host.getAllVmStats, params: {}>
2017-01-09 10:11:33,336-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParameters
Base:{runAsync='true', hostId='f6ad90f7-1b37-49f0-a958-7151efa0039c'})' execution failed: VDSGenericException: VDSNetworkException: Unrecognized message received
2017-01-09 10:11:33,336-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7cad9211] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNe
tworkException: Unrecognized message received
        at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:]

VDSM logs on host1:

2017-01-09 10:11:27,120 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',)
2017-01-09 10:11:27,452 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for unfetched domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:151)
2017-01-09 10:11:27,453 ERROR (jsonrpc/4) [storage.StorageDomainCache] looking for domain 80985016-bdd8-4778-abd9-becc8fedcab4 (sdc:168)
2017-01-09 10:11:27,552 WARN  (jsonrpc/4) [storage.LVM] lvm vgs failed: 5 [] ['  WARNING: Not using lvmetad because config setting use_lvmetad=0.', '  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).'
, '  Volume group "80985016-bdd8-4778-abd9-becc8fedcab4" not found', '  Cannot process volume group 80985016-bdd8-4778-abd9-becc8fedcab4'] (lvm:377)
2017-01-09 10:11:27,559 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain 80985016-bdd8-4778-abd9-becc8fedcab4 not found (sdc:157)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',)
2017-01-09 10:11:27,560 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='e2381f1f-eee5-4922-a56d-f6ca40d76eec') Unexpected error (task:870)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 877, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1159, in attachStorageDomain
    pool.attachSD(sdUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 924, in attachSD
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 112, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'80985016-bdd8-4778-abd9-becc8fedcab4',)
....
2017-01-09 10:19:31,467 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') Unexpected error (task:870)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 877, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2212, in getAllTasksInfo
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
2017-01-09 10:19:31,471 INFO  (jsonrpc/6) [storage.TaskManager.Task] (Task='700015ba-4aed-4eaf-961b-5a4373b2d4d7') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175)
2017-01-09 10:19:31,471 ERROR (jsonrpc/6) [storage.Dispatcher] {'status': {'message': 'Not SPM: ()', 'code': 654}} (dispatcher:77)
2017-01-09 10:19:31,472 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksInfo failed (error 654) in 0.01 seconds (__init__:515)
2017-01-09 10:19:31,479 INFO  (jsonrpc/7) [dispatcher] Run and protect: getAllTasksStatuses(spUUID=None, options=None) (logUtils:49)
2017-01-09 10:19:31,479 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') Unexpected error (task:870)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 877, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2172, in getAllTasksStatuses
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
2017-01-09 10:19:31,480 INFO  (jsonrpc/7) [storage.TaskManager.Task] (Task='2841da07-b3b4-4573-ae38-b1500f793221') aborting: Task is aborted: 'Not SPM' - code 654 (task:1175)

Full engine logs can be found here: