hot plug disk test failed on system tests

FYI, We're seeing a single test 'add hotplug disk' failing recently on OST, the test only failed once on the jobs [1] and was fixed on its own after one run, but now I see it also on check-patch [2] jobs. Looking at the changelog, I can't find immediate suspects, and if this is a race it might make sense it got in at some point and didn't fail on the 1st time. Anyone can have a look to help debug this? it also failed once on 4.0 job. I see some exceptions on VDSM.LOG [3] [1] http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_exp... [2] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-fc24-x... [3] 2016-12-11 04:51:19,055 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getDeviceList succeeded in 0.39 seconds (__init__:515) 2016-12-11 04:51:19,106 INFO (jsonrpc/5) [dispatcher] Run and protect: getAllTasksInfo(spUUID=None, options=None) (logUtils:49) 2016-12-11 04:51:19,106 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='ef8d6b6f-256c-4420-9998-bef8a140a20f') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2193, in getAllTasksInfo allTasksInfo = self._pool.getAllTasksInfo() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2016-12-11 04:51:19,109 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='ef8d6b6f-256c-4420-9998-bef8a140a20f') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1175) 2016-12-11 04:51:19,109 ERROR (jsonrpc/5) [storage.Dispatcher] Secured object is not in safe state (dispatcher:80) Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 105, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1183, in prepare raise self.error second error: 2016-12-11 04:55:21,837 ERROR (libvirt/events) [vds] Error running VM callback (clientIF:543) Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 514, in dispatchLibvirtEvents v.onLibvirtLifecycleEvent(event, detail, None) File "/usr/share/vdsm/virt/vm.py", line 4318, in onLibvirtLifecycleEvent elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY: AttributeError: 'module' object has no attribute 'VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY' -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)