Re: [ovirt-users] ovirt 4.1 hosted engine: clear Task error: Operation is not allowed in this task state: ("can't clean in state running", )

Hi all, ovirt 4.1 hosted engine on 2 node cluster and FC LUN Storage I'm trying to clear some task pending from months using vdsClient but I can't do anything. Below are the steps (on node 1, the SPM): 1. Show all tasks: *# vdsClient -s 0 getAllTasksInfofd319af4-d160-48ce-b682-5a908333a5e1 : verb = createVolume id = fd319af4-d160-48ce-b682-5a908333a5e19bbc2bc4-3c73-4814-a785-6ea737904528 : verb = prepareMerge id = 9bbc2bc4-3c73-4814-a785-6ea737904528e70feb21-964d-49d9-9b5a-8e3f70a92db1 : verb = prepareMerge id = e70feb21-964d-49d9-9b5a-8e3f70a92db1cf064461-f0ab-4e44-a68f-b2d58fa83a21 : verb = prepareMerge id = cf064461-f0ab-4e44-a68f-b2d58fa83a2185b7cf4e-d658-4785-94f0-391fe9616b41 : verb = prepareMerge id = 85b7cf4e-d658-4785-94f0-391fe9616b417416627a-fe50-4353-b129-e01bba066a66 : verb = prepareMerge id = 7416627a-fe50-4353-b129-e01bba066a66* 2. Stop all tasks (repeted for every task): *# vdsClient -s 0 stopTask 7416627a-fe50-4353-b129-e01bba066a66 * Task is aborted: u'7416627a-fe50-4353-b129-e01bba066a66' - code 411 3. Tring to clear tasks: * # vdsClient -s 0 clearTask 7416627a-fe50-4353-b129-e01bba066a66* *Operation is not allowed in this task state: ("can't clean in state running",)* *On Node 01 (the SPM) I have multiple errors in /var/log/vdsm/vdsm.log like this:* *2017-10-11 15:09:53,719+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='9519d4db-2960-4b88-82f2-e4c1094eac54') aborting: Task is aborted: u'Operation is not allowed in this task state: ("can\'t clean in state running",)' - code 100 (task:1175)* *2017-10-11 15:09:53,719+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH clearTask error=Operation is not allowed in this task state: ("can't clean in state running",) (dispatcher:78)* *2017-10-11 15:09:53,720+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.clear failed (error 410) in 0.01 seconds (__init__:539)* *2017-10-11 15:09:53,743+0200 INFO (jsonrpc/6) [vdsm.api] START clearTask(taskID=u'7416627a-fe50-4353-b129-e01bba066a66', spUUID=None, options=None) from=::ffff:192.168.0.226,36724, flow_id=7cd340ec (api:46)* *2017-10-11 15:09:53,743+0200 INFO (jsonrpc/6) [vdsm.api] FINISH clearTask error=Operation is not allowed in this task state: ("can't clean in state running",) from=::ffff:192.168.0.226,36724, flow_id=7cd340ec (api:50)* *2017-10-11 15:09:53,743+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='0e12e052-2aca-480d-b50f-5de01ddebe35') Unexpected error (task:870)* *Traceback (most recent call last):* * File "/usr/share/vdsm/storage/task.py", line 877, in _run* * return fn(*args, **kargs)* * File "<string>", line 2, in clearTask* * File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method* * ret = func(*args, **kwargs)* * File "/usr/share/vdsm/storage/hsm.py", line 2258, in clearTask* * return self.taskMng.clearTask(taskID=taskID)* * File "/usr/share/vdsm/storage/taskManager.py", line 175, in clearTask* * t.clean()* * File "/usr/share/vdsm/storage/task.py", line 1047, in clean* * raise se.TaskStateError("can't clean in state %s" % self.state)* *TaskStateError: Operation is not allowed in this task state: ("can't clean in state running",)* *On Node 02 (is a 2 node cluster) I have other errors (I don't know if are related):* *2017-10-11 15:11:57,083+0200 INFO (jsonrpc/7) [storage.LVM] Refreshing lvs: vg=b50c1f5c-aa2c-4a53-9f89-83517fa70d3b lvs=['leases'] (lvm:1291)* *2017-10-11 15:11:57,084+0200 INFO (jsonrpc/7) [storage.LVM] Refreshing LVs (vg=b50c1f5c-aa2c-4a53-9f89-83517fa70d3b, lvs=['leases']) (lvm:1319)* *2017-10-11 15:11:57,124+0200 INFO (jsonrpc/7) [storage.VolumeManifest] b50c1f5c-aa2c-4a53-9f89-83517fa70d3b/d42f671e-1745-46c1-9e1c-2833245675fc/c86afaa5-6ca8-4fcb-a27e-ffbe0133fe23 info is {'status': 'OK', 'domain': 'b50c1f5c-aa2c-4a53-9f89-83517fa70d3b', 'voltype': 'LEAF', 'description': 'hosted-engine.metadata', 'parent': '00000000-0000-0000-0000-000000000000', 'format': 'RAW', 'generation': 0, 'image': 'd42f671e-1745-46c1-9e1c-2833245675fc', 'ctime': '1499437345', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '134217728', 'children': [], 'pool': '', 'capacity': '134217728', 'uuid': u'c86afaa5-6ca8-4fcb-a27e-ffbe0133fe23', 'truesize': '134217728', 'type': 'PREALLOCATED', 'lease': {'owners': [], 'version': None}} (volume:272)* *2017-10-11 15:11:57,125+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getVolumeInfo return={'info': {'status': 'OK', 'domain': 'b50c1f5c-aa2c-4a53-9f89-83517fa70d3b', 'voltype': 'LEAF', 'description': 'hosted-engine.metadata', 'parent': '00000000-0000-0000-0000-000000000000', 'format': 'RAW', 'generation': 0, 'image': 'd42f671e-1745-46c1-9e1c-2833245675fc', 'ctime': '1499437345', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '134217728', 'children': [], 'pool': '', 'capacity': '134217728', 'uuid': u'c86afaa5-6ca8-4fcb-a27e-ffbe0133fe23', 'truesize': '134217728', 'type': 'PREALLOCATED', 'lease': {'owners': [], 'version': None}}} from=::1,56906 (api:52)* *2017-10-11 15:11:57,126+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Volume.getInfo succeeded in 0.05 seconds (__init__:539)* *2017-10-11 15:11:57,758+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:56908 (protocoldetector:72)* *2017-10-11 15:11:57,764+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:56908 (protocoldetector:127)* *2017-10-11 15:11:57,765+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)* *2017-10-11 15:11:57,765+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:130)* *2017-10-11 15:11:57,930+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:539)* *2017-10-11 15:11:57,933+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::1,56908 (api:46)* *2017-10-11 15:11:57,933+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'b50c1f5c-aa2c-4a53-9f89-83517fa70d3b': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000138003', 'lastCheck': '4.9', 'valid': True}, u'b6730d64-2cf8-42a3-8f08-24b8cc2c0cd8': {'code': 200, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.7', 'valid': False}, u'c7d32f1b-f32c-4a21-995b-2e3b415aae4e': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618471', 'lastCheck': '1.4', 'valid': True}, u'05ab1dd9-24bc-409b-80b8-6c5b00c52aa9': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00027591', 'lastCheck': '5.2', 'valid': True}} from=::1,56908 (api:52)* *2017-10-11 15:11:57,998+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.06 seconds (__init__:539)* *2017-10-11 15:11:58,253+0200 ERROR (monitor/b6730d6) [storage.Monitor] Setting up monitor for b6730d64-2cf8-42a3-8f08-24b8cc2c0cd8 failed (monitor:329)* *Traceback (most recent call last):* * File "/usr/share/vdsm/storage/monitor.py", line 326, in _setupLoop* * self._setupMonitor()* * File "/usr/share/vdsm/storage/monitor.py", line 349, in _setupMonitor* * self._produceDomain()* * File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 401, in wrapper* * value = meth(self, *a, **kw)* * File "/usr/share/vdsm/storage/monitor.py", line 367, in _produceDomain* * self.domain = sdCache.produce(self.sdUUID)* * File "/usr/share/vdsm/storage/sdc.py", line 112, in produce* * domain.getRealDomain()* * File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain* * return self._cache._realProduce(self._sdUUID)* * File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce* * domain = self._findDomain(sdUUID)* * File "/usr/share/vdsm/storage/sdc.py", line 153, in _findDomain* * return findMethod(sdUUID)* * File "/usr/share/vdsm/storage/nfsSD.py", line 126, in findDomain* * return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))* * File "/usr/share/vdsm/storage/fileSD.py", line 359, in __init__* * manifest = self.manifestClass(domainPath)* * File "/usr/share/vdsm/storage/fileSD.py", line 171, in __init__* * sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata)* * File "/usr/share/vdsm/storage/sd.py", line 332, in __init__* * self._domainLock = self._makeDomainLock()* * File "/usr/share/vdsm/storage/sd.py", line 526, in _makeDomainLock* * domVersion = self.getVersion()* * File "/usr/share/vdsm/storage/sd.py", line 403, in getVersion* * return self.getMetaParam(DMDK_VERSION)* * File "/usr/share/vdsm/storage/sd.py", line 400, in getMetaParam* * return self._metadata[key]* * File "/usr/lib/python2.7/site-packages/vdsm/storage/persistent.py", line 91, in __getitem__* * return dec(self._dict[key])* * File "/usr/lib/python2.7/site-packages/vdsm/storage/persistent.py", line 203, in __getitem__* * raise KeyError(key)* *KeyError: 'VERSION'* Can you help me? Restart hosted engine don't solve the problem Thank you p.s. Related question: tasks above are the same/related reported by the engine in the screenshot here? https://snag.gy/XDmoUt.jpg ... How Can I clear also these tasks from engine? _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users