<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Feb 21, 2017 at 11:47 AM, Fred Rolland <span dir="ltr">&lt;<a href="mailto:frolland@redhat.com" target="_blank">frolland@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Add before the command (with your db password): PGPASSWORD=engine<br><br>for example:<br>PGPASSWORD=engine /usr/share/ovirt-engine/setup/<wbr>dbutils/taskcleaner.sh -T<br><br>PGPASSWORD=engine /usr/share/ovirt-engine/setup/<wbr>dbutils/unlock_entity.sh -t disk -u engine -q<br></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br></div></div></div></blockquote><div><br></div><div>From taskcleaner, if I use the &quot;-T&quot; option I get error</div><div><br></div><div><div>[root@ovmgr1 ovirt-engine]# PGPASSWORD=my_pwd /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -d engine -u engine -T</div><div> t</div><div>ERROR:  column &quot;job_id&quot; does not exist</div><div>LINE 1: ...created_at,status,return_value,return_value_class,job_id,ste...</div><div>                                                             ^</div><div>FATAL: Cannot execute sql command: --command=SELECT command_id,command_type,root_command_id,command_parameters,command_params_class,created_at,status,return_value,return_value_class,job_id,step_id,executed FROM GetAllCommandsWithRunningTasks();</div></div><div><br></div><div>I see the function GetAllCommandsWithRunningTasks as defined only in /usr/share/ovirt-engine/setup/dbutils/taskcleaner_sp_3_5.sql</div><div>and it seems it makes query on commands_entities, but if I directly go inside db, the table doesn&#39;t contain indeed a job_id column....</div><div> </div><div>I&#39;m on 4.1 upgraded from 4.0.6</div><div><br></div><div><div>engine=# \d command_entities</div><div>                          Table &quot;public.command_entities&quot;</div><div>        Column         |           Type           |            Modifiers            </div><div>-----------------------+--------------------------+---------------------------------</div><div> command_id            | uuid                     | not null</div><div> command_type          | integer                  | not null</div><div> root_command_id       | uuid                     | </div><div> command_parameters    | text                     | </div><div> command_params_class  | character varying(256)   | </div><div> created_at            | timestamp with time zone | </div><div> status                | character varying(20)    | default NULL::character varying</div><div> callback_enabled      | boolean                  | default false</div><div> callback_notified     | boolean                  | default false</div><div> return_value          | text                     | </div><div> return_value_class    | character varying(256)   | </div><div> executed              | boolean                  | default false</div><div> user_id               | uuid                     | </div><div> parent_command_id     | uuid                     | </div><div> data                  | text                     | </div><div> engine_session_seq_id | bigint                   | </div><div> command_context       | text                     | </div><div>Indexes:</div><div>    &quot;pk_command_entities&quot; PRIMARY KEY, btree (command_id)</div><div>    &quot;idx_root_command_id&quot; btree (root_command_id) WHERE root_command_id IS NOT NULL</div><div>Referenced by:</div><div>    TABLE &quot;command_assoc_entities&quot; CONSTRAINT &quot;fk_coco_command_assoc_entity&quot; FOREIGN KEY (command_id) REFERENCES comm</div><div>and_entities(command_id) ON DELETE CASCADE</div><div><br></div><div>engine=# </div></div><div><br></div><div>Anyway after unlocking the disk and retrying the move, I get the same error while creating auto snapshot... the first problem on host (that is a different host from the chosen yesterday) seems </div><div><br></div><div>MetaDataKeyNotFoundError: Meta Data key not found error: (&quot;Missing metadata key: &#39;DOMAIN&#39;: found: {&#39;NONE&#39;: <br></div><div><br></div><div><br></div><div><div>2017-02-21 11:38:58,985 INFO  (jsonrpc/0) [dispatcher] Run and protect: createVolume(sdUUID=u&#39;900b1853-e192-4661-a0f9-7c7c396f6f49&#39;, spUUID=u&#39;588237b8-0031-02f6-035d-000000000136&#39;, imgUUID=u&#39;f0b5a0e4-ee5d-44a7-ba07-08285791368a&#39;, size=u&#39;461708984320&#39;, volFormat=4, preallocate=2, diskType=2, volUUID=u&#39;c39c3d9f-dde8-45ab-b4a9-7c3b45c6391d&#39;, desc=u&#39;&#39;, srcImgUUID=u&#39;f0b5a0e4-ee5d-44a7-ba07-08285791368a&#39;, srcVolUUID=u&#39;7ed43974-1039-4a68-a8b3-321e7594fe4c&#39;, initialSize=None) (logUtils:49)</div><div>2017-02-21 11:38:58,987 INFO  (jsonrpc/0) [IOProcessClient] Starting client ioprocess-6269 (__init__:330)</div><div>2017-02-21 11:38:59,006 INFO  (ioprocess/32170) [IOProcess] Starting ioprocess (__init__:452)</div><div>2017-02-21 11:38:59,040 INFO  (jsonrpc/0) [dispatcher] Run and protect: createVolume, Return response: None (logUtils:52)</div><div>2017-02-21 11:38:59,053 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Volume.create succeeded in 0.07 seconds (__init__:515)</div><div>2017-02-21 11:38:59,054 INFO  (tasks/9) [storage.ThreadPool.WorkerThread] START task 08d7797a-af46-489f-ada0-c70bf4359366 (cmd=&lt;bound method Task.commit of &lt;storage.task.Task instance at 0x7f8eb84aacf8&gt;&gt;, args=None) (threadPool:208)</div><div>2017-02-21 11:38:59,150 WARN  (tasks/9) [storage.ResourceManager] Resource factory failed to create resource &#39;01_img_900b1853-e192-4661-a0f9-7c7c396f6f49.f0b5a0e4-ee5d-44a7-ba07-08285791368a&#39;. Canceling request. (resourceManager:542)</div><div>Traceback (most recent call last):</div><div>  File &quot;/usr/share/vdsm/storage/resourceManager.py&quot;, line 538, in registerResource</div><div>    obj = namespaceObj.factory.createResource(name, lockType)</div><div>  File &quot;/usr/share/vdsm/storage/resourceFactories.py&quot;, line 190, in createResource</div><div>    lockType)</div><div>  File &quot;/usr/share/vdsm/storage/resourceFactories.py&quot;, line 119, in __getResourceCandidatesList</div><div>    imgUUID=resourceName)</div></div><div><div>  File &quot;/usr/share/vdsm/storage/image.py&quot;, line 220, in getChain</div><div>    if srcVol.isLeaf():</div><div>  File &quot;/usr/share/vdsm/storage/volume.py&quot;, line 1261, in isLeaf</div><div>    return self._manifest.isLeaf()</div><div>  File &quot;/usr/share/vdsm/storage/volume.py&quot;, line 133, in isLeaf</div><div>    return self.getVolType() == sc.type2name(sc.LEAF_VOL)</div><div>  File &quot;/usr/share/vdsm/storage/volume.py&quot;, line 129, in getVolType</div><div>    self.voltype = self.getMetaParam(sc.VOLTYPE)</div><div>  File &quot;/usr/share/vdsm/storage/volume.py&quot;, line 113, in getMetaParam</div><div>    meta = self.getMetadata()</div><div>  File &quot;/usr/share/vdsm/storage/blockVolume.py&quot;, line 115, in getMetadata</div><div>    md = VolumeMetadata.from_lines(lines)</div><div>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/volumemetadata.py&quot;, line 103, in from_lines</div><div>    &quot;Missing metadata key: %s: found: %s&quot; % (e, md))</div><div>MetaDataKeyNotFoundError: Meta Data key not found error: (&quot;Missing metadata key: &#39;DOMAIN&#39;: found: {&#39;NONE&#39;: &#39;######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################&#39;}&quot;,)</div><div>2017-02-21 11:38:59,183 WARN  (tasks/9) [storage.ResourceManager.Request] (ResName=&#39;01_img_900b1853-e192-4661-a0f9-7c7c396f6f49.f0b5a0e4-ee5d-44a7-ba07-08285791368a&#39;, ReqID=&#39;571de7e4-6da2-4d40-911e-defb0eb735d8&#39;) Tried to cancel a processed request (resourceManager:183)</div><div>2017-02-21 11:38:59,184 ERROR (tasks/9) [storage.TaskManager.Task] (Task=&#39;08d7797a-af46-489f-ada0-c70bf4359366&#39;) Unexpected error (task:870)</div><div>Traceback (most recent call last):</div><div>  File &quot;/usr/share/vdsm/storage/task.py&quot;, line 877, in _run</div><div>    return fn(*args, **kargs)</div><div>  File &quot;/usr/share/vdsm/storage/task.py&quot;, line 333, in run</div></div><div><div>    return self.cmd(*self.argslist, **self.argsdict)</div><div>  File &quot;/usr/lib/python2.7/site-packages/vdsm/storage/securable.py&quot;, line 79, in wrapper</div><div>    return method(self, *args, **kwargs)</div><div>  File &quot;/usr/share/vdsm/storage/sp.py&quot;, line 1924, in createVolume</div><div>    with rm.acquireResource(img_ns, imgUUID, rm.EXCLUSIVE):</div><div>  File &quot;/usr/share/vdsm/storage/resourceManager.py&quot;, line 1025, in acquireResource</div><div>    return _manager.acquireResource(namespace, name, lockType, timeout=timeout)</div><div>  File &quot;/usr/share/vdsm/storage/resourceManager.py&quot;, line 473, in acquireResource</div><div>    raise se.ResourceAcqusitionFailed()</div><div>ResourceAcqusitionFailed: Could not acquire resource. Probably resource factory threw an exception.: ()</div></div><div><br></div><div><br></div></div></div></div>