Failed to delete snapshot '<UNKNOWN>' for VM 'vmname'.
Hi, We're running oVirt 4.1.9, recently we had an issue with snapshots so we had to fix them manually. The issue is mostly solved but now we're seeing a lot of events like this one: 2018-06-25 07:58:06,637+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [91430fc5-284d-4c26-8d8d-b7bf4053a7e4] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Correlation ID: 91430fc5-284d-4c26-8d8d-b7bf4053a7e4, Job ID: da8120a4-9c6d-4379-ad67-a3808db1fd46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to delete snapshot '<UNKNOWN>' for VM 'vmname'. Which generates the following event in the manager: Failed to delete snapshot '<UNKNOWN>' for VM 'vmname'. This event is being generated every 10 seconds, so it's kind of annoying. Any way to remove it manually? It doesn't matter if it entails touching the DB directly. Thanks.
engine=# SELECT status FROM command_entities; status -------------------- FAILED ENDED_SUCCESSFULLY FAILED FAILED UNKNOWN UNKNOWN ENDED_WITH_FAILURE ENDED_WITH_FAILURE ENDED_WITH_FAILURE ENDED_WITH_FAILURE ACTIVE ACTIVE SUCCEEDED SUCCEEDED ACTIVE ACTIVE ACTIVE SUCCEEDED SUCCEEDED SUCCEEDED ACTIVE SUCCEEDED (22 rows) Is it safe to just remove the events in this table? Thanks El 2018-06-25 08:43, Ala Hino escribió:
So if I have this entry, for example, what should I do next? cc7764c5-dbc3-4886-bdd6-dabcc756cf6a | 235 | cc7764c5-dbc3-4886-bdd6-dabcc756cf6a | { +| org.ovir t.engine.core.common.action.RemoveDiskSnapshotsParameters | 2018-06-15 09:39:16.846795+01 | ACTIVE | t | f | { +| org.ovirt.engine.core.common.action.VdcReturnValueBase | t | 5e0f9455-e9b5-4445-adb0-0058fc604bef | 00000000-0000-0000-0000-000000000000 | { } | 699165 | { + | | | "@class" : "org.ovirt.engine.core.common.action.RemoveDiskSnapshotsParameters", +| | | | | | "valid" : true, +| | | | | | | "jobId" : [ "org.ovirt.engine.core.compat.Guid", {+ | | | "commandId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | "validationMessages" : [ "VAR__ACTION__REMOVE", "VAR__TYPE__DISK__SNAP SHOT", "VAR__ACTION__REMOVE", "VAR__TYPE__DISK__SNAPSHOT" ],+| | | | | | | "uuid" : "2f1ee2a3-fa2f-48b1-924b-9bfa6497f3b3" + | | | "uuid" : "cc7764c5-dbc3-4886-bdd6-dabcc756cf6a" +| | | | | | "succeeded" : true, +| | | | | | | } ], + | | | } ], +| | | | | | "isSyncronious" : true, +| | | | | | | "stepId" : null, + | | | "parametersCurrentUser" : { +| | | | | | "description" : "", +| | | | | | | "executionMethod" : "AsJob", + | | | "@class" : "org.ovirt.engine.core.common.businessentities.aaa.DbUser", +| | | | | | "taskPlaceHolderIdList" : [ ], +| | | | | | | "monitored" : true, + | | | "id" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | "vdsmTaskIdList" : [ ], +| | | | | | | "tasksMonitored" : true, + | | | "uuid" : "5e0f9455-e9b5-4445-adb0-0058fc604bef" +| | | | | | "internalVdsmTaskIdList" : [ ], +| | | | | | | "completed" : false, + | | | } ], +| | | | | | "endActionTryAgain" : true, +| | | | | | | "jobRequired" : false + | | | "externalId" : "fdfc627c-d875-11e0-90f0-83df133b58cc", +| | | | | | "executeFailedMessages" : [ ], +| | | | | | | } | | | "domain" : "internal", +| | | | | | "fault" : { +| | | | | | | | | | "namespace" : "*", +| | | | | | "@class" : "org.ovirt.engine.core.common.errors.EngineFault", +| | | | | | | | | | "loginName" : "admin", +| | | | | | "sessionID" : null, +| | | | | | | | | | "firstName" : "admin", +| | | | | | "details" : null, +| | | | | | | | | | "lastName" : "", +| | | | | | "errorCode" : 5001, +| | | | | | | | | | "department" : "", +| | | | | | "message" : "ENGINE" +| | | | | | | | | | "email" : "", +| | | | | | }, +| | | | | | | | | | "note" : "", +| | | | | | "correlationId" : null, +| | | | | | | | | | "groupNames" : [ "java.util.ArrayList", [ ] ], +| | | | | | "jobId" : null, +| | | | | | | | | | "groupIds" : [ "java.util.ArrayList", [ ] ], +| | | | | | "actionReturnValue" : null, +| | | | | | | | | | "admin" : true, +| | | | | | "hasAsyncTasks" : false +| | | | | | | | | | "group" : false +| | | | | | } | | | | | | | | | | }, +| | | | | | | | | | | | | | | | "compensationEnabled" : false, +| | | | | | | | | | | | | | | | "parentCommand" : "Unknown", +| | | | | | | | | | | | | | | | "commandType" : "Unknown", +| | | | | | | | | | | | | | | | "multipleAction" : true, +| | | | | | | | | | | | | | | | "entityInfo" : null, +| | | | | | | | | | | | | | | | "taskGroupSuccess" : true, +| | | | | | | | | | | | | | | | "vdsmTaskIds" : null, +| | | | | | | | | | | | | | | | "vdsRunningOn" : null, +| | | | | | | | | | | | | | | | "endProcedure" : "PARENT_MANAGED", +| | | | | | | | | | | | | | | | "useCinderCommandCallback" : false, +| | | | | | | | | | | | | | | | "correlationId" : "1db468cb-85fd-4189-b356-d31781461504", +| | | | | | | | | | | | | | | | "jobId" : null, +| | | | | | | | | | | | | | | | "stepId" : null, +| | | | | | | | | | | | | | | | "jobWeight" : null, +| | | | | | | | | | | | | | | | "operationsJobWeight" : [ "java.util.Collections$EmptyMap", { } ], +| | | | | | | | | | | | | | | | "lockProperties" : { +| | | | | | | | | | | | | | | | "scope" : "Execution", +| | | | | | | | | | | | | | | | "wait" : false +| | | | | | | | | | | | | | | | }, +| | | | | | | | | | | | | | | | "lifeInMinutes" : null, +| | | | | | | | | | | | | | | | "prevVdsStatus" : "Unassigned", +| | | | | | | | | | | | | | | | "vdsId" : null, +| | | | | | | | | | | | | | | | "storagePoolId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "00000000-0000-0000-0000-000000000000" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "forceDelete" : false, +| | | | | | | | | | | | | | | | "storageDomainId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "12d099c6-3c4b-4c7a-8597-e0c3d45bb929" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "isInternal" : false, +| | | | | | | | | | | | | | | | "quotaId" : null, +| | | | | | | | | | | | | | | | "diskProfileId" : null, +| | | | | | | | | | | | | | | | "imageId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "00000000-0000-0000-0000-000000000000" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "destinationImageId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "00000000-0000-0000-0000-000000000000" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "diskAlias" : "ubuntu-1604-xilinx_Disk1", +| | | | | | | | | | | | | | | | "description" : null, +| | | | | | | | | | | | | | | | "oldLastModifiedValue" : null, +| | | | | | | | | | | | | | | | "vmSnapshotId" : null, +| | | | | | | | | | | | | | | | "imageGroupID" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "d8129d2b-5405-4bd2-9a6c-beca237bad76" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "importEntity" : false, +| | | | | | | | | | | | | | | | "leaveLocked" : false, +| | | | | | | | | | | | | | | | "wipeAfterDelete" : false, +| | | | | | | | | | | | | | | | "imageIds" : [ [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "dfbaca3a-aca1-4bda-9887-463c1f716b10" +| | | | | | | | | | | | | | | | } ] ], +| | | | | | | | | | | | | | | | "imageIdsSorted" : true, +| | | | | | | | | | | | | | | | "childImageIds" : null, +| | | | | | | | | | | | | | | | "snapshotNames" : [ "java.util.LinkedList", [ "Auto-Snapshot para clonado" ] ], +| | | | | | | | | | | | | | | | "liveMerge" : false, +| | | | | | | | | | | | | | | | "containerId" : [ "org.ovirt.engine.core.compat.Guid", { +| | | | | | | | | | | | | | | | "uuid" : "4cff718d-41ad-4934-aceb-0050c3b29487" +| | | | | | | | | | | | | | | | } ], +| | | | | | | | | | | | | | | | "sessionId" : "Yl+fI6EgRGm1N2WRLK/Kd+nbf0UhVlwsSozQnMCnPIFhayyaTATOlhSiDwmkCUHCunNIHcIvi3pM+9wFUe7m5Q==",+| | | | | | | | | | | | | | | | "shouldBeLogged" : true, +| | | | | | | | | | | | | | | | "executionReason" : "REGULAR_FLOW", +| | | | | | | | | | | | | | | | "transactionScopeOption" : "Required" +| | | | | | | | | | | | | | | | } | | | | | | | | | | | | | El 2018-06-25 08:49, Ala Hino escribió:
Do you have any running merge related commands now? Is it possible for you to restart the engine? I want to see if restarting the engine, while the merge commands already manually repaired, may cause that log message to stop appearing. On Mon, Jun 25, 2018 at 11:06 AM, <nicolas@devels.es> wrote:
Currently there are no merge related commands running, if you mean that we're aware of some merge should be happening (it shouldn't). I restarted the engine but the event is still showing up every 10 seconds. El 2018-06-25 09:39, Ala Hino escribió:
I understand. Back to command entities. For each active command_parameter_class org.ovirt.engine.core.common.action.RemoveSnapshotParameters, you will find the command params in command_parameters column. One of these params, is the snapshotId. You need to check whether there is a snapshot with that Id in snapshots table. On Mon, Jun 25, 2018 at 11:48 AM, <nicolas@devels.es> wrote:
Hi Ala, All vmSnapshotId fields are null. Could it be that this could be solved by running what there's on comment 19 in [1]? [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1450000#c19 El 2018-06-25 10:20, Ala Hino escribió:
I am looking for something similar to the following (in the value of command_parameters): "snapshotId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "d01a50fe-503d-4081-90d3-a71e464c5c6c" } Do you see this correlationId 91430fc5-284d-4c26-8d8d-b7bf4053a7e4 in any command_parameters? On Mon, Jun 25, 2018 at 12:28 PM, <nicolas@devels.es> wrote:
There's no such field in the output. There's a similar one called commandId (maybe because this is 4.1.9?). An output has this format: { + "@class" : "org.ovirt.engine.core.common.action.RemoveDiskSnapshotsParameters", + "commandId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "a13f9eb4-1b0b-4ea3-924c-bd519af7853b" + } ], + "parametersCurrentUser" : { + "@class" : "org.ovirt.engine.core.common.businessentities.aaa.DbUser", + "id" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "5e0f9455-e9b5-4445-adb0-0058fc604bef" + } ], + "externalId" : "fdfc627c-d875-11e0-90f0-83df133b58cc", + "domain" : "internal", + "namespace" : "*", + "loginName" : "admin", + "firstName" : "admin", + "lastName" : "", + "department" : "", + "email" : "", + "note" : "", + "groupNames" : [ "java.util.ArrayList", [ ] ], + "groupIds" : [ "java.util.ArrayList", [ ] ], + "admin" : true, + "group" : false + }, + "compensationEnabled" : false, + "parentCommand" : "Unknown", + "commandType" : "Unknown", + "multipleAction" : true, + "entityInfo" : null, + "taskGroupSuccess" : true, + "vdsmTaskIds" : null, + "vdsRunningOn" : null, + "endProcedure" : "PARENT_MANAGED", + "useCinderCommandCallback" : false, + "correlationId" : "adf5f92f-7fa8-41c9-af2f-4c116f7b6d55", + "jobId" : null, + "stepId" : null, + "jobWeight" : null, + "operationsJobWeight" : [ "java.util.Collections$EmptyMap", { } ], + "lockProperties" : { + "scope" : "Execution", + "wait" : false + }, + "lifeInMinutes" : null, + "prevVdsStatus" : "Unassigned", + "vdsId" : null, + "storagePoolId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "00000000-0000-0000-0000-000000000000" + } ], + "forceDelete" : false, + "storageDomainId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "fc2a49fc-fc53-4040-9d61-9dc1e47f51f4" + } ], + "isInternal" : false, + "quotaId" : null, + "diskProfileId" : null, + "imageId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "00000000-0000-0000-0000-000000000000" + } ], + "destinationImageId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "00000000-0000-0000-0000-000000000000" + } ], + "diskAlias" : "ubuntu-1604-xilinx_Disk1", + "description" : null, + "oldLastModifiedValue" : null, + "vmSnapshotId" : null, + "imageGroupID" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "5e7df75c-57f1-4cee-a178-94a9946482a6" + } ], + "importEntity" : false, + "leaveLocked" : false, + "wipeAfterDelete" : false, + "imageIds" : [ [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "10f8adad-7b21-4d39-9583-00bbcc42266a" + } ] ], + "imageIdsSorted" : true, + "childImageIds" : null, + "snapshotNames" : [ "java.util.LinkedList", [ "Auto-Snapshot para clonado" ] ], + "liveMerge" : false, + "containerId" : [ "org.ovirt.engine.core.compat.Guid", { + "uuid" : "b0f24a9f-b1ae-4357-b06a-06dbb8f440e6" + } ], + "sessionId" : "Yl+fI6EgRGm1N2WRLK/Kd+nbf0UhVlwsSozQnMCnPIFhayyaTATOlhSiDwmkCUHCunNIHcIvi3pM+9wFUe7m5Q==",+ "shouldBeLogged" : true, + "executionReason" : "REGULAR_FLOW", + "transactionScopeOption" : "Required" + } El 2018-06-25 10:33, Ala Hino escribió:
There is a correlationId field, I marked below. Can you please the following statement and send the output? select command_parameters from command_entities where command_params_class = 'org.ovirt.engine.core.common.action.RemoveSnapshotParameters' and status = 'ACTIVE' On Mon, Jun 25, 2018 at 1:21 PM, <nicolas@devels.es> wrote:
Yes, it returned 0 rows: engine=# select command_parameters from command_entities where command_params_class = 'org.ovirt.engine.core.common.action.RemoveSnapshotParameters' and status = 'ACTIVE'; command_parameters -------------------- (0 rows) El 2018-06-25 11:55, Ala Hino escribió:
I tried cloning the VM and deleting the original so maybe it would also delete the related tasks... but it didn't, instead of: Failed to delete snapshot '<UNKNOWN>' for VM 'vmname'. It now shows: Failed to delete snapshot '<UNKNOWN>' for VM '<UNKNOWN>'. Any tip on how to solve this, please? Thanks El 2018-06-25 13:28, nicolas@devels.es escribió:
participants (2)
-
Ala Hino -
nicolas＠devels.es