Cannot remove Snapshot. Related operation is currently in progress. Please try again later.

13 Apr 2022

      Hello,

my storage got disconnected during the image migration, resulting is stuck migration. Task had been cleaned with restart of vdsdm service, but lead to stuck disk image. The VM got powered down in the meantime, and currently I cannot do anything with it, I always get a message like "Cannot.... Related operation is currently in progress."  I already used the unlock_entity.sh script, it cleaned something, but did not help me. I cannot delete the VM, delete the disk, delete the snapshots. I even deleted the wrong snapshot from disk. All I need is it get rid of the VM and its disk, as I can restore it from the backup. Could you please somebody try to navigate me how to solve the problem? Engine log keeps filling up with messages which indicate there is an orphaned task, however I am not sure what to delete from database:

2022-04-13 22:26:41,822+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command CopyData id: '62b1976b-729a-41ae-972c-a42de750ad1d': couldn't get the status of job '6ef3cb28-14cf-486a-ad71-96484c1a596e' on host 'kvm2.finamis.com' (id: 'f880cfbe-5827-4853-a4fc-11199c439f79'), assuming it's still running
2022-04-13 22:26:41,846+02 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command 'LiveMigrateDisk' (id: '3cd698fd-08a2-419e-bdd8-6b3516bb3127') waiting on child command id: '2e9ad954-8f4e-41ed-9c75-015afeed337d' type:'CopyImageGroupVolumesData' to complete
2022-04-13 22:26:41,847+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command FenceVolumeJob id: '831828a3-2e0f-4989-965b-91761fa9601f': job '9dc88836-db04-49ae-b01e-18a56e24c91a' wasn't executed on any host, considering the job status as failed
2022-04-13 22:26:41,847+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command FenceVolumeJob id: '831828a3-2e0f-4989-965b-91761fa9601f': execution was completed, the command status is 'FAILED'
2022-04-13 22:26:42,850+02 ERROR [org.ovirt.engine.core.bll.job.JobRepositoryImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-6) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Failed to save step '499b0207-61ba-4998-8cb8-2cb93b89422b', 'FINALIZING': CallableStatementCallback; SQL [{call insertstep(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ed10f993-cfd2-4a8b-806f-3fbd823101ef) is not present in table "job".
  Where: SQL statement "INSERT INTO step (
        step_id,
        parent_step_id,
        job_id,
        step_type,
        description,
        step_number,
        status,
        progress,
        start_time,
        end_time,
        correlation_id,
        external_id,
        external_system_type,
        is_external
        )
    VALUES (
        v_step_id,
        v_parent_step_id,
        v_job_id,
        v_step_type,
        v_description,
        v_step_number,
        v_status,
        v_progress,
        v_start_time,
        v_end_time,
        v_correlation_id,
        v_external_id,
        v_external_system_type,
        v_is_external
        )"
PL/pgSQL function insertstep(uuid,uuid,uuid,character varying,text,integer,character varying,smallint,timestamp with time zone,timestamp with time zone,character varying,uuid,character varying,boolean) line 3 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ed10f993-cfd2-4a8b-806f-3fbd823101ef) is not present in table "job".
  Where: SQL statement "INSERT INTO step (
        step_id,
        parent_step_id,
        job_id,
        step_type,
        description,
        step_number,
        status,
        progress,
        start_time,
        end_time,
        correlation_id,
        external_id,
        external_system_type,
        is_external
        )
    VALUES (
        v_step_id,
        v_parent_step_id,
        v_job_id,
        v_step_type,
        v_description,
        v_step_number,
        v_status,
        v_progress,
        v_start_time,
        v_end_time,
        v_correlation_id,
        v_external_id,
        v_external_system_type,
        v_is_external
        )"
PL/pgSQL function insertstep(uuid,uuid,uuid,character varying,text,integer,character varying,smallint,timestamp with time zone,timestamp with time zone,character varying,uuid,character varying,boolean) line 3 at SQL statement
2022-04-13 22:26:42,851+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-6) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand' with failure.
2022-04-13 22:26:51,855+02 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command 'CopyImageGroupVolumesData' (id: '2e9ad954-8f4e-41ed-9c75-015afeed337d') waiting on child command id: '62b1976b-729a-41ae-972c-a42de750ad1d' type:'CopyData' to complete
2022-04-13 22:26:51,858+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] START, GetHostJobsVDSCommand(HostName = kvm2.finamis.com, GetHostJobsVDSCommandParameters:{hostId='f880cfbe-5827-4853-a4fc-11199c439f79', type='storage', jobIds='[6ef3cb28-14cf-486a-ad71-96484c1a596e]'}), log id: 480d931c
2022-04-13 22:26:51,862+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] FINISH, GetHostJobsVDSCommand, return: {}, log id: 480d931c
2022-04-13 22:26:51,862+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command CopyData id: '62b1976b-729a-41ae-972c-a42de750ad1d': attempting to determine the job status by polling the entity.
2022-04-13 22:26:51,862+02 ERROR [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command CopyData id: '62b1976b-729a-41ae-972c-a42de750ad1d': failed to poll the command entity
2022-04-13 22:26:51,862+02 INFO  [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command CopyData id: '62b1976b-729a-41ae-972c-a42de750ad1d': attempting to fence job ed10f993-cfd2-4a8b-806f-3fbd823101ef
2022-04-13 22:26:51,868+02 INFO  [org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Running command: FenceVolumeJobCommand internal: true.
2022-04-13 22:26:51,868+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command 'org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand' failed: null
2022-04-13 22:26:51,868+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Exception: java.lang.NullPointerException
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand.executeCommand(FenceVolumeJobCommand.java:31)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2008)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:638)
        at jdk.internal.reflect.GeneratedMethodAccessor273.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.as.weld.common@23.0.2.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79)
        at org.jboss.as.weld.common@23.0.2.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)
        at org.jboss.as.weld.common@23.0.2.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
        at org.jboss.as.weld.common@23.0.2.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
        at org.wildfly.security.elytron-private@1.15.3.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source)
        at jdk.internal.reflect.GeneratedMethodAccessor272.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2386)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2411)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2406)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand.attemptToFenceJob(CopyDataCommand.java:144)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.HostJobCallback.handleUndeterminedJobStatus(HostJobCallback.java:196)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.HostJobCallback.childCommandsExecutionEnded(HostJobCallback.java:84)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:181)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360)
        at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227)

2022-04-13 22:26:51,872+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command CopyData id: '62b1976b-729a-41ae-972c-a42de750ad1d': couldn't get the status of job '6ef3cb28-14cf-486a-ad71-96484c1a596e' on host 'kvm2.finamis.com' (id: 'f880cfbe-5827-4853-a4fc-11199c439f79'), assuming it's still running
2022-04-13 22:26:51,899+02 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command 'LiveMigrateDisk' (id: '3cd698fd-08a2-419e-bdd8-6b3516bb3127') waiting on child command id: '2e9ad954-8f4e-41ed-9c75-015afeed337d' type:'CopyImageGroupVolumesData' to complete
2022-04-13 22:26:51,900+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command FenceVolumeJob id: '3c8b0471-c405-4b7f-ac54-43732407d3cb': job '3c38e07a-44f3-46b7-97f9-56b6218e69b4' wasn't executed on any host, considering the job status as failed
2022-04-13 22:26:51,900+02 INFO  [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-76) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Command FenceVolumeJob id: '3c8b0471-c405-4b7f-ac54-43732407d3cb': execution was completed, the command status is 'FAILED'
2022-04-13 22:26:52,903+02 ERROR [org.ovirt.engine.core.bll.job.JobRepositoryImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Failed to save step 'ad184ef6-e708-41f3-acff-962d39f01c30', 'FINALIZING': CallableStatementCallback; SQL [{call insertstep(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ed10f993-cfd2-4a8b-806f-3fbd823101ef) is not present in table "job".
  Where: SQL statement "INSERT INTO step (
        step_id,
        parent_step_id,
        job_id,
        step_type,
        description,
        step_number,
        status,
        progress,
        start_time,
        end_time,
        correlation_id,
        external_id,
        external_system_type,
        is_external
        )
    VALUES (
        v_step_id,
        v_parent_step_id,
        v_job_id,
        v_step_type,
        v_description,
        v_step_number,
        v_status,
        v_progress,
        v_start_time,
        v_end_time,
        v_correlation_id,
        v_external_id,
        v_external_system_type,
        v_is_external
        )"
PL/pgSQL function insertstep(uuid,uuid,uuid,character varying,text,integer,character varying,smallint,timestamp with time zone,timestamp with time zone,character varying,uuid,character varying,boolean) line 3 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ed10f993-cfd2-4a8b-806f-3fbd823101ef) is not present in table "job".
  Where: SQL statement "INSERT INTO step (
        step_id,
        parent_step_id,
        job_id,
        step_type,
        description,
        step_number,
        status,
        progress,
        start_time,
        end_time,
        correlation_id,
        external_id,
        external_system_type,
        is_external
        )
    VALUES (
        v_step_id,
        v_parent_step_id,
        v_job_id,
        v_step_type,
        v_description,
        v_step_number,
        v_status,
        v_progress,
        v_start_time,
        v_end_time,
        v_correlation_id,
        v_external_id,
        v_external_system_type,
        v_is_external
        )"
PL/pgSQL function insertstep(uuid,uuid,uuid,character varying,text,integer,character varying,smallint,timestamp with time zone,timestamp with time zone,character varying,uuid,character varying,boolean) line 3 at SQL statement
2022-04-13 22:26:52,904+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) [e546bb30-7d18-4348-9c28-9dbbf4ab1893] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.FenceVolumeJobCommand' with failure.
^C

Martin Marusinec

tags

participants (1)