Re: [ovirt-users] Re: Parent checkpoint ID does not match the actual leaf checkpoint
after some investigation we have foud that: via virsh our VM have 71 checkpoint on engine's db, in the table vm_checkpoints there are ZERO checkpoint. Is ther a way to sync the checkpoints?!? Regards, Tommaso Il 26/05/2021 10:27, Tommaso - Shellrent via Users ha scritto:
Hi to all. We have almost the same problem.
After a snapshot restore in a checkpoint taken previous last incremental backup, we have alwes the error "Parent checkpoint ID does not match the actual leaf checkpoint". we have also the " parent_checkpoint_id: None"
there is a wey to fix it on a production env?
our engine version is 4.4.5.11 and vdsm 4.40.50
Il 19/07/2020 23:27, Nir Soffer ha scritto:
On Sun, Jul 19, 2020 at 5:38 PM Łukasz Kołaciński <l.kolacinski@storware.eu <mailto:l.kolacinski@storware.eu>> wrote:
Hello, Thanks to previous answers, I was able to make backups. Unfortunately, we had some infrastructure issues and after the host reboots new problems appeared. I am not able to do any backup using the commands that worked yesterday. I looked through the logs and there is something like this:
2020-07-17 15:06:30,644+02 ERROR [org.ovirt.engine.core.bll.StartVmBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-54) [944a1447-4ea5-4a1c-b971-0bc612b6e45e] Failed to execute VM backup operation 'StartVmBackup': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartVmBackupVDS, error = Checkpoint Error: {'parent_checkpoint_id': None, 'leaf_checkpoint_id': 'cd078706-84c0-4370-a6ec-654ccd6a21aa', 'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'reason': '*Parent checkpoint ID does not match the actual leaf checkpoint*'}, code = 1610 (Failed with error unexpected and code 16)
It looks like engine sent:
parent_checkpoint_id: None
This issue was fix in engine few weeks ago.
Which engine and vdsm versions are you testing?
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2114) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performVmBackupOperation(StartVmBackupCommand.java:368) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.runVmBackup(StartVmBackupCommand.java:225) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performNextOperation(StartVmBackupCommand.java:199) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
And the last error is:
2020-07-17 15:13:45,835+02 ERROR [org.ovirt.engine.core.bll.StartVmBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) [f553c1f2-1c99-4118-9365-ba6b862da936] Failed to execute VM backup operation 'GetVmBackupInfo': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVmBackupInfoVDS, error = No such backup Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup_id': 'bf1c26f7-c3e5-437c-bb5a-255b8c1b3b73', 'reason': '*VM backup not exists: Domain backup job id not found: no domain backup job present'*}, code = 1601 (Failed with error unexpected and code 16)
This is likely a result of the first error. If starting backup failed the backup entity is deleted.
(these errors are from full backup)
Like I said this is very strange because everything was working correctly.
Regards
Łukasz Kołaciński
Junior Java Developer
e-mail: l.kolacinski@storware.eu <mailto:l.kolacinski@storware.eu> <mailto:m.helbert@storware.eu>
*STORWARE* <http://www.storware.eu/>**
*ul. Leszno 8/44 01-192 Warszawa www.storware.eu <https://www.storware.eu/>*
*facebook* <https://www.facebook.com/storware>**
*twitter* <https://twitter.com/storware>**
*linkedin* <https://www.linkedin.com/company/storware>**
*Storware_Stopka_09* <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw>**
*//*
*/Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa 000510131/*//*/, NIP 5213672602./*/ Wiadomość ta jest przeznaczona jedynie dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie tej wiadomości z wszelkich komputerów. //This message is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this message in error, please contact the sender and remove the material from all of your computer systems./
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3PLYPOZGT6TLJ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3PLYPOZGT6TLJFI2XAMLNBKLXOEHVQN/>
_______________________________________________ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-leave@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/7XWQUEIS7X5BQO... -- -- Shellrent - Il primo hosting italiano Security First
*Tommaso De Marchi* /COO - Chief Operating Officer/ Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 <tel:+390444321155> | Fax 04441492177
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DD2R2PWCIQD5HN...
-- -- Shellrent - Il primo hosting italiano Security First *Tommaso De Marchi* /COO - Chief Operating Officer/ Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 <tel:+390444321155> | Fax 04441492177
On Wed, May 26, 2021 at 12:48 PM Tommaso - Shellrent <tommaso@shellrent.com> wrote:
after some investigation we have foud that:
via virsh our VM have 71 checkpoint
on engine's db, in the table vm_checkpoints there are ZERO checkpoint.
Is ther a way to sync the checkpoints?!?
Engine deletes all checkpoints from engine if it cannot redefine the checkpoints in libvirt. After this you must start again with full backup. The stale checkpoint in libvirt should not happen, please file a bug for this. We need to understand why engine deleted the checkpoints without deleting the checkpoints in libvirt. To checkpoints in libvirt are deleted when you stop the vm. You can also delete them safely using virsh. Nir
On Wed, 26 May 2021 at 12:51, Tommaso - Shellrent via Users <users@ovirt.org> wrote:
after some investigation we have foud that:
via virsh our VM have 71 checkpoint
on engine's db, in the table vm_checkpoints there are ZERO checkpoint.
Is ther a way to sync the checkpoints?!?
Regards, Tommaso Il 26/05/2021 10:27, Tommaso - Shellrent via Users ha scritto:
Hi to all. We have almost the same problem.
After a snapshot restore in a checkpoint taken previous last incremental backup, we have alwes the error "Parent checkpoint ID does not match the actual leaf checkpoint". we have also the " parent_checkpoint_id: None"
If you restored a snapshot then all the checkpoints that were taken after that snapshot are now gone - https://bugzilla.redhat.com/show_bug.cgi?id=1952577. A full backup is needed in this case. In order to get over "Parent checkpoint ID does not match the actual leaf checkpoint" you should upgrade to a newer version, fixed in 4.4.6.
there is a wey to fix it on a production env?
our engine version is 4.4.5.11 and vdsm 4.40.50
Il 19/07/2020 23:27, Nir Soffer ha scritto:
On Sun, Jul 19, 2020 at 5:38 PM Łukasz Kołaciński < l.kolacinski@storware.eu> wrote:
Hello, Thanks to previous answers, I was able to make backups. Unfortunately, we had some infrastructure issues and after the host reboots new problems appeared. I am not able to do any backup using the commands that worked yesterday. I looked through the logs and there is something like this:
2020-07-17 15:06:30,644+02 ERROR [org.ovirt.engine.core.bll.StartVmBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-54) [944a1447-4ea5-4a1c-b971-0bc612b6e45e] Failed to execute VM backup operation 'StartVmBackup': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartVmBackupVDS, error = Checkpoint Error: {'parent_checkpoint_id': None, 'leaf_checkpoint_id': 'cd078706-84c0-4370-a6ec-654ccd6a21aa', 'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'reason': '*Parent checkpoint ID does not match the actual leaf checkpoint*'}, code = 1610 (Failed with error unexpected and code 16)
It looks like engine sent:
parent_checkpoint_id: None
This issue was fix in engine few weeks ago.
Which engine and vdsm versions are you testing?
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2114) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performVmBackupOperation(StartVmBackupCommand.java:368) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.runVmBackup(StartVmBackupCommand.java:225) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performNextOperation(StartVmBackupCommand.java:199) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
And the last error is:
2020-07-17 15:13:45,835+02 ERROR [org.ovirt.engine.core.bll.StartVmBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) [f553c1f2-1c99-4118-9365-ba6b862da936] Failed to execute VM backup operation 'GetVmBackupInfo': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVmBackupInfoVDS, error = No such backup Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup_id': 'bf1c26f7-c3e5-437c-bb5a-255b8c1b3b73', 'reason': '*VM backup not exists: Domain backup job id not found: no domain backup job present'*}, code = 1601 (Failed with error unexpected and code 16)
This is likely a result of the first error. If starting backup failed the backup entity is deleted.
(these errors are from full backup)
Like I said this is very strange because everything was working correctly.
Regards
Łukasz Kołaciński
Junior Java Developer
e-mail: l.kolacinski@storware.eu <m.helbert@storware.eu>
*[image: STORWARE]* <http://www.storware.eu/>
*ul. Leszno 8/44 01-192 Warszawa www.storware.eu <https://www.storware.eu/>*
*[image: facebook]* <https://www.facebook.com/storware>
*[image: twitter]* <https://twitter.com/storware>
*[image: linkedin]* <https://www.linkedin.com/company/storware>
*[image: Storware_Stopka_09]* <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw>
*Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie tej wiadomości z wszelkich komputerów. **This message is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this message in error, please contact the sender and remove the material from all of your computer systems.*
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3PLYPOZGT6TLJ...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7XWQUEIS7X5BQO...
-- -- [image: Shellrent - Il primo hosting italiano Security First] *Tommaso De Marchi* *COO - Chief Operating Officer* Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 <+390444321155> | Fax 04441492177
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DD2R2PWCIQD5HN...
-- -- [image: Shellrent - Il primo hosting italiano Security First] *Tommaso De Marchi* *COO - Chief Operating Officer* Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 <+390444321155> | Fax 04441492177 _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4PRP6322NHKZWM...
-- Regards, Eyal Shenitzky
participants (3)
-
Eyal Shenitzky -
Nir Soffer -
Tommaso - Shellrent