[ovirt-users] Re: how to remove a failed backup operation

On Sat, Sep 4, 2021 at 1:08 AM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: ... >>> ovirt_imageio._internal.nbd.ReplyError: Writing to file failed: [Error 28] No space left on device >> This error is expected if you don't have space to write the data. > ok. I forgot to mention that running backup on engine host is not recommended. It is better to run the backup on the hypervisor, speeding up the data copy. You can mount the backup directory on the hypervisor (e.g. nfs) and use --backup-dir to store the backup where it should be. >>> Now if I try the same backup command (so with "full" option) and I get >>> >>> ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. >> This looks like a bug in the backup script - the backup should be finalized >> even if the image transfer failed, but the error you get say the vm is still >> in backup mode. >> >>> How can I clean the situation? >> >> 1. Stop the current backup >> >> If you still have the output from the command, we log the backup UUID. >> >> If you lost the backup id, you can get it using the API - visit this address in your browser: >> >> https://myengine/ovirt-engine/api/vms/{vm-id}/backups/ >> >> Then stop the current backup using: >> >> /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py stop vm-id backup-id >> >> If stopping the backup failed, stopping the VM will stop the backup. >> I hope you are running recent enough version, since in early versions there >> was a bug when you cannot stop the vm during a backup. > > It is the latest 4.4.7. I run the backup_vm.py script from the engine: > > ovirt-engine-4.4.7.7-1.el8.noarch > ovirt-engine-setup-plugin-imageio-4.4.7.7-1.el8.noarch > ovirt-imageio-common-2.2.0-1.el8.x86_64 > ovirt-imageio-client-2.2.0-1.el8.x86_64 > ovirt-imageio-daemon-2.2.0-1.el8.x86_64 > python3-ovirt-engine-sdk4-4.4.13-1.el8.x86_64 Looks good. > But if I try the stop command I get the error > > [g.cecchi@ovmgr1 ~]$ python3 /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py -c ovmgr1 stop dc386237-1e98-40c8-9d3d-45658163d1e2 68f83141-9d03-4cb0-84d4-e71fdd8753bb > [ 0.0 ] Finalizing backup '68f83141-9d03-4cb0-84d4-e71fdd8753bb' > Traceback (most recent call last): ... > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot stop VM backup. The VM backup is not in READY phase, backup phase is FINALIZING. Please try again when the backup is in READY phase.]". HTTP response code is 409. So your backup was already finalized, and it is stuck in "finalizing" phase. Usually this means the backup on libvirt side was already stopped, but engine failed to detect this and failed to complete the finalize step (ovirt-engine bug). You need to ensure if the backup was stopped on vdsm side. - If the vm was stopped, the bacukp is not running - If the vm is running, we can make sure the backup is stopped using vdsm-client VM stop_backup vmID=dc386237-1e98-40c8-9d3d-45658163d1e2 backup_id=68f83141-9d03-4cb0-84d4-e71fdd8753bb If this succeeds, the backup is not running on vdsm side. If this fails, you may need stop the VM to end the backup. If the backup was stopped, you may need to delete the scratch disks used in this backup. You can find the scratch disks ids in engine logs, and delete them from engine UI. Finally, after you cleaned up vdsm side, you can delete the backup from engine database, and unlock the disks. Pavel, can you provide instructions on how to clean up engine db after stuck backup?

After vdsm and engine were cleaned, new backup should work normally. >> 2. File a bug about this > Filed this one, hope its is correct; I chose ovirt-imageio as the product and Client as the component: In general backup bugs should be filed for ovirt-engine. ovirt-imageio is rarely the cause for a bug. We will move the bug to ovirt-imageio if needed. > https://bugzilla.redhat.com/show_bug.cgi?id=2001136 Thanks! Nir