Re: [Users] VMs and volumes disappearing

From: "Martijn Grendelman" <martijn.grendelman(a)isaac.nl&gt; To: users(a)ovirt.org Sent: Tuesday, October 1, 2013 12:29:58 PM Subject: Re: [Users] VMs and volumes disappearing Hi, >> I have recently set up an oVirt environment, I think in a pretty >> standard fashion, with engine 3.3 on one host, one oVirt host on a >> physical machine, both running CentOS 6.4, using NFS for all storage >> domains. > > Please provide rpm -qa on the ovirt rpms (ovirt engine). martijn@ovirt:~> rpm -qa | grep ovirt ovirt-log-collector-3.3.0-1.el6.noarch ovirt-engine-3.3.0-1.el6.noarch ovirt-host-deploy-1.1.1-1.el6.noarch ovirt-engine-cli-3.3.0.4-1.el6.noarch ovirt-engine-userportal-3.3.0-1.el6.noarch ovirt-engine-tools-3.3.0-1.el6.noarch ovirt-engine-setup-3.3.0-4.el6.noarch ovirt-engine-sdk-python-3.3.0.6-1.el6.noarch ovirt-image-uploader-3.3.0-1.el6.noarch ovirt-engine-restapi-3.3.0-1.el6.noarch ovirt-engine-webadmin-portal-3.3.0-1.el6.noarch ovirt-host-deploy-java-1.1.1-1.el6.noarch ovirt-engine-backend-3.3.0-1.el6.noarch ovirt-release-el6-8-1.noarch ovirt-iso-uploader-3.3.0-1.el6.noarch ovirt-engine-dbscripts-3.3.0-1.el6.noarch ovirt-engine-lib-3.3.0-4.el6.noarch >> Today I was playing around with snapshots, when I noticed that the >> Snapshots panel didn't show any of the snapshots I created, not even the >> 'Current - Active VM' snapshot that all VMs have. > > Not sure why this has happened. How do you know that snapshot > creation was completed? Did you look at the events tab? (Asking to be > sure) engine.log will be quite helpful here. I find engine.log somewhat hard to read, to be honest, and documentation is hard to find, but I think I found some clues.

I tried to create 4 snapshots of a certain VM, 2 of which completed normally and 2 of which failed: "Failed with VDSM error SNAPSHOT_FAILED and code 48" However, what I find most upsetting, is that the VMs that disappeared were not the subject of my experiments. I was creating snapshots of a single VM, and the VMs that disappeared were unrelated. As a matter of fact, the VM I was experimenting with IS THE ONLY ONE that survived. By the way, the Snapshots panel has been displaying snapshots correctly for a while, but when I logged in this morning, it appeared empty again, for all VMs. Is there anything I can check to see what causes this? >> Not sure what to do, I decided to restart the ovirt-engine process. >> >> When I logged back on to the administrator panel, I was shocked to >> see 2endWith of my 4 VMs completely missing from the inventory. I >> haven't been able to find back a single trace of either machine, >> neither in the portal nor on disk. It seems like they never >> existed. The storage of both VMs seems to be erased from the data >> domain. > Not sure why storage domain was erased. About Vms disappeared - there > were previous discussions on that at users(a)ovirt.org. In a nutshell, > due to a bug (that was already fixed) prior to the restart you might > have had records at the table that contained value of > "empty guid" (a string in UUID format with only 0 and - ) at the > vdsm_task_id_column. This means that the task is not associated with > a real SPM task, and when the engine restarts, if for a given flow > (let's say - snapshot creation) there are tasks with such > vdsm_task_id, the flow will end with failure. For some flows , > ending with failure means erasing the vm (for example - real failure > of importing a vm). By the way, similar issue can probably occur with > disks as well, as there are flows that run async tasks that deal with > disks. I think I have an idea about what happended now. The 2 disappeared VMs have been imported into oVirt using virt-v2v. The 3rd one that's now missing a disk volume was not, but I have been playing with storage migration in the past.

Yesterday's engine.log seems to suggest, that all of these tasks (importing the 2 VMs and trying to move a volume) have been restarted immediately after restarting Engine. After failure, the VMs and volume were removed. It seems to fit the above description of the bug. So... What can I do to prevent this from happening again? Should I periodically check the 'async_tasks' table for anomalies? Is there a bugfix I can apply, or should I wait for a new release of oVirt? If the latter, when is that expected to happen?

Thanks, Martijn. _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users