Re: [ovirt-users] Can't remove snapshot

On 03/18/2016 03:10 PM, Nir Soffer wrote: > On Fri, Mar 18, 2016 at 7:55 PM, Nathanaël Blanchet <blanchet(a)abes.fr&gt; > wrote: > >> Hello, >> >> I can create snapshot when no one exists but I'm not able to remove it >> after. >> > > Do you try to remove it when the vm is running? > > It concerns many of my vms, and when stopping them, they can't boot >> anymore >> because of the illegal status of the disks, this leads me in a critical >> situation >> >> VM fedora23 is down with error. Exit message: Unable to get volume size >> for >> domain 5ef8572c-0ab5-4491-994a-e4c30230a525 volume >> e5969faa-97ea-41df-809b-cc62161ab1bc >> >> As far as I didn't initiate any live merge, am I concerned by this bug >> https://bugzilla.redhat.com/show_bug.cgi?id=1306741? >> I'm running 3.6.2, will upgrade to 3.6.3 solve this issue? >> > > If you tried to remove a snapshot while the vm is running you did > initiate live merge, and this bug may effect you. > > Adding Greg for adding more info about this. > > Hi Nathanaël, From the logs you pasted below, showing RemoveSnapshotSingleDiskCommand (not ..SingleDiskLiveCommand), it looks like a non-live snapshot. In that case, bug 1306741 would not affect you. To dig deeper, we'd need to know the root cause of why the image could not be deleted. You should be able to find some clues in your engine log above the snippet you pasted below, or perhaps something in the vdsm log will reveal the reason. Thanks, Greg >> 2016-03-18 18:26:57,652 ERROR >> [org.ovirt.engine.core.bll.RemoveSnapshotCommand] >> (org.ovirt.thread.pool-8-thread-39) [a1e222d] Ending command >> 'org.ovirt.engine.core.bll.RemoveSnapshotCommand' with failure. >> 2016-03-18 18:26:57,663 ERROR >> [org.ovirt.engine.core.bll.RemoveSnapshotCommand] >> (org.ovirt.thread.pool-8-thread-39) [a1e222d] Could not delete image >> '46e9ecc8-e168-4f4d-926c-e769f5df1f2c' from snapshot >> '88fcf167-4302-405e-825f-ad7e0e9f6564' >> 2016-03-18 18:26:57,678 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (org.ovirt.thread.pool-8-thread-39) [a1e222d] Correlation ID: a1e222d, >> Job >> ID: 00d3e364-7e47-4022-82ff-f772cd79d4a1, Call Stack: null, Custom Event >> ID: >> -1, Message: Due to partial snapshot removal, Snapshot 'test' of VM >> 'fedora23' now contains only the following disks: 'fedora23_Disk1'. >> 2016-03-18 18:26:57,695 ERROR >> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskCommand] >> (org.ovirt.thread.pool-8-thread-39) [724e99fd] Ending command >> 'org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskCommand' with failure. >> 2016-03-18 18:26:57,708 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandlin >> >> Thank you for your help. >> >> >> Le 23/02/2016 19:51, Greg Padgett a écrit : >> >>> >>> On 02/22/2016 07:10 AM, Marcelo Leandro wrote: >>> >>>> >>>> Hello, >>>> >>>> The bug with snapshot it will be fixed in ovirt 3.6.3? >>>> >>>> thanks. >>>> >>>> >>> Hi Marcelo, >>> >>> Yes, the bug below (bug 1301709) is now targeted to 3.6.3. >>> >>> Thanks, >>> Greg >>> >>> 2016-02-18 11:34 GMT-03:00 Adam Litke <alitke(a)redhat.com&gt;: >>>> >>>>> >>>>> On 18/02/16 10:37 +0100, Rik Theys wrote: >>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> On 02/17/2016 05:29 PM, Adam Litke wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 17/02/16 11:14 -0500, Greg Padgett wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 02/17/2016 03:42 AM, Rik Theys wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On 02/16/2016 10:52 PM, Greg Padgett wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 02/16/2016 08:50 AM, Rik Theys wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> From the above I conclude that the disk with id that ends >>>>>>>>>>> with >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Similar to what I wrote to Marcelo above in the thread, I'd >>>>>>>>>> recommend >>>>>>>>>> running the "VM disk info gathering tool" attached to [1]. It's >>>>>>>>>> the >>>>>>>>>> best way to ensure the merge was completed and determine which >>>>>>>>>> image >>>>>>>>>> is >>>>>>>>>> the "bad" one that is no longer in use by any volume chains. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I've ran the disk info gathering tool and this outputs (for the >>>>>>>>> affected >>>>>>>>> VM): >>>>>>>>> >>>>>>>>> VM lena >>>>>>>>> Disk b2390535-744f-4c02-bdc8-5a897226554b >>>>>>>>> (sd:a7ba2db3-517c-408a-8b27-ea45989d6416) >>>>>>>>> Volumes: >>>>>>>>> 24d78600-22f4-44f7-987b-fbd866736249 >>>>>>>>> >>>>>>>>> The id of the volume is the ID of the snapshot that is marked >>>>>>>>> "illegal". >>>>>>>>> So the "bad" image would be the dc39 one, which according to the >>>>>>>>> UI >>>>>>>>> is >>>>>>>>> in use by the "Active VM" snapshot. Can this make sense? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> It looks accurate. Live merges are "backwards" merges, so the >>>>>>>> merge >>>>>>>> would have pushed data from the volume associated with "Active VM" >>>>>>>> into the volume associated with the snapshot you're trying to >>>>>>>> remove. >>>>>>>> >>>>>>>> Upon completion, we "pivot" so that the VM uses that older volume, >>>>>>>> and >>>>>>>> we update the engine database to reflect this (basically we >>>>>>>> re-associate that older volume with, in your case, "Active VM"). >>>>>>>> >>>>>>>> In your case, it seems the pivot operation was done, but the >>>>>>>> database >>>>>>>> wasn't updated to reflect it. Given snapshot/image associations >>>>>>>> e.g.: >>>>>>>> >>>>>>>> VM Name Snapshot Name Volume >>>>>>>> ------- ------------- ------ >>>>>>>> My-VM Active VM 123-abc >>>>>>>> My-VM My-Snapshot 789-def >>>>>>>> >>>>>>>> My-VM in your case is actually running on volume 789-def. If you >>>>>>>> run >>>>>>>> the db fixup script and supply ("My-VM", "My-Snapshot", "123-abc") >>>>>>>> (note the volume is the newer, "bad" one), then it will switch the >>>>>>>> volume association for you and remove the invalid entries. >>>>>>>> >>>>>>>> Of course, I'd shut down the VM, and back up the db beforehand. >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> I've executed the sql script and it seems to have worked. Thanks! >>>>>> >>>>>> "Active VM" should now be unused; it previously (pre-merge) was the >>>>>>>> data written since the snapshot was taken. Normally the larger >>>>>>>> actual >>>>>>>> size might be from qcow format overhead. If your listing above is >>>>>>>> complete (ie one volume for the vm), then I'm not sure why the base >>>>>>>> volume would have a larger actual size than virtual size. >>>>>>>> >>>>>>>> Adam, Nir--any thoughts on this? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> There is a bug which has caused inflation of the snapshot volumes >>>>>>> when >>>>>>> performing a live merge. We are submitting fixes for 3.5, 3.6, and >>>>>>> master right at this moment. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Which bug number is assigned to this bug? Will upgrading to a release >>>>>> with a fix reduce the disk usage again? >>>>>> >>>>> >>>>> >>>>> >>>>> See https://bugzilla.redhat.com/show_bug.cgi?id=1301709 for the bug. >>>>> It's about a clone disk failure after the problem occurs. >>>>> Unfortunately, there is not an automatic way to repair the raw base >>>>> volumes if they were affected by this bug. They will need to be >>>>> manually shrunk using lvreduce if you are certain that they are >>>>> inflated. >>>>> >>>>> >>>>> -- >>>>> Adam Litke >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> >> >> -- >> Nathanaël Blanchet >> >> Supervision réseau >> Pôle Infrastrutures Informatiques >> 227 avenue Professeur-Jean-Louis-Viala >> 34193 MONTPELLIER CEDEX 5 >> Tél. 33 (0)4 67 54 84 55 >> Fax 33 (0)4 67 54 84 14 >> blanchet(a)abes.fr >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users