Snapshot Locked for an extended period of time
Hi All, I’ve recently written a script which: 1. Takes a snapshot with memory. 2. Creates a VM from that snapshot. 3. Exports the snapshot to the export domain. 4. Deletes the snapshot. 5. Deletes the cloned VM. Whilst running the script for the first time last night, the script timed out whilst deleting the snapshot on the VM at step 4. Now the logs are filled with: 2016-02-25 10:32:53,272 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-25) [4559dd29] Waiting on Live Merge child commands to complete 2016-02-25 10:32:53,332 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-25) [225b841f] Waiting on Live Merge command step MERGE to complete 2016-02-25 10:32:53,335 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-25) [4559dd29] Waiting on merge command to complete 2016-02-25 10:32:55,331 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:32:58,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress, updating 2016-02-25 10:33:01,521 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:03,337 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-12) [4559dd29] Waiting on Live Merge child commands to complete 2016-02-25 10:33:03,350 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-12) [225b841f] Waiting on Live Merge command step MERGE to complete 2016-02-25 10:33:03,352 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-12) [4559dd29] Waiting on merge command to complete 2016-02-25 10:33:04,678 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:07,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:10,993 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress, updating 2016-02-25 10:33:13,354 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-14) [4559dd29] Waiting on Live Merge child commands to complete I have attached the log of when the snapshot started deleting (2016-02- 24 23:51:17,094). Any idea what's gone wrong? What are my next steps to clear this? Cheers, Simon Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine Winner of the Environment & Conservation category, the Charity Awards 2014. Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
Hi Simon, 1) What is the ovirt version you are using? 2) On what is the storage type (nfs, iscsi, ...) you creating the snapshot? * It seems the the logs are not from the time this issue occur Thanks, Raz Tamir Red Hat Israel On Thu, Feb 25, 2016 at 1:43 PM, Simon Hallam <sha@pml.ac.uk> wrote:
Hi All,
I’ve recently written a script which: 1. Takes a snapshot with memory. 2. Creates a VM from that snapshot. 3. Exports the snapshot to the export domain. 4. Deletes the snapshot. 5. Deletes the cloned VM.
Whilst running the script for the first time last night, the script timed out whilst deleting the snapshot on the VM at step 4.
Now the logs are filled with:
2016-02-25 10:32:53,272 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-25) [4559dd29] Waiting on Live Merge child commands to complete 2016-02-25 10:32:53,332 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-25) [225b841f] Waiting on Live Merge command step MERGE to complete 2016-02-25 10:32:53,335 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-25) [4559dd29] Waiting on merge command to complete 2016-02-25 10:32:55,331 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:32:58,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress, updating 2016-02-25 10:33:01,521 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:03,337 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-12) [4559dd29] Waiting on Live Merge child commands to complete 2016-02-25 10:33:03,350 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-12) [225b841f] Waiting on Live Merge command step MERGE to complete 2016-02-25 10:33:03,352 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-12) [4559dd29] Waiting on merge command to complete 2016-02-25 10:33:04,678 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:07,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress (no change) 2016-02-25 10:33:10,993 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [4559dd29] VM job c1064551-022f-4549-a9d4-bb915e177e80: In progress, updating 2016-02-25 10:33:13,354 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-14) [4559dd29] Waiting on Live Merge child commands to complete
I have attached the log of when the snapshot started deleting (2016-02- 24 23:51:17,094).
Any idea what's gone wrong? What are my next steps to clear this?
Cheers,
Simon
Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine
Winner of the Environment & Conservation category, the Charity Awards 2014.
Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK.
This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, 2016-02-25 at 17:25 +0200, Raz Tamir wrote:
Hi Simon, 1) What is the ovirt version you are using?
I'm currently using oVirt Engine Version: 3.5.1-1.el6 The problem is I would like to backup all the VMs before I try upgrading the hosted engine.
2) On what is the storage type (nfs, iscsi, ...) you creating the snapshot?
All of the storage domains are NFS.
* It seems the the logs are not from the time this issue occur
Hmm, it should be starting about line 75618 of that log. Cheers, Simon Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine Winner of the Environment & Conservation category, the Charity Awards 2014. Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
Hi Simon, It looks like a bug. Please file a bug and attach vdsm and engine logs Thanks, Raz Tamir Red Hat Israel On Thu, Feb 25, 2016 at 6:36 PM, Simon Hallam <sha@pml.ac.uk> wrote:
On Thu, 2016-02-25 at 17:25 +0200, Raz Tamir wrote:
Hi Simon, 1) What is the ovirt version you are using?
I'm currently using oVirt Engine Version: 3.5.1-1.el6 The problem is I would like to backup all the VMs before I try upgrading the hosted engine.
2) On what is the storage type (nfs, iscsi, ...) you creating the snapshot?
All of the storage domains are NFS.
* It seems the the logs are not from the time this issue occur
Hmm, it should be starting about line 75618 of that log.
Cheers,
Simon
Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine
Winner of the Environment & Conservation category, the Charity Awards 2014.
Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK.
This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
On Thu, Feb 25, 2016 at 6:36 PM, Simon Hallam <sha@pml.ac.uk> wrote:
On Thu, 2016-02-25 at 17:25 +0200, Raz Tamir wrote:
Hi Simon, 1) What is the ovirt version you are using?
I'm currently using oVirt Engine Version: 3.5.1-1.el6
It makes sense to upgrade to a newer version of oVirt. Since 3.5.1, many issues have been fixed. Y.
The problem is I would like to backup all the VMs before I try upgrading the hosted engine.
2) On what is the storage type (nfs, iscsi, ...) you creating the snapshot?
All of the storage domains are NFS.
* It seems the the logs are not from the time this issue occur
Hmm, it should be starting about line 75618 of that log.
Cheers,
Simon
Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine
Winner of the Environment & Conservation category, the Charity Awards 2014.
Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK.
This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Raz Tamir -
Simon Hallam -
Yaniv Kaul