<div dir="ltr"><div>The guest-agent processes are running. I recreated the error today doing a live snapshot with memory. The snapshot was initiated at 13:05:42,854, VM name is 'ov1', snapshot name is prodsnap10. The engine log goes back far enough that you can see previous failures as well, but the vdsm log is only for today.<br><br></div>Kevin<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 23, 2016 at 3:31 AM, Daniel Erez <span dir="ltr"><<a href="mailto:derez@redhat.com" target="_blank">derez@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Tue, Apr 12, 2016 at 11:13 PM, Kevin Hrpcek <span dir="ltr"><<a href="mailto:khrpcek@gmail.com" target="_blank">khrpcek@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div>Hello,<br><br></div>I'm running into a problem with live snapshots not working when using cinder/ceph disks. There are different failures for including and not including memory, but in each case cinder/ceph creates a new snapshot that can be seen in cinder and ceph. When doing a memory/disk snapshot the VM ends up in a paused state and I need to kill -9 the qemu process to be able to boot the vm again. The engine seems to be losing connection with the vdsm process on the VM host after freezing the guest's filesystems. The guest never receives the thaw command and it fails in the logs. I am pasting in some log snippets.<br><br>2016-04-12 19:24:58,851 INFO [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] (org.ovirt.thread.pool-8-thread-27) [5c4493e] Ending command 'org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand' successfully.<br>2016-04-12 19:27:56,873 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [4d97ca06] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM OVCL1A command failed: Message timeout which can be caused by communication issues<br>2016-04-12 19:27:56,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (DefaultQuartzScheduler_Worker-27) [4d97ca06] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=5022, message=Message timeout which can be caused by communication issues]]'<br>2016-04-12 19:27:56,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (DefaultQuartzScheduler_Worker-27) [4d97ca06] HostName = OVCL1A<br>2016-04-12 19:27:56,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (DefaultQuartzScheduler_Worker-27) [4d97ca06] Command 'SnapshotVDSCommand(HostName = OVCL1A, SnapshotVDSCommandParameters:{runAsync='true', hostId='9bdfaedc-34a8-4a08-ad8a-c117835a6094', vmId='040609f6-cfe0-4763-8b32-08ffad158c93'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues<br>2016-04-12 19:27:56,875 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-8-thread-16) [4d97ca06] Host 'OVCL1A' is not responding.<br><br></div>Disk only live snapshots freeze the guest file systems, the vm receives the thaw command, but the VM is no longer responsive. The VM pings on the network but it is hung and it also needs a kill -9 to the qemu process so that it can be booted again.<br><br>jsonrpc.Executor/0::DEBUG::2016-04-12 19:41:58,342::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.snapshot' in bridge with {u'frozen': True, u'vmID': u'040609f6-cfe0-4763-8b32-08ffad158c93', u'snapDrives': []}<br>jsonrpc.Executor/0::INFO::2016-04-12 19:41:58,343::vm::3237::virt.vm::(snapshot) vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::<domainsnapshot><br> <disks/><br></domainsnapshot><br><br>jsonrpc.Executor/0::ERROR::2016-04-12 19:41:58,346::vm::3252::virt.vm::(snapshot) vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::Unable to take snapshot<br>Traceback (most recent call last):<br> File "/usr/share/vdsm/virt/vm.py", line 3250, in snapshot<br> self._dom.snapshotCreateXML(snapxml, snapFlags)<br> File "/usr/share/vdsm/virt/virdomain.py", line 68, in f<br> ret = attr(*args, **kwargs)<br> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper<br> ret = f(*args, **kwargs)<br> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper<br> return func(inst, *args, **kwargs)<br> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2581, in snapshotCreateXML<br> if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)<br>libvirtError: unsupported configuration: nothing selected for snapshot<br>jsonrpc.Executor/7::DEBUG::2016-04-12 19:41:58,391::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.thaw' in bridge with {u'vmID': u'040609f6-cfe0-4763-8b32-08ffad158c93'}<br>jsonrpc.Executor/7::INFO::2016-04-12 19:41:58,391::vm::3041::virt.vm::(thaw) vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::Thawing guest filesystems<br>jsonrpc.Executor/7::INFO::2016-04-12 19:41:58,396::vm::3056::virt.vm::(thaw) vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::6 guest filesystems thawed<br></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></div></div><div>It could be an issue of a guest agent. Please make sure the ovirt-guest-agent and qemu-guest-agent are installed and running in the VM. Further details are available at: <a href="http://www.ovirt.org/documentation/internal/guest-agent/understanding-guest-agents-and-other-tools/" target="_blank">http://www.ovirt.org/documentation/internal/guest-agent/understanding-guest-agents-and-other-tools/</a></div><div>In addition, can you please attach full engine/vdsm logs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><div dir="ltr"><div><div><div><div><div><div><div><div><div><br></div>Everything else is working well with cinder for running VMs (making disks, running VMs, live migration, etc...). I was able to get live snapshots when using a CephFS Posix storage domain.<br><br></div>Versions..<br></div>Ceph 9.2.0<br></div>oVirt Latest<br></div>CentOS 7.2<br></div>Cinder 7.0.1-1.el7<br><br></div>Any help would be appreciated.<br><br></div>Thanks,<br></div>Kevin<br></div>
<br></span>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>