[ovirt-users] Cinder Snapshot Issues

Mon May 23 10:09:07 EDT 2016

The guest-agent processes are running. I recreated the error today doing a
live snapshot with memory. The snapshot was initiated at 13:05:42,854, VM
name is 'ov1', snapshot name is prodsnap10. The engine log goes back far
enough that you can see previous failures as well, but the vdsm log is only
for today.

Kevin

On Mon, May 23, 2016 at 3:31 AM, Daniel Erez <derez at redhat.com> wrote:

>
>
> On Tue, Apr 12, 2016 at 11:13 PM, Kevin Hrpcek <khrpcek at gmail.com> wrote:
>
>> Hello,
>>
>> I'm running into a problem with live snapshots not working when using
>> cinder/ceph disks. There are different failures for including and not
>> including memory, but in each case cinder/ceph creates a new snapshot that
>> can be seen in cinder and ceph. When doing a memory/disk snapshot the VM
>> ends up in a paused state and I need to kill -9 the qemu process to be able
>> to boot the vm again. The engine seems to be losing connection with the
>> vdsm process on the VM host after freezing the guest's filesystems. The
>> guest never receives the thaw command and it fails in the logs. I am
>> pasting in some log snippets.
>>
>> 2016-04-12 19:24:58,851 INFO
>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
>> (org.ovirt.thread.pool-8-thread-27) [5c4493e] Ending command
>> 'org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand' successfully.
>> 2016-04-12 19:27:56,873 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-27) [4d97ca06] Correlation ID: null, Call
>> Stack: null, Custom Event ID: -1, Message: VDSM OVCL1A command failed:
>> Message timeout which can be caused by communication issues
>> 2016-04-12 19:27:56,873 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>> (DefaultQuartzScheduler_Worker-27) [4d97ca06] Command
>> 'org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand' return value
>> 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=5022,
>> message=Message timeout which can be caused by communication issues]]'
>> 2016-04-12 19:27:56,874 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>> (DefaultQuartzScheduler_Worker-27) [4d97ca06] HostName = OVCL1A
>> 2016-04-12 19:27:56,874 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>> (DefaultQuartzScheduler_Worker-27) [4d97ca06] Command
>> 'SnapshotVDSCommand(HostName = OVCL1A,
>> SnapshotVDSCommandParameters:{runAsync='true',
>> hostId='9bdfaedc-34a8-4a08-ad8a-c117835a6094',
>> vmId='040609f6-cfe0-4763-8b32-08ffad158c93'})' execution failed:
>> VDSGenericException: VDSNetworkException: Message timeout which can be
>> caused by communication issues
>> 2016-04-12 19:27:56,875 WARN
>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>> (org.ovirt.thread.pool-8-thread-16) [4d97ca06] Host 'OVCL1A' is not
>> responding.
>>
>> Disk only live snapshots freeze the guest file systems, the vm receives
>> the thaw command, but the VM is no longer responsive. The VM pings on the
>> network but it is hung and it also needs a kill -9 to the qemu process so
>> that it can be booted again.
>>
>> jsonrpc.Executor/0::DEBUG::2016-04-12
>> 19:41:58,342::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
>> 'VM.snapshot' in bridge with {u'frozen': True, u'vmID':
>> u'040609f6-cfe0-4763-8b32-08ffad158c93', u'snapDrives': []}
>> jsonrpc.Executor/0::INFO::2016-04-12
>> 19:41:58,343::vm::3237::virt.vm::(snapshot)
>> vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::<domainsnapshot>
>>         <disks/>
>> </domainsnapshot>
>>
>> jsonrpc.Executor/0::ERROR::2016-04-12
>> 19:41:58,346::vm::3252::virt.vm::(snapshot)
>> vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::Unable to take snapshot
>> Traceback (most recent call last):
>>   File "/usr/share/vdsm/virt/vm.py", line 3250, in snapshot
>>     self._dom.snapshotCreateXML(snapxml, snapFlags)
>>   File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
>>     ret = attr(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line
>> 124, in wrapper
>>     ret = f(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in
>> wrapper
>>     return func(inst, *args, **kwargs)
>>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2581, in
>> snapshotCreateXML
>>     if ret is None:raise libvirtError('virDomainSnapshotCreateXML()
>> failed', dom=self)
>> libvirtError: unsupported configuration: nothing selected for snapshot
>> jsonrpc.Executor/7::DEBUG::2016-04-12
>> 19:41:58,391::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
>> 'VM.thaw' in bridge with {u'vmID': u'040609f6-cfe0-4763-8b32-08ffad158c93'}
>> jsonrpc.Executor/7::INFO::2016-04-12
>> 19:41:58,391::vm::3041::virt.vm::(thaw)
>> vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::Thawing guest filesystems
>> jsonrpc.Executor/7::INFO::2016-04-12
>> 19:41:58,396::vm::3056::virt.vm::(thaw)
>> vmId=`040609f6-cfe0-4763-8b32-08ffad158c93`::6 guest filesystems thawed
>>
>
> It could be an issue of a guest agent. Please make sure the
> ovirt-guest-agent and qemu-guest-agent are installed and running in the VM.
> Further details are available at:
> http://www.ovirt.org/documentation/internal/guest-agent/understanding-guest-agents-and-other-tools/
> In addition, can you please attach full engine/vdsm logs.
>
>
>>
>> Everything else is working well with cinder for running VMs (making
>> disks, running VMs, live migration, etc...). I was able to get live
>> snapshots when using a CephFS Posix storage domain.
>>
>> Versions..
>> Ceph 9.2.0
>> oVirt Latest
>> CentOS 7.2
>> Cinder 7.0.1-1.el7
>>
>> Any help would be appreciated.
>>
>> Thanks,
>> Kevin
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160523/ce838849/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine.log.20160523.gz
Type: application/x-gzip
Size: 368942 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160523/ce838849/attachment-0002.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.log.20160523.gz
Type: application/x-gzip
Size: 39591 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160523/ce838849/attachment-0003.gz>