Hi all, i¹ve encountered an issue with snapshots on running VM¹s. I¹ve searched the forums but can¹t find anything that would be wrong with my setup. Environment: CentOS release 6.5 (Final Linux dl380.dmz.vireone.com 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Ovirt : 3.4.2-1 qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64 vdsm-cli-4.14.9-0.el6.noarch Log from engine log: 2014-07-15 20:46:38,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-6-thread-28) [5201147a] START, GetImageInfoVDSCommand( storagePoolId = deb50055-1955-4993-bfcb-a642d71e84ce, ignoreFailoverLimit = false, storageDomainId = c5ad71bf-e7a9-4ea0-b7a5-89afd5c2a61e, imageGroupId = a9210478-e9f8-4cae-a074-9cb49b1445eb, imageId = 0df50d34-157f-4272-9326-f8f8d4beab0d), log id: 74fde04 2014-07-15 20:46:38,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-6-thread-28) [5201147a] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@298f6049, log id: 74fde04 2014-07-15 20:46:38,245 INFO [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] (org.ovirt.thread.pool-6-thread-28) Lock freed to object EngineLock [exclusiveLocks= key: 6a507e40-3d0d-48cc-8507-677921f3fd04 value: VM , sharedLocks= ] 2014-07-15 20:46:38,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) Correlation ID: 6bc2972a, Job ID: b32e4c57-6255-4398-8e58-bd5053fbec01, Call Stack: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFron tendImpl.java:33) at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1981) at org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransactio n(CreateAllSnapshotsFromVmCommand.java:354) at org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransactio n(CreateAllSnapshotsFromVmCommand.java:351) at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppres sed(TransactionSupport.java:174) at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(T ransactionSupport.java:116) at org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapsh ot(CreateAllSnapshotsFromVmCommand.java:351) at org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(Crea teAllSnapshotsFromVmCommand.java:273) at org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304) at org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.j ava:614) at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBa se.java:560) at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:189 6) at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTran saction(TransactionSupport.java:210) at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequire d(TransactionSupport.java:149) at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(T ransactionSupport.java:118) at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492) at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5 7) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedR eferenceMethodInterceptor.processInvocation(ManagedReferenceMethodIntercept orFactory.java:72) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorConte xt.java:374) at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor .injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13 ) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory $ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedRefere nceLifecycleMethodInterceptorFactory.java:123) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor. java:53) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvo cation(UserInterceptorFactory.java:36) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercepto r.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercepto r.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proce ssInvocation(ComponentDispatcherInterceptor.java:53) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociation Interceptor.processInvocation(SingletonComponentInstanceAssociationIntercep tor.java:53) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:21 1) at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.ja va:194) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercepto r.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocati on(LoggingInterceptor.java:59) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Nam espaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor .java:45) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercepto r.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescripti on.java:173) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288 ) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercepto r.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHand ler.java:72) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.endAction(Unk nown Source) at org.ovirt.engine.core.bll.CommandAsyncTask.EndCommandAction(CommandAsyncTas k.java:147) at org.ovirt.engine.core.bll.CommandAsyncTask.access$000(CommandAsyncTask.java :25) at org.ovirt.engine.core.bll.CommandAsyncTask$1.run(CommandAsyncTask.java:106) at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnab le.run(ThreadPoolUtil.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:744) , Custom Event ID: -1, Message: Failed to complete snapshot 'c' creation for VM 'Chris_VM_Linux'. 2014-07-15 20:46:38,279 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-28) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateAllSnapshotsFromVm completed, handling the result. 2014-07-15 20:46:38,280 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-28) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateAllSnapshotsFromVm hasn't succeeded, clearing tasks. 2014-07-15 20:46:38,285 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 1f07c8f3-d59e-4dfe-a4b4-04de6180dc07 2014-07-15 20:46:38,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, SPMClearTaskVDSCommand( storagePoolId = deb50055-1955-4993-bfcb-a642d71e84ce, ignoreFailoverLimit = false, taskId = 1f07c8f3-d59e-4dfe-a4b4-04de6180dc07), log id: 7ca88030 2014-07-15 20:46:38,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, HSMClearTaskVDSCommand(HostName = local_host, HostId = 1d734ebb-f32f-41ca-8da8-86cfae93b1e6, taskId=1f07c8f3-d59e-4dfe-a4b4-04de6180dc07), log id: 1a780f40 2014-07-15 20:46:38,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, HSMClearTaskVDSCommand, log id: 1a780f40 2014-07-15 20:46:38,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, SPMClearTaskVDSCommand, log id: 7ca88030 2014-07-15 20:46:38,297 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) BaseAsyncTask::RemoveTaskFromDB: Removed task 1f07c8f3-d59e-4dfe-a4b4-04de6180dc07 from DataBase 2014-07-15 20:46:38,297 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 8d8a5e36-5c65-4ae5-b7a1-8a2140a8fc50 2014-07-15 20:46:38,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, SPMClearTaskVDSCommand( storagePoolId = deb50055-1955-4993-bfcb-a642d71e84ce, ignoreFailoverLimit = false, taskId = 8d8a5e36-5c65-4ae5-b7a1-8a2140a8fc50), log id: 31e0689b 2014-07-15 20:46:38,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, HSMClearTaskVDSCommand(HostName = local_host, HostId = 1d734ebb-f32f-41ca-8da8-86cfae93b1e6, taskId=8d8a5e36-5c65-4ae5-b7a1-8a2140a8fc50), log id: 3f3b1b2 2014-07-15 20:46:38,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, HSMClearTaskVDSCommand, log id: 3f3b1b2 2014-07-15 20:46:38,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, SPMClearTaskVDSCommand, log id: 31e0689b 2014-07-15 20:46:38,349 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) BaseAsyncTask::RemoveTaskFromDB: Removed task 8d8a5e36-5c65-4ae5-b7a1-8a2140a8fc50 from DataBase 2014-07-15 20:46:38,349 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 814be30c-550c-4a86-accc-2ab8ff7d0bf8 2014-07-15 20:46:38,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, SPMClearTaskVDSCommand( storagePoolId = deb50055-1955-4993-bfcb-a642d71e84ce, ignoreFailoverLimit = false, taskId = 814be30c-550c-4a86-accc-2ab8ff7d0bf8), log id: 26d84272 2014-07-15 20:46:38,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) START, HSMClearTaskVDSCommand(HostName = local_host, HostId = 1d734ebb-f32f-41ca-8da8-86cfae93b1e6, taskId=814be30c-550c-4a86-accc-2ab8ff7d0bf8), log id: 3ff25e4a 2014-07-15 20:46:38,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, HSMClearTaskVDSCommand, log id: 3ff25e4a 2014-07-15 20:46:38,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-28) FINISH, SPMClearTaskVDSCommand, log id: 26d84272 2014-07-15 20:46:38,402 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-28) BaseAsyncTask::RemoveTaskFromDB: Removed task 814be30c-550c-4a86-accc-2ab8ff7d0bf8 from DataBase 2014-07-15 20:46:38,402 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-28) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity f4645466-7b94-4dc6-a2aa-0349f3c2336c