Re: [ovirt-users] Error creating Disks

The offline disk migration works ----- Original Message ----- From: "Dafna Ron" <dron(a)redhat.com&gt; To: "Maurice James" <mjames(a)media-node.com&gt; Cc: "elad Ben Aharon" <ebenahar(a)redhat.com&gt;, "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org Sent: Wednesday, April 16, 2014 11:57:20 AM Subject: Re: [ovirt-users] Error creating Disks Thanks Maurice. so you are saying that there is some sort of caching. can you migrate the disk off line? Dafna On 04/16/2014 04:06 PM, Maurice James wrote: > I ran "tail -f /var/log/vdsm/vdsm.log |grep ERROR" while attempting a live migration and nothing is coming up, but > "tail -f /var/log/ovirt-engine/engine.log |grep ERROR" returns: > 2014-04-16 11:02:59,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Failed in SnapshotVDS method > 2014-04-16 11:02:59,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48 > 2014-04-16 11:02:59,966 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13) > 2014-04-16 11:02:59,970 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler > > > > ----- Original Message ----- > From: "Dafna Ron" <dron(a)redhat.com&gt; > To: "Maurice James" <mjames(a)media-node.com&gt;, "elad Ben Aharon" <ebenahar(a)redhat.com&gt; > Cc: "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org > Sent: Wednesday, April 16, 2014 11:00:47 AM > Subject: Re: [ovirt-users] Error creating Disks > > Since LiveStorageMigration is a complicated and long task which divides > into 4 different jobs I asked for the log with the echo before the task > so we can easily follow the task from start to end. > > however, I did grep for the task id in the logs you just attached and > there is nothing there. > I'm adding Elad to try and see if he can reproduce ERROR populated by > engine because of cache. > > Dafna > > > On 04/16/2014 03:45 PM, Maurice James wrote: >> I attached a few of the rotated logs, something might be in there >> >> ----- Original Message ----- >> From: "Dafna Ron" <dron(a)redhat.com&gt; >> To: "Liron Aravot" <laravot(a)redhat.com&gt; >> Cc: "Maurice James" <mjames(a)media-node.com&gt;, users(a)ovirt.org >> Sent: Wednesday, April 16, 2014 10:41:59 AM >> Subject: Re: [ovirt-users] Error creating Disks >> >> Thanks Liron, >> >> well, unless the first vdsm log was cut we were not seeing any errors in >> the vdsm log (engine was reporting an issue and vdsm did not show any >> errors at all), after the engine restart we can see an error in the vdsm >> log. >> Maurice, was is possible that the vdsm log was cut? if not, there might >> be a second bug with engine cache. >> >> The Error we are seeing in the vdsm log now does indeed look like the >> live snapshot issue in qemu :) >> >> >> Thanks, >> Dafna >> >> >> >> On 04/16/2014 03:23 PM, Liron Aravot wrote: >>> Hi Maurice, Dafna >>> The creation of the snapshot for each of the disks succeeds, but performing the "live" part of it fails - we can see the following error in the vdsm log. >>> >>> Thread-4444::DEBUG::2014-04-16 09:40:39,071::vm::4007::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Snapshot faile >>> d using the quiesce flag, trying again without it (unsupported configuration: reuse is not supported with this QEMU binary) >>> Thread-4444::DEBUG::2014-04-16 09:40:39,081::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 67 edom: 10 level: >>> 2 message: unsupported configuration: reuse is not supported with this QEMU binary >>> Thread-4444::ERROR::2014-04-16 09:40:39,082::vm::4011::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Unable to take >>> snapshot >>> Traceback (most recent call last): >>> File "/usr/share/vdsm/vm.py", line 4009, in snapshot >>> self._dom.snapshotCreateXML(snapxml, snapFlags) >>> File "/usr/share/vdsm/vm.py", line 859, in f >>> ret = attr(*args, **kwargs) >>> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper >>> ret = f(*args, **kwargs) >>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML >>> if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self) >>> libvirtError: unsupported configuration: reuse is not supported with this QEMU binary >>> Thread-4444::DEBUG::2014-04-16 09:40:39,091::BindingXMLRPC::1074::vds::(wrapper) return vmSnapshot with {'status': {'message': 'Snap >>> shot failed', 'code': 48}} >>> >>> there's open bug on possibly the same issue - but let's verify that it's the same >>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 >>> >>> what OS are you running? if it's not centos, please try to upgrade libvirt and try again. >>> >>> if it's urgent - as i see that you already stopped your vm during those tries, as a temporary solution you can stop the vm, move the disk while it's stopped (which won't be live storage migration) and than start it. >>> >>> ----- Original Message ----- >>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>> Cc: users(a)ovirt.org >>>> Sent: Wednesday, April 16, 2014 4:46:10 PM >>>> Subject: Re: [ovirt-users] Error creating Disks >>>> >>>> can you try to restart the engine? >>>> This should clean the cache and some of the tables holding temporary >>>> task info. >>>> >>>> Dafna >>>> >>>> >>>> >>>> On 04/16/2014 02:37 PM, Maurice James wrote: >>>>> I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What should >>>>> my next step be? >>>>> >>>>> ----- Original Message ----- >>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>> Sent: Wednesday, April 16, 2014 4:44:48 AM >>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>> >>>>> ok... Now I am starting to understand what's going on and if you look at >>>>> the vdsm log, the live snapshot succeeds and no other ERROR are >>>>> reported. I think it's related to the task management on engine side. >>>>> >>>>> Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo >>>>> If you have tasks we would have to stop and clear them (vdsClient -s 0 >>>>> stopTask <task> ; vdsClient -s 0 clearTask <task>) >>>>> after you clear the tasks you will have to restart the engine >>>>> >>>>> Also, what version of ovirt are you using? is it 3.4? because this was >>>>> suppose to be fixed... >>>>> >>>>> here is the explanation from what I see in the logs: >>>>> >>>>> you are sending a command to Live Migrate >>>>> >>>>> 2014-04-15 12:50:54,381 INFO >>>>> [org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4) >>>>> [4c089392] Running command: MoveDisksCommand internal: false. Entities >>>>> affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk >>>>> 2014-04-15 12:50:54,520 INFO >>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>> (ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock >>>>> [exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK >>>>> , sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM >>>>> ] >>>>> >>>>> The fist step is creating the snapshot: >>>>> >>>>> 2014-04-15 12:50:54,734 INFO >>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>> (org.ovirt.thread.pool-6-thread-47) Running command: >>>>> CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID: >>>>> ba49605b-fb7e-4a70-a3 >>>>> 80-6286d3903e50 Type: VM >>>>> 2014-04-15 12:50:54,748 INFO >>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] >>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command: >>>>> CreateSnapshotCommand internal: true. Entities affected : ID: >>>>> 00000000-0000-0000-0000-0000000 >>>>> 00000 Type: Storage >>>>> 2014-04-15 12:50:54,760 INFO >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] START, >>>>> CreateSnapshotVDSCommand( storagePoolId = >>>>> a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor >>>>> eFailoverLimit = false, storageDomainId = >>>>> 3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId = >>>>> c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400, >>>>> volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803, >>>>> newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661, >>>>> sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: f3b7d8b >>>>> >>>>> >>>>> which actually succeeds in vdsm and reported as successful to the engine: >>>>> >>>>> engine: >>>>> >>>>> 2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>> (DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task >>>>> db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command LiveMigrateVmDisks, >>>>> Parameters Type >>>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned >>>>> status finished, result 'success'. >>>>> >>>>> vdsm: >>>>> >>>>> Thread-4670::DEBUG::2014-04-15 >>>>> 12:51:07,287::task::1185::TaskManager.Task::(prepare) >>>>> Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished: >>>>> {'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0, >>>>> 'message': '1 jobs comple >>>>> ted successfully', 'taskState': 'finished', 'taskResult': 'success', >>>>> 'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}} >>>>> >>>>> >>>>> now engine has to end the task: >>>>> >>>>> 2014-04-15 12:51:07,301 INFO >>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>> (DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending >>>>> action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6): >>>>> calling endAction . >>>>> 2014-04-15 12:51:07,303 INFO >>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>> (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction >>>>> [within thread] context: Attempting to endAction LiveMigrateVmDisks, >>>>> executionIndex: 0 >>>>> 2014-04-15 12:51:07,315 INFO >>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand >>>>> 2014-04-15 12:51:07,319 INFO >>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand >>>>> >>>>> and than >>>>> >>>>> we see an exception: >>>>> >>>>> 2014-04-15 12:51:07,498 WARN >>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>> (org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to >>>>> error: VdcBLLException: VdcBLLException: >>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code >>>>> 48). VM will still be configured to the new created snapshot >>>>> 2014-04-15 12:51:07,543 INFO >>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> (org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack: >>>>> org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: >>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48) >>>>> at >>>>> org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) >>>>> at >>>>> org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) >>>>> at >>>>> org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) >>>>> at >>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354) >>>>> at >>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351) >>>>> at >>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) >>>>> at >>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) >>>>> at >>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351) >>>>> at >>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273) >>>>> at >>>>> org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304) >>>>> at >>>>> org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614) >>>>> at >>>>> org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560) >>>>> at >>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886) >>>>> at >>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151) >>>>> at >>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118) >>>>> at >>>>> org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492) >>>>> at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446) >>>>> at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 04/15/2014 05:54 PM, Maurice James wrote: >>>>>> Logs are attached. >>>>>> >>>>>> Live Migration failed >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>> Sent: Tuesday, April 15, 2014 10:46:29 AM >>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>> >>>>>> yeah, but is it the same error we see before in vdsm? :) >>>>>> lets try to follow the action in both logs so we can debug it better, >>>>>> can you do the following? >>>>>> >>>>>> 1. stop the vm and clean all snapshots. >>>>>> 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >> >>>>>> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >> >>>>>> /var/log/vdsm/vdsm.log) >>>>>> 3. start the vm >>>>>> 4. try to live migrate the disk >>>>>> 5. attach the full engine and vdsm logs. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Dafna >>>>>> >>>>>> >>>>>> On 04/15/2014 03:41 PM, Maurice James wrote: >>>>>>> It failed again. Its reported in the same location. >>>>>>> >>>>>>> Snip of engine.log >>>>>>> >>>>>>> 2014-04-15 10:41:17,174 ERROR >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method >>>>>>> 2014-04-15 10:41:17,178 ERROR >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName >>>>>>> = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, >>>>>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: >>>>>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>>>>>> SnapshotVDS, error = Snapshot failed, code = 48 >>>>>>> 2014-04-15 10:41:17,531 ERROR >>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-34) Command >>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll >>>>>>> exception. With error message VdcBLLException: Auto-generated live >>>>>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with >>>>>>> error imageErr and code 13) >>>>>>> 2014-04-15 10:41:17,534 ERROR >>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: >>>>>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>> Sent: Tuesday, April 15, 2014 10:37:41 AM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> ok. >>>>>>> so the vm was down and you deleted all the snapshots. >>>>>>> in the webadmin, when you select the disk you want to live migrate, is >>>>>>> it the same storage reported in vdsm? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> On 04/15/2014 03:25 PM, Maurice James wrote: >>>>>>>> I restarted from the VM. The VM now has 0 snapshots. All of the >>>>>>>> snapshots in the screen shots never completed, so I deleted them while >>>>>>>> the VM was powerd off from the ui >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>> Sent: Tuesday, April 15, 2014 10:21:06 AM >>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>> >>>>>>>> did you restart the vm from the vm internally or stop -> start the vm? >>>>>>>> >>>>>>>> how many snapshots does the vm have? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>>> >>>>>>>> On 04/15/2014 03:16 PM, Maurice James wrote: >>>>>>>>> I restarted the VM. Same issue. I was able to find the physical >>>>>>>>> location on disk. I attached 2 screen shots showing the location of >>>>>>>>> the disks on the file system. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>> Sent: Tuesday, April 15, 2014 10:12:56 AM >>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>> >>>>>>>>> it seems that the issue is the links between the engine and the vdsm >>>>>>>>> >>>>>>>>> why don't you check where the disk is physically located and than try >>>>>>>>> to >>>>>>>>> restart the vm? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dafna >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/15/2014 02:57 PM, Maurice James wrote: >>>>>>>>>> yes I did see a warning about restarting. It said: >>>>>>>>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'. >>>>>>>>>> VM restart is recommended >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM >>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>> >>>>>>>>>> Hi Maurice, >>>>>>>>>> >>>>>>>>>> are you getting any errors/alert about restarting the vm in the event >>>>>>>>>> log (in the webadmin). >>>>>>>>>> >>>>>>>>>> I believe that you have a mismatch in the directory link between the >>>>>>>>>> db >>>>>>>>>> and vdsm which was created when you tried to live migrate and failed >>>>>>>>>> the >>>>>>>>>> fist time). >>>>>>>>>> >>>>>>>>>> we can see in the vdsm log that createVolume (which is part of the >>>>>>>>>> live >>>>>>>>>> migration) is failing with this error: >>>>>>>>>> >>>>>>>>>> OSError: [Errno 2] No such file or directory: >>>>>>>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b' >>>>>>>>>> >>>>>>>>>> so it seems that the engine is pointing to a link that does not exist >>>>>>>>>> and we need to find on what storage the disk actually exists on and >>>>>>>>>> where the engine is pointing to. >>>>>>>>>> >>>>>>>>>> you can look at the webadmin at the disk and see where it's suppose to >>>>>>>>>> be and than look in /rhev and see where the disk link is in vdsm (if >>>>>>>>>> you >>>>>>>>>> can see where the disk is actually is - physically in the storage, >>>>>>>>>> that >>>>>>>>>> would be good too). >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Dafna >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/15/2014 01:16 PM, Maurice James wrote: >>>>>>>>>>> As far as the disk creation , I got that sorted out. It seem that my >>>>>>>>>>> vdsm versions were out of sync on my hosts. When I updated them the >>>>>>>>>>> disk creation began working, but im still having issues with live >>>>>>>>>>> disk migrations. >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM >>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>> >>>>>>>>>>> Also, before the task fails, there are ERRORs in the log which are >>>>>>>>>>> reporting problems in the DC >>>>>>>>>>> >>>>>>>>>>> 2014-04-14 09:56:05,161 ERROR >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-16) Command >>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>> connect timed out >>>>>>>>>>> 2014-04-14 09:56:10,226 ERROR >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-49) Command >>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>> connect timed out >>>>>>>>>>> 2014-04-14 09:56:15,283 ERROR >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-81) Command >>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>> connect timed out >>>>>>>>>>> 2014-04-14 09:56:20,342 ERROR >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-58) Command >>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>> connect timed out >>>>>>>>>>> 2014-04-14 09:56:25,409 ERROR >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-84) Command >>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>> connect timed out >>>>>>>>>>> 2014-04-14 09:56:30,542 INFO >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-91) START, >>>>>>>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: >>>>>>>>>>> 30f312c >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> than you stop the vm and fail the AddDisk. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> is there a chance you were having problems with your storage? >>>>>>>>>>> >>>>>>>>>>> Dafna >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/14/2014 04:04 PM, Maurice James wrote: >>>>>>>>>>>> Snapshot creation seems to be failing as well >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt;, "Federico Simoncelli" >>>>>>>>>>>> <fsimonce(a)redhat.com&gt; >>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM >>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>> >>>>>>>>>>>> Hi Federico, >>>>>>>>>>>> Can you please take a look? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM >>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>> >>>>>>>>>>>>> Logs attached >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM >>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> IMHO not enough info is provided, >>>>>>>>>>>>> Can you please provide full engine.log and relevant vdsm.log? >>>>>>>>>>>>> >>>>>>>>>>>>> THanks, >>>>>>>>>>>>> Yair >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM >>>>>>>>>>>>>> Subject: [ovirt-users] Error creating Disks >>>>>>>>>>>>>> >>>>>>>>>>>>>> oVirt Engine Version: >>>>>>>>>>>>>> 3.4.1-0.0.master.20140412010845.git43746c6.el6 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> While attempting to create a disk on an NFS storage domain, it >>>>>>>>>>>>>> fails with >>>>>>>>>>>>>> the >>>>>>>>>>>>>> following error in the engine.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2014-04-14 09:58:12,127 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in >>>>>>>>>>>>>> HSMGetAllTasksStatusesVDS >>>>>>>>>>>>>> method >>>>>>>>>>>>>> 2014-04-14 09:58:12,139 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) >>>>>>>>>>>>>> BaseAsyncTask::LogEndTaskFailure: Task >>>>>>>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, >>>>>>>>>>>>>> Parameters >>>>>>>>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) >>>>>>>>>>>>>> ended >>>>>>>>>>>>>> with >>>>>>>>>>>>>> failure: >>>>>>>>>>>>>> 2014-04-14 09:58:12,159 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddDiskCommand] >>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with >>>>>>>>>>>>>> failure: >>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand >>>>>>>>>>>>>> 2014-04-14 09:58:12,212 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand] >>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with >>>>>>>>>>>>>> failure: >>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> -- >>>> Dafna Ron >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>>