Re: [ovirt-users] Error creating Disks

Just curious. I have come across a few major problems since upgrading to 3.4.x (External Authentication BZ1081204, Quota assignment BZ1081014, Live migration "qemu bug" ) to name a few. I imagine that all of the features need to be polished before inclusion into RHEL. I'm trying to get my company to adopt RHEL instead of V-Sphere but that is becoming a little tricky. Minus the cool new features of 3.4.x 3.3.4 was pretty stable IMO. ----- Original Message ----- From: "Dafna Ron" <dron(a)redhat.com&gt; To: "Maurice James" <mjames(a)media-node.com&gt;, "Liron Aravot" <laravot(a)redhat.com&gt;, "itamar Heim" <iheim(a)redhat.com&gt; Cc: "elad Ben Aharon" <ebenahar(a)redhat.com&gt;, users(a)ovirt.org Sent: Thursday, April 17, 2014 5:31:46 AM Subject: Re: [ovirt-users] Error creating Disks I am not sure this question relates to this thread and perhaps it should be posted in a different one :) can you explain what you mean by that question? On 04/17/2014 01:09 AM, Maurice James wrote: > Which version of Ovirt are you guys going to build the new RHEV from? > > ----- Original Message ----- > From: "Dafna Ron" <dron(a)redhat.com&gt; > To: "Maurice James" <mjames(a)media-node.com&gt; > Cc: "elad Ben Aharon" <ebenahar(a)redhat.com&gt;, "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org > Sent: Wednesday, April 16, 2014 5:39:20 PM > Subject: Re: [ovirt-users] Error creating Disks > > Ok. > so it's the qemu bug that Liron sent and I think there is some bug there > with the engine cache since we did not see the job failing in vdsm log. > hopefully there will be a qemu patch for centos soon... > > Thanks Maurice! > > Dafna > > > On 04/16/2014 07:14 PM, Maurice James wrote: >> The offline disk migration works >> >> ----- Original Message ----- >> From: "Dafna Ron" <dron(a)redhat.com&gt; >> To: "Maurice James" <mjames(a)media-node.com&gt; >> Cc: "elad Ben Aharon" <ebenahar(a)redhat.com&gt;, "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org >> Sent: Wednesday, April 16, 2014 11:57:20 AM >> Subject: Re: [ovirt-users] Error creating Disks >> >> Thanks Maurice. >> so you are saying that there is some sort of caching. >> >> can you migrate the disk off line? >> >> Dafna >> >> >> On 04/16/2014 04:06 PM, Maurice James wrote: >>> I ran "tail -f /var/log/vdsm/vdsm.log |grep ERROR" while attempting a live migration and nothing is coming up, but >>> "tail -f /var/log/ovirt-engine/engine.log |grep ERROR" returns: >>> 2014-04-16 11:02:59,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Failed in SnapshotVDS method >>> 2014-04-16 11:02:59,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48 >>> 2014-04-16 11:02:59,966 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13) >>> 2014-04-16 11:02:59,970 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>> >>> >>> >>> ----- Original Message ----- >>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>> To: "Maurice James" <mjames(a)media-node.com&gt;, "elad Ben Aharon" <ebenahar(a)redhat.com&gt; >>> Cc: "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org >>> Sent: Wednesday, April 16, 2014 11:00:47 AM >>> Subject: Re: [ovirt-users] Error creating Disks >>> >>> Since LiveStorageMigration is a complicated and long task which divides >>> into 4 different jobs I asked for the log with the echo before the task >>> so we can easily follow the task from start to end. >>> >>> however, I did grep for the task id in the logs you just attached and >>> there is nothing there. >>> I'm adding Elad to try and see if he can reproduce ERROR populated by >>> engine because of cache. >>> >>> Dafna >>> >>> >>> On 04/16/2014 03:45 PM, Maurice James wrote: >>>> I attached a few of the rotated logs, something might be in there >>>> >>>> ----- Original Message ----- >>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>> To: "Liron Aravot" <laravot(a)redhat.com&gt; >>>> Cc: "Maurice James" <mjames(a)media-node.com&gt;, users(a)ovirt.org >>>> Sent: Wednesday, April 16, 2014 10:41:59 AM >>>> Subject: Re: [ovirt-users] Error creating Disks >>>> >>>> Thanks Liron, >>>> >>>> well, unless the first vdsm log was cut we were not seeing any errors in >>>> the vdsm log (engine was reporting an issue and vdsm did not show any >>>> errors at all), after the engine restart we can see an error in the vdsm >>>> log. >>>> Maurice, was is possible that the vdsm log was cut? if not, there might >>>> be a second bug with engine cache. >>>> >>>> The Error we are seeing in the vdsm log now does indeed look like the >>>> live snapshot issue in qemu :) >>>> >>>> >>>> Thanks, >>>> Dafna >>>> >>>> >>>> >>>> On 04/16/2014 03:23 PM, Liron Aravot wrote: >>>>> Hi Maurice, Dafna >>>>> The creation of the snapshot for each of the disks succeeds, but performing the "live" part of it fails - we can see the following error in the vdsm log. >>>>> >>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,071::vm::4007::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Snapshot faile >>>>> d using the quiesce flag, trying again without it (unsupported configuration: reuse is not supported with this QEMU binary) >>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,081::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 67 edom: 10 level: >>>>> 2 message: unsupported configuration: reuse is not supported with this QEMU binary >>>>> Thread-4444::ERROR::2014-04-16 09:40:39,082::vm::4011::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Unable to take >>>>> snapshot >>>>> Traceback (most recent call last): >>>>> File "/usr/share/vdsm/vm.py", line 4009, in snapshot >>>>> self._dom.snapshotCreateXML(snapxml, snapFlags) >>>>> File "/usr/share/vdsm/vm.py", line 859, in f >>>>> ret = attr(*args, **kwargs) >>>>> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper >>>>> ret = f(*args, **kwargs) >>>>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML >>>>> if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self) >>>>> libvirtError: unsupported configuration: reuse is not supported with this QEMU binary >>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,091::BindingXMLRPC::1074::vds::(wrapper) return vmSnapshot with {'status': {'message': 'Snap >>>>> shot failed', 'code': 48}} >>>>> >>>>> there's open bug on possibly the same issue - but let's verify that it's the same >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 >>>>> >>>>> what OS are you running? if it's not centos, please try to upgrade libvirt and try again. >>>>> >>>>> if it's urgent - as i see that you already stopped your vm during those tries, as a temporary solution you can stop the vm, move the disk while it's stopped (which won't be live storage migration) and than start it. >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>> Cc: users(a)ovirt.org >>>>>> Sent: Wednesday, April 16, 2014 4:46:10 PM >>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>> >>>>>> can you try to restart the engine? >>>>>> This should clean the cache and some of the tables holding temporary >>>>>> task info. >>>>>> >>>>>> Dafna >>>>>> >>>>>> >>>>>> >>>>>> On 04/16/2014 02:37 PM, Maurice James wrote: >>>>>>> I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What should >>>>>>> my next step be? >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>> Sent: Wednesday, April 16, 2014 4:44:48 AM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> ok... Now I am starting to understand what's going on and if you look at >>>>>>> the vdsm log, the live snapshot succeeds and no other ERROR are >>>>>>> reported. I think it's related to the task management on engine side. >>>>>>> >>>>>>> Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo >>>>>>> If you have tasks we would have to stop and clear them (vdsClient -s 0 >>>>>>> stopTask <task> ; vdsClient -s 0 clearTask <task>) >>>>>>> after you clear the tasks you will have to restart the engine >>>>>>> >>>>>>> Also, what version of ovirt are you using? is it 3.4? because this was >>>>>>> suppose to be fixed... >>>>>>> >>>>>>> here is the explanation from what I see in the logs: >>>>>>> >>>>>>> you are sending a command to Live Migrate >>>>>>> >>>>>>> 2014-04-15 12:50:54,381 INFO >>>>>>> [org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4) >>>>>>> [4c089392] Running command: MoveDisksCommand internal: false. Entities >>>>>>> affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk >>>>>>> 2014-04-15 12:50:54,520 INFO >>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>> (ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock >>>>>>> [exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK >>>>>>> , sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM >>>>>>> ] >>>>>>> >>>>>>> The fist step is creating the snapshot: >>>>>>> >>>>>>> 2014-04-15 12:50:54,734 INFO >>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-47) Running command: >>>>>>> CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID: >>>>>>> ba49605b-fb7e-4a70-a3 >>>>>>> 80-6286d3903e50 Type: VM >>>>>>> 2014-04-15 12:50:54,748 INFO >>>>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command: >>>>>>> CreateSnapshotCommand internal: true. Entities affected : ID: >>>>>>> 00000000-0000-0000-0000-0000000 >>>>>>> 00000 Type: Storage >>>>>>> 2014-04-15 12:50:54,760 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] START, >>>>>>> CreateSnapshotVDSCommand( storagePoolId = >>>>>>> a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor >>>>>>> eFailoverLimit = false, storageDomainId = >>>>>>> 3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId = >>>>>>> c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400, >>>>>>> volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803, >>>>>>> newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661, >>>>>>> sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: f3b7d8b >>>>>>> >>>>>>> >>>>>>> which actually succeeds in vdsm and reported as successful to the engine: >>>>>>> >>>>>>> engine: >>>>>>> >>>>>>> 2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>> (DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task >>>>>>> db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command LiveMigrateVmDisks, >>>>>>> Parameters Type >>>>>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned >>>>>>> status finished, result 'success'. >>>>>>> >>>>>>> vdsm: >>>>>>> >>>>>>> Thread-4670::DEBUG::2014-04-15 >>>>>>> 12:51:07,287::task::1185::TaskManager.Task::(prepare) >>>>>>> Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished: >>>>>>> {'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0, >>>>>>> 'message': '1 jobs comple >>>>>>> ted successfully', 'taskState': 'finished', 'taskResult': 'success', >>>>>>> 'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}} >>>>>>> >>>>>>> >>>>>>> now engine has to end the task: >>>>>>> >>>>>>> 2014-04-15 12:51:07,301 INFO >>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>>>> (DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending >>>>>>> action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6): >>>>>>> calling endAction . >>>>>>> 2014-04-15 12:51:07,303 INFO >>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>>>> (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction >>>>>>> [within thread] context: Attempting to endAction LiveMigrateVmDisks, >>>>>>> executionIndex: 0 >>>>>>> 2014-04-15 12:51:07,315 INFO >>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand >>>>>>> 2014-04-15 12:51:07,319 INFO >>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand >>>>>>> >>>>>>> and than >>>>>>> >>>>>>> we see an exception: >>>>>>> >>>>>>> 2014-04-15 12:51:07,498 WARN >>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>> (org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to >>>>>>> error: VdcBLLException: VdcBLLException: >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code >>>>>>> 48). VM will still be configured to the new created snapshot >>>>>>> 2014-04-15 12:51:07,543 INFO >>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>> (org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack: >>>>>>> org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351) >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886) >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151) >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118) >>>>>>> at >>>>>>> org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492) >>>>>>> at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446) >>>>>>> at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source) >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 04/15/2014 05:54 PM, Maurice James wrote: >>>>>>>> Logs are attached. >>>>>>>> >>>>>>>> Live Migration failed >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>> Sent: Tuesday, April 15, 2014 10:46:29 AM >>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>> >>>>>>>> yeah, but is it the same error we see before in vdsm? :) >>>>>>>> lets try to follow the action in both logs so we can debug it better, >>>>>>>> can you do the following? >>>>>>>> >>>>>>>> 1. stop the vm and clean all snapshots. >>>>>>>> 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >> >>>>>>>> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >> >>>>>>>> /var/log/vdsm/vdsm.log) >>>>>>>> 3. start the vm >>>>>>>> 4. try to live migrate the disk >>>>>>>> 5. attach the full engine and vdsm logs. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Dafna >>>>>>>> >>>>>>>> >>>>>>>> On 04/15/2014 03:41 PM, Maurice James wrote: >>>>>>>>> It failed again. Its reported in the same location. >>>>>>>>> >>>>>>>>> Snip of engine.log >>>>>>>>> >>>>>>>>> 2014-04-15 10:41:17,174 ERROR >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method >>>>>>>>> 2014-04-15 10:41:17,178 ERROR >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName >>>>>>>>> = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, >>>>>>>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: >>>>>>>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>>>>>>>> SnapshotVDS, error = Snapshot failed, code = 48 >>>>>>>>> 2014-04-15 10:41:17,531 ERROR >>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command >>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll >>>>>>>>> exception. With error message VdcBLLException: Auto-generated live >>>>>>>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with >>>>>>>>> error imageErr and code 13) >>>>>>>>> 2014-04-15 10:41:17,534 ERROR >>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: >>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>> Sent: Tuesday, April 15, 2014 10:37:41 AM >>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>> >>>>>>>>> ok. >>>>>>>>> so the vm was down and you deleted all the snapshots. >>>>>>>>> in the webadmin, when you select the disk you want to live migrate, is >>>>>>>>> it the same storage reported in vdsm? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Dafna >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/15/2014 03:25 PM, Maurice James wrote: >>>>>>>>>> I restarted from the VM. The VM now has 0 snapshots. All of the >>>>>>>>>> snapshots in the screen shots never completed, so I deleted them while >>>>>>>>>> the VM was powerd off from the ui >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>> Sent: Tuesday, April 15, 2014 10:21:06 AM >>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>> >>>>>>>>>> did you restart the vm from the vm internally or stop -> start the vm? >>>>>>>>>> >>>>>>>>>> how many snapshots does the vm have? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Dafna >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/15/2014 03:16 PM, Maurice James wrote: >>>>>>>>>>> I restarted the VM. Same issue. I was able to find the physical >>>>>>>>>>> location on disk. I attached 2 screen shots showing the location of >>>>>>>>>>> the disks on the file system. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:12:56 AM >>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>> >>>>>>>>>>> it seems that the issue is the links between the engine and the vdsm >>>>>>>>>>> >>>>>>>>>>> why don't you check where the disk is physically located and than try >>>>>>>>>>> to >>>>>>>>>>> restart the vm? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Dafna >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/15/2014 02:57 PM, Maurice James wrote: >>>>>>>>>>>> yes I did see a warning about restarting. It said: >>>>>>>>>>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'. >>>>>>>>>>>> VM restart is recommended >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM >>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>> >>>>>>>>>>>> Hi Maurice, >>>>>>>>>>>> >>>>>>>>>>>> are you getting any errors/alert about restarting the vm in the event >>>>>>>>>>>> log (in the webadmin). >>>>>>>>>>>> >>>>>>>>>>>> I believe that you have a mismatch in the directory link between the >>>>>>>>>>>> db >>>>>>>>>>>> and vdsm which was created when you tried to live migrate and failed >>>>>>>>>>>> the >>>>>>>>>>>> fist time). >>>>>>>>>>>> >>>>>>>>>>>> we can see in the vdsm log that createVolume (which is part of the >>>>>>>>>>>> live >>>>>>>>>>>> migration) is failing with this error: >>>>>>>>>>>> >>>>>>>>>>>> OSError: [Errno 2] No such file or directory: >>>>>>>>>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b' >>>>>>>>>>>> >>>>>>>>>>>> so it seems that the engine is pointing to a link that does not exist >>>>>>>>>>>> and we need to find on what storage the disk actually exists on and >>>>>>>>>>>> where the engine is pointing to. >>>>>>>>>>>> >>>>>>>>>>>> you can look at the webadmin at the disk and see where it's suppose to >>>>>>>>>>>> be and than look in /rhev and see where the disk link is in vdsm (if >>>>>>>>>>>> you >>>>>>>>>>>> can see where the disk is actually is - physically in the storage, >>>>>>>>>>>> that >>>>>>>>>>>> would be good too). >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Dafna >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/15/2014 01:16 PM, Maurice James wrote: >>>>>>>>>>>>> As far as the disk creation , I got that sorted out. It seem that my >>>>>>>>>>>>> vdsm versions were out of sync on my hosts. When I updated them the >>>>>>>>>>>>> disk creation began working, but im still having issues with live >>>>>>>>>>>>> disk migrations. >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM >>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>> >>>>>>>>>>>>> Also, before the task fails, there are ERRORs in the log which are >>>>>>>>>>>>> reporting problems in the DC >>>>>>>>>>>>> >>>>>>>>>>>>> 2014-04-14 09:56:05,161 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-16) Command >>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>> connect timed out >>>>>>>>>>>>> 2014-04-14 09:56:10,226 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-49) Command >>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>> connect timed out >>>>>>>>>>>>> 2014-04-14 09:56:15,283 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-81) Command >>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>> connect timed out >>>>>>>>>>>>> 2014-04-14 09:56:20,342 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-58) Command >>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>> connect timed out >>>>>>>>>>>>> 2014-04-14 09:56:25,409 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-84) Command >>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>> connect timed out >>>>>>>>>>>>> 2014-04-14 09:56:30,542 INFO >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-91) START, >>>>>>>>>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: >>>>>>>>>>>>> 30f312c >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> than you stop the vm and fail the AddDisk. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> is there a chance you were having problems with your storage? >>>>>>>>>>>>> >>>>>>>>>>>>> Dafna >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 04/14/2014 04:04 PM, Maurice James wrote: >>>>>>>>>>>>>> Snapshot creation seems to be failing as well >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt;, "Federico Simoncelli" >>>>>>>>>>>>>> <fsimonce(a)redhat.com&gt; >>>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM >>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Federico, >>>>>>>>>>>>>> Can you please take a look? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM >>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Logs attached >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM >>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> IMHO not enough info is provided, >>>>>>>>>>>>>>> Can you please provide full engine.log and relevant vdsm.log? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> THanks, >>>>>>>>>>>>>>> Yair >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM >>>>>>>>>>>>>>>> Subject: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> oVirt Engine Version: >>>>>>>>>>>>>>>> 3.4.1-0.0.master.20140412010845.git43746c6.el6 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> While attempting to create a disk on an NFS storage domain, it >>>>>>>>>>>>>>>> fails with >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> following error in the engine.log >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2014-04-14 09:58:12,127 ERROR >>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in >>>>>>>>>>>>>>>> HSMGetAllTasksStatusesVDS >>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>> 2014-04-14 09:58:12,139 ERROR >>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) >>>>>>>>>>>>>>>> BaseAsyncTask::LogEndTaskFailure: Task >>>>>>>>>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, >>>>>>>>>>>>>>>> Parameters >>>>>>>>>>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) >>>>>>>>>>>>>>>> ended >>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>> 2014-04-14 09:58:12,159 ERROR >>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddDiskCommand] >>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with >>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand >>>>>>>>>>>>>>>> 2014-04-14 09:58:12,212 ERROR >>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand] >>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with >>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> -- >>>>>> Dafna Ron >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>