Re: [ovirt-users] Error creating Disks

I ran "tail -f /var/log/vdsm/vdsm.log |grep ERROR" while attempting a live migration and nothing is coming up, but "tail -f /var/log/ovirt-engine/engine.log |grep ERROR" returns: 2014-04-16 11:02:59,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Failed in SnapshotVDS method 2014-04-16 11:02:59,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48 2014-04-16 11:02:59,966 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13) 2014-04-16 11:02:59,970 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler ----- Original Message ----- From: "Dafna Ron" <dron(a)redhat.com&gt; To: "Maurice James" <mjames(a)media-node.com&gt;, "elad Ben Aharon" <ebenahar(a)redhat.com&gt; Cc: "Liron Aravot" <laravot(a)redhat.com&gt;, users(a)ovirt.org Sent: Wednesday, April 16, 2014 11:00:47 AM Subject: Re: [ovirt-users] Error creating Disks Since LiveStorageMigration is a complicated and long task which divides into 4 different jobs I asked for the log with the echo before the task so we can easily follow the task from start to end. however, I did grep for the task id in the logs you just attached and there is nothing there. I'm adding Elad to try and see if he can reproduce ERROR populated by engine because of cache. Dafna On 04/16/2014 03:45 PM, Maurice James wrote: > I attached a few of the rotated logs, something might be in there > > ----- Original Message ----- > From: "Dafna Ron" <dron(a)redhat.com&gt; > To: "Liron Aravot" <laravot(a)redhat.com&gt; > Cc: "Maurice James" <mjames(a)media-node.com&gt;, users(a)ovirt.org > Sent: Wednesday, April 16, 2014 10:41:59 AM > Subject: Re: [ovirt-users] Error creating Disks > > Thanks Liron, > > well, unless the first vdsm log was cut we were not seeing any errors in > the vdsm log (engine was reporting an issue and vdsm did not show any > errors at all), after the engine restart we can see an error in the vdsm > log. > Maurice, was is possible that the vdsm log was cut? if not, there might > be a second bug with engine cache. > > The Error we are seeing in the vdsm log now does indeed look like the > live snapshot issue in qemu :) > > > Thanks, > Dafna > > > > On 04/16/2014 03:23 PM, Liron Aravot wrote: >> Hi Maurice, Dafna >> The creation of the snapshot for each of the disks succeeds, but performing the "live" part of it fails - we can see the following error in the vdsm log. >> >> Thread-4444::DEBUG::2014-04-16 09:40:39,071::vm::4007::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Snapshot faile >> d using the quiesce flag, trying again without it (unsupported configuration: reuse is not supported with this QEMU binary) >> Thread-4444::DEBUG::2014-04-16 09:40:39,081::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 67 edom: 10 level: >> 2 message: unsupported configuration: reuse is not supported with this QEMU binary >> Thread-4444::ERROR::2014-04-16 09:40:39,082::vm::4011::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Unable to take >> snapshot >> Traceback (most recent call last): >> File "/usr/share/vdsm/vm.py", line 4009, in snapshot >> self._dom.snapshotCreateXML(snapxml, snapFlags) >> File "/usr/share/vdsm/vm.py", line 859, in f >> ret = attr(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper >> ret = f(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML >> if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self) >> libvirtError: unsupported configuration: reuse is not supported with this QEMU binary >> Thread-4444::DEBUG::2014-04-16 09:40:39,091::BindingXMLRPC::1074::vds::(wrapper) return vmSnapshot with {'status': {'message': 'Snap >> shot failed', 'code': 48}} >> >> there's open bug on possibly the same issue - but let's verify that it's the same >> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 >> >> what OS are you running? if it's not centos, please try to upgrade libvirt and try again. >> >> if it's urgent - as i see that you already stopped your vm during those tries, as a temporary solution you can stop the vm, move the disk while it's stopped (which won't be live storage migration) and than start it. >> >> ----- Original Message ----- >>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>> To: "Maurice James" <mjames(a)media-node.com&gt; >>> Cc: users(a)ovirt.org >>> Sent: Wednesday, April 16, 2014 4:46:10 PM >>> Subject: Re: [ovirt-users] Error creating Disks >>> >>> can you try to restart the engine? >>> This should clean the cache and some of the tables holding temporary >>> task info. >>> >>> Dafna >>> >>> >>> >>> On 04/16/2014 02:37 PM, Maurice James wrote: >>>> I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What should >>>> my next step be? >>>> >>>> ----- Original Message ----- >>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>> Sent: Wednesday, April 16, 2014 4:44:48 AM >>>> Subject: Re: [ovirt-users] Error creating Disks >>>> >>>> ok... Now I am starting to understand what's going on and if you look at >>>> the vdsm log, the live snapshot succeeds and no other ERROR are >>>> reported. I think it's related to the task management on engine side. >>>> >>>> Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo >>>> If you have tasks we would have to stop and clear them (vdsClient -s 0 >>>> stopTask <task> ; vdsClient -s 0 clearTask <task>) >>>> after you clear the tasks you will have to restart the engine >>>> >>>> Also, what version of ovirt are you using? is it 3.4? because this was >>>> suppose to be fixed... >>>> >>>> here is the explanation from what I see in the logs: >>>> >>>> you are sending a command to Live Migrate >>>> >>>> 2014-04-15 12:50:54,381 INFO >>>> [org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4) >>>> [4c089392] Running command: MoveDisksCommand internal: false. Entities >>>> affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk >>>> 2014-04-15 12:50:54,520 INFO >>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>> (ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock >>>> [exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK >>>> , sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM >>>> ] >>>> >>>> The fist step is creating the snapshot: >>>> >>>> 2014-04-15 12:50:54,734 INFO >>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>> (org.ovirt.thread.pool-6-thread-47) Running command: >>>> CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID: >>>> ba49605b-fb7e-4a70-a3 >>>> 80-6286d3903e50 Type: VM >>>> 2014-04-15 12:50:54,748 INFO >>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] >>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command: >>>> CreateSnapshotCommand internal: true. Entities affected : ID: >>>> 00000000-0000-0000-0000-0000000 >>>> 00000 Type: Storage >>>> 2014-04-15 12:50:54,760 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] >>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] START, >>>> CreateSnapshotVDSCommand( storagePoolId = >>>> a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor >>>> eFailoverLimit = false, storageDomainId = >>>> 3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId = >>>> c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400, >>>> volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803, >>>> newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661, >>>> sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: f3b7d8b >>>> >>>> >>>> which actually succeeds in vdsm and reported as successful to the engine: >>>> >>>> engine: >>>> >>>> 2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] >>>> (DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task >>>> db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command LiveMigrateVmDisks, >>>> Parameters Type >>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned >>>> status finished, result 'success'. >>>> >>>> vdsm: >>>> >>>> Thread-4670::DEBUG::2014-04-15 >>>> 12:51:07,287::task::1185::TaskManager.Task::(prepare) >>>> Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished: >>>> {'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0, >>>> 'message': '1 jobs comple >>>> ted successfully', 'taskState': 'finished', 'taskResult': 'success', >>>> 'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}} >>>> >>>> >>>> now engine has to end the task: >>>> >>>> 2014-04-15 12:51:07,301 INFO >>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>> (DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending >>>> action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6): >>>> calling endAction . >>>> 2014-04-15 12:51:07,303 INFO >>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>> (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction >>>> [within thread] context: Attempting to endAction LiveMigrateVmDisks, >>>> executionIndex: 0 >>>> 2014-04-15 12:51:07,315 INFO >>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand >>>> 2014-04-15 12:51:07,319 INFO >>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand >>>> >>>> and than >>>> >>>> we see an exception: >>>> >>>> 2014-04-15 12:51:07,498 WARN >>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>> (org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to >>>> error: VdcBLLException: VdcBLLException: >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code >>>> 48). VM will still be configured to the new created snapshot >>>> 2014-04-15 12:51:07,543 INFO >>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>> (org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack: >>>> org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48) >>>> at >>>> org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) >>>> at >>>> org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) >>>> at >>>> org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) >>>> at >>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354) >>>> at >>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351) >>>> at >>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) >>>> at >>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) >>>> at >>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351) >>>> at >>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273) >>>> at >>>> org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304) >>>> at >>>> org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614) >>>> at >>>> org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560) >>>> at >>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886) >>>> at >>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151) >>>> at >>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118) >>>> at >>>> org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492) >>>> at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446) >>>> at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 04/15/2014 05:54 PM, Maurice James wrote: >>>>> Logs are attached. >>>>> >>>>> Live Migration failed >>>>> >>>>> ----- Original Message ----- >>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>> Sent: Tuesday, April 15, 2014 10:46:29 AM >>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>> >>>>> yeah, but is it the same error we see before in vdsm? :) >>>>> lets try to follow the action in both logs so we can debug it better, >>>>> can you do the following? >>>>> >>>>> 1. stop the vm and clean all snapshots. >>>>> 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >> >>>>> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >> >>>>> /var/log/vdsm/vdsm.log) >>>>> 3. start the vm >>>>> 4. try to live migrate the disk >>>>> 5. attach the full engine and vdsm logs. >>>>> >>>>> Thanks, >>>>> >>>>> Dafna >>>>> >>>>> >>>>> On 04/15/2014 03:41 PM, Maurice James wrote: >>>>>> It failed again. Its reported in the same location. >>>>>> >>>>>> Snip of engine.log >>>>>> >>>>>> 2014-04-15 10:41:17,174 ERROR >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>> (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method >>>>>> 2014-04-15 10:41:17,178 ERROR >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>> (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName >>>>>> = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, >>>>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: >>>>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>>>>> SnapshotVDS, error = Snapshot failed, code = 48 >>>>>> 2014-04-15 10:41:17,531 ERROR >>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>> (org.ovirt.thread.pool-6-thread-34) Command >>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll >>>>>> exception. With error message VdcBLLException: Auto-generated live >>>>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with >>>>>> error imageErr and code 13) >>>>>> 2014-04-15 10:41:17,534 ERROR >>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>> (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: >>>>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>> Sent: Tuesday, April 15, 2014 10:37:41 AM >>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>> >>>>>> ok. >>>>>> so the vm was down and you deleted all the snapshots. >>>>>> in the webadmin, when you select the disk you want to live migrate, is >>>>>> it the same storage reported in vdsm? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Dafna >>>>>> >>>>>> >>>>>> On 04/15/2014 03:25 PM, Maurice James wrote: >>>>>>> I restarted from the VM. The VM now has 0 snapshots. All of the >>>>>>> snapshots in the screen shots never completed, so I deleted them while >>>>>>> the VM was powerd off from the ui >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>> Sent: Tuesday, April 15, 2014 10:21:06 AM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> did you restart the vm from the vm internally or stop -> start the vm? >>>>>>> >>>>>>> how many snapshots does the vm have? >>>>>>> >>>>>>> Thanks, >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> On 04/15/2014 03:16 PM, Maurice James wrote: >>>>>>>> I restarted the VM. Same issue. I was able to find the physical >>>>>>>> location on disk. I attached 2 screen shots showing the location of >>>>>>>> the disks on the file system. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>> Sent: Tuesday, April 15, 2014 10:12:56 AM >>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>> >>>>>>>> it seems that the issue is the links between the engine and the vdsm >>>>>>>> >>>>>>>> why don't you check where the disk is physically located and than try >>>>>>>> to >>>>>>>> restart the vm? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 04/15/2014 02:57 PM, Maurice James wrote: >>>>>>>>> yes I did see a warning about restarting. It said: >>>>>>>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'. >>>>>>>>> VM restart is recommended >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM >>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>> >>>>>>>>> Hi Maurice, >>>>>>>>> >>>>>>>>> are you getting any errors/alert about restarting the vm in the event >>>>>>>>> log (in the webadmin). >>>>>>>>> >>>>>>>>> I believe that you have a mismatch in the directory link between the >>>>>>>>> db >>>>>>>>> and vdsm which was created when you tried to live migrate and failed >>>>>>>>> the >>>>>>>>> fist time). >>>>>>>>> >>>>>>>>> we can see in the vdsm log that createVolume (which is part of the >>>>>>>>> live >>>>>>>>> migration) is failing with this error: >>>>>>>>> >>>>>>>>> OSError: [Errno 2] No such file or directory: >>>>>>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b' >>>>>>>>> >>>>>>>>> so it seems that the engine is pointing to a link that does not exist >>>>>>>>> and we need to find on what storage the disk actually exists on and >>>>>>>>> where the engine is pointing to. >>>>>>>>> >>>>>>>>> you can look at the webadmin at the disk and see where it's suppose to >>>>>>>>> be and than look in /rhev and see where the disk link is in vdsm (if >>>>>>>>> you >>>>>>>>> can see where the disk is actually is - physically in the storage, >>>>>>>>> that >>>>>>>>> would be good too). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Dafna >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/15/2014 01:16 PM, Maurice James wrote: >>>>>>>>>> As far as the disk creation , I got that sorted out. It seem that my >>>>>>>>>> vdsm versions were out of sync on my hosts. When I updated them the >>>>>>>>>> disk creation began working, but im still having issues with live >>>>>>>>>> disk migrations. >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM >>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>> >>>>>>>>>> Also, before the task fails, there are ERRORs in the log which are >>>>>>>>>> reporting problems in the DC >>>>>>>>>> >>>>>>>>>> 2014-04-14 09:56:05,161 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-16) Command >>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>> connect timed out >>>>>>>>>> 2014-04-14 09:56:10,226 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-49) Command >>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>> connect timed out >>>>>>>>>> 2014-04-14 09:56:15,283 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-81) Command >>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>> connect timed out >>>>>>>>>> 2014-04-14 09:56:20,342 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-58) Command >>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>> connect timed out >>>>>>>>>> 2014-04-14 09:56:25,409 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-84) Command >>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>> connect timed out >>>>>>>>>> 2014-04-14 09:56:30,542 INFO >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>>>>>>> (DefaultQuartzScheduler_Worker-91) START, >>>>>>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId = >>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: >>>>>>>>>> 30f312c >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> than you stop the vm and fail the AddDisk. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> is there a chance you were having problems with your storage? >>>>>>>>>> >>>>>>>>>> Dafna >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/14/2014 04:04 PM, Maurice James wrote: >>>>>>>>>>> Snapshot creation seems to be failing as well >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt;, "Federico Simoncelli" >>>>>>>>>>> <fsimonce(a)redhat.com&gt; >>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM >>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>> >>>>>>>>>>> Hi Federico, >>>>>>>>>>> Can you please take a look? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM >>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>> >>>>>>>>>>>> Logs attached >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM >>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> IMHO not enough info is provided, >>>>>>>>>>>> Can you please provide full engine.log and relevant vdsm.log? >>>>>>>>>>>> >>>>>>>>>>>> THanks, >>>>>>>>>>>> Yair >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM >>>>>>>>>>>>> Subject: [ovirt-users] Error creating Disks >>>>>>>>>>>>> >>>>>>>>>>>>> oVirt Engine Version: >>>>>>>>>>>>> 3.4.1-0.0.master.20140412010845.git43746c6.el6 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> While attempting to create a disk on an NFS storage domain, it >>>>>>>>>>>>> fails with >>>>>>>>>>>>> the >>>>>>>>>>>>> following error in the engine.log >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2014-04-14 09:58:12,127 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in >>>>>>>>>>>>> HSMGetAllTasksStatusesVDS >>>>>>>>>>>>> method >>>>>>>>>>>>> 2014-04-14 09:58:12,139 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) >>>>>>>>>>>>> BaseAsyncTask::LogEndTaskFailure: Task >>>>>>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, >>>>>>>>>>>>> Parameters >>>>>>>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) >>>>>>>>>>>>> ended >>>>>>>>>>>>> with >>>>>>>>>>>>> failure: >>>>>>>>>>>>> 2014-04-14 09:58:12,159 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddDiskCommand] >>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with >>>>>>>>>>>>> failure: >>>>>>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand >>>>>>>>>>>>> 2014-04-14 09:58:12,212 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand] >>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with >>>>>>>>>>>>> failure: >>>>>>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>> -- >>> Dafna Ron >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>>