[ovirt-users] Error creating Disks
Maurice James
mjames at media-node.com
Thu Apr 17 06:55:00 EDT 2014
Sure thing
----- Original Message -----
From: "Dafna Ron" <dron at redhat.com>
To: "Maurice James" <mjames at media-node.com>
Cc: "Liron Aravot" <laravot at redhat.com>, "itamar Heim" <iheim at redhat.com>, "elad Ben Aharon" <ebenahar at redhat.com>, users at ovirt.org
Sent: Thursday, April 17, 2014 6:50:44 AM
Subject: Re: [ovirt-users] Error creating Disks
valid and important discussion indeed.
important enough I think that this should be raised in a separate thread
allowing others to discuss it (since this is an on-going issue some
people may not be aware of a subject change).
Can you please send a new mail with a relevant headline raising this issue?
Thanks,
Dafna
On 04/17/2014 11:47 AM, Maurice James wrote:
> Just curious. I have come across a few major problems since upgrading to 3.4.x (External Authentication BZ1081204, Quota assignment BZ1081014, Live migration "qemu bug" ) to name a few. I imagine that all of the features need to be polished before inclusion into RHEL. I'm trying to get my company to adopt RHEL instead of V-Sphere but that is becoming a little tricky. Minus the cool new features of 3.4.x 3.3.4 was pretty stable IMO.
>
> ----- Original Message -----
> From: "Dafna Ron" <dron at redhat.com>
> To: "Maurice James" <mjames at media-node.com>, "Liron Aravot" <laravot at redhat.com>, "itamar Heim" <iheim at redhat.com>
> Cc: "elad Ben Aharon" <ebenahar at redhat.com>, users at ovirt.org
> Sent: Thursday, April 17, 2014 5:31:46 AM
> Subject: Re: [ovirt-users] Error creating Disks
>
> I am not sure this question relates to this thread and perhaps it should
> be posted in a different one :) can you explain what you mean by that
> question?
>
>
> On 04/17/2014 01:09 AM, Maurice James wrote:
>> Which version of Ovirt are you guys going to build the new RHEV from?
>>
>> ----- Original Message -----
>> From: "Dafna Ron" <dron at redhat.com>
>> To: "Maurice James" <mjames at media-node.com>
>> Cc: "elad Ben Aharon" <ebenahar at redhat.com>, "Liron Aravot" <laravot at redhat.com>, users at ovirt.org
>> Sent: Wednesday, April 16, 2014 5:39:20 PM
>> Subject: Re: [ovirt-users] Error creating Disks
>>
>> Ok.
>> so it's the qemu bug that Liron sent and I think there is some bug there
>> with the engine cache since we did not see the job failing in vdsm log.
>> hopefully there will be a qemu patch for centos soon...
>>
>> Thanks Maurice!
>>
>> Dafna
>>
>>
>> On 04/16/2014 07:14 PM, Maurice James wrote:
>>> The offline disk migration works
>>>
>>> ----- Original Message -----
>>> From: "Dafna Ron" <dron at redhat.com>
>>> To: "Maurice James" <mjames at media-node.com>
>>> Cc: "elad Ben Aharon" <ebenahar at redhat.com>, "Liron Aravot" <laravot at redhat.com>, users at ovirt.org
>>> Sent: Wednesday, April 16, 2014 11:57:20 AM
>>> Subject: Re: [ovirt-users] Error creating Disks
>>>
>>> Thanks Maurice.
>>> so you are saying that there is some sort of caching.
>>>
>>> can you migrate the disk off line?
>>>
>>> Dafna
>>>
>>>
>>> On 04/16/2014 04:06 PM, Maurice James wrote:
>>>> I ran "tail -f /var/log/vdsm/vdsm.log |grep ERROR" while attempting a live migration and nothing is coming up, but
>>>> "tail -f /var/log/ovirt-engine/engine.log |grep ERROR" returns:
>>>> 2014-04-16 11:02:59,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Failed in SnapshotVDS method
>>>> 2014-04-16 11:02:59,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-40) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48
>>>> 2014-04-16 11:02:59,966 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13)
>>>> 2014-04-16 11:02:59,970 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-40) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Dafna Ron" <dron at redhat.com>
>>>> To: "Maurice James" <mjames at media-node.com>, "elad Ben Aharon" <ebenahar at redhat.com>
>>>> Cc: "Liron Aravot" <laravot at redhat.com>, users at ovirt.org
>>>> Sent: Wednesday, April 16, 2014 11:00:47 AM
>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>
>>>> Since LiveStorageMigration is a complicated and long task which divides
>>>> into 4 different jobs I asked for the log with the echo before the task
>>>> so we can easily follow the task from start to end.
>>>>
>>>> however, I did grep for the task id in the logs you just attached and
>>>> there is nothing there.
>>>> I'm adding Elad to try and see if he can reproduce ERROR populated by
>>>> engine because of cache.
>>>>
>>>> Dafna
>>>>
>>>>
>>>> On 04/16/2014 03:45 PM, Maurice James wrote:
>>>>> I attached a few of the rotated logs, something might be in there
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>> To: "Liron Aravot" <laravot at redhat.com>
>>>>> Cc: "Maurice James" <mjames at media-node.com>, users at ovirt.org
>>>>> Sent: Wednesday, April 16, 2014 10:41:59 AM
>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>
>>>>> Thanks Liron,
>>>>>
>>>>> well, unless the first vdsm log was cut we were not seeing any errors in
>>>>> the vdsm log (engine was reporting an issue and vdsm did not show any
>>>>> errors at all), after the engine restart we can see an error in the vdsm
>>>>> log.
>>>>> Maurice, was is possible that the vdsm log was cut? if not, there might
>>>>> be a second bug with engine cache.
>>>>>
>>>>> The Error we are seeing in the vdsm log now does indeed look like the
>>>>> live snapshot issue in qemu :)
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Dafna
>>>>>
>>>>>
>>>>>
>>>>> On 04/16/2014 03:23 PM, Liron Aravot wrote:
>>>>>> Hi Maurice, Dafna
>>>>>> The creation of the snapshot for each of the disks succeeds, but performing the "live" part of it fails - we can see the following error in the vdsm log.
>>>>>>
>>>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,071::vm::4007::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Snapshot faile
>>>>>> d using the quiesce flag, trying again without it (unsupported configuration: reuse is not supported with this QEMU binary)
>>>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,081::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 67 edom: 10 level:
>>>>>> 2 message: unsupported configuration: reuse is not supported with this QEMU binary
>>>>>> Thread-4444::ERROR::2014-04-16 09:40:39,082::vm::4011::vm.Vm::(snapshot) vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Unable to take
>>>>>> snapshot
>>>>>> Traceback (most recent call last):
>>>>>> File "/usr/share/vdsm/vm.py", line 4009, in snapshot
>>>>>> self._dom.snapshotCreateXML(snapxml, snapFlags)
>>>>>> File "/usr/share/vdsm/vm.py", line 859, in f
>>>>>> ret = attr(*args, **kwargs)
>>>>>> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper
>>>>>> ret = f(*args, **kwargs)
>>>>>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML
>>>>>> if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
>>>>>> libvirtError: unsupported configuration: reuse is not supported with this QEMU binary
>>>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,091::BindingXMLRPC::1074::vds::(wrapper) return vmSnapshot with {'status': {'message': 'Snap
>>>>>> shot failed', 'code': 48}}
>>>>>>
>>>>>> there's open bug on possibly the same issue - but let's verify that it's the same
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100
>>>>>>
>>>>>> what OS are you running? if it's not centos, please try to upgrade libvirt and try again.
>>>>>>
>>>>>> if it's urgent - as i see that you already stopped your vm during those tries, as a temporary solution you can stop the vm, move the disk while it's stopped (which won't be live storage migration) and than start it.
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>> Cc: users at ovirt.org
>>>>>>> Sent: Wednesday, April 16, 2014 4:46:10 PM
>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>
>>>>>>> can you try to restart the engine?
>>>>>>> This should clean the cache and some of the tables holding temporary
>>>>>>> task info.
>>>>>>>
>>>>>>> Dafna
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 04/16/2014 02:37 PM, Maurice James wrote:
>>>>>>>> I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What should
>>>>>>>> my next step be?
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>> Sent: Wednesday, April 16, 2014 4:44:48 AM
>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>
>>>>>>>> ok... Now I am starting to understand what's going on and if you look at
>>>>>>>> the vdsm log, the live snapshot succeeds and no other ERROR are
>>>>>>>> reported. I think it's related to the task management on engine side.
>>>>>>>>
>>>>>>>> Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo
>>>>>>>> If you have tasks we would have to stop and clear them (vdsClient -s 0
>>>>>>>> stopTask <task> ; vdsClient -s 0 clearTask <task>)
>>>>>>>> after you clear the tasks you will have to restart the engine
>>>>>>>>
>>>>>>>> Also, what version of ovirt are you using? is it 3.4? because this was
>>>>>>>> suppose to be fixed...
>>>>>>>>
>>>>>>>> here is the explanation from what I see in the logs:
>>>>>>>>
>>>>>>>> you are sending a command to Live Migrate
>>>>>>>>
>>>>>>>> 2014-04-15 12:50:54,381 INFO
>>>>>>>> [org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4)
>>>>>>>> [4c089392] Running command: MoveDisksCommand internal: false. Entities
>>>>>>>> affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk
>>>>>>>> 2014-04-15 12:50:54,520 INFO
>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
>>>>>>>> (ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock
>>>>>>>> [exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK
>>>>>>>> , sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM
>>>>>>>> ]
>>>>>>>>
>>>>>>>> The fist step is creating the snapshot:
>>>>>>>>
>>>>>>>> 2014-04-15 12:50:54,734 INFO
>>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-47) Running command:
>>>>>>>> CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID:
>>>>>>>> ba49605b-fb7e-4a70-a3
>>>>>>>> 80-6286d3903e50 Type: VM
>>>>>>>> 2014-04-15 12:50:54,748 INFO
>>>>>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command:
>>>>>>>> CreateSnapshotCommand internal: true. Entities affected : ID:
>>>>>>>> 00000000-0000-0000-0000-0000000
>>>>>>>> 00000 Type: Storage
>>>>>>>> 2014-04-15 12:50:54,760 INFO
>>>>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] START,
>>>>>>>> CreateSnapshotVDSCommand( storagePoolId =
>>>>>>>> a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor
>>>>>>>> eFailoverLimit = false, storageDomainId =
>>>>>>>> 3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId =
>>>>>>>> c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400,
>>>>>>>> volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803,
>>>>>>>> newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661,
>>>>>>>> sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: f3b7d8b
>>>>>>>>
>>>>>>>>
>>>>>>>> which actually succeeds in vdsm and reported as successful to the engine:
>>>>>>>>
>>>>>>>> engine:
>>>>>>>>
>>>>>>>> 2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
>>>>>>>> (DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task
>>>>>>>> db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command LiveMigrateVmDisks,
>>>>>>>> Parameters Type
>>>>>>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned
>>>>>>>> status finished, result 'success'.
>>>>>>>>
>>>>>>>> vdsm:
>>>>>>>>
>>>>>>>> Thread-4670::DEBUG::2014-04-15
>>>>>>>> 12:51:07,287::task::1185::TaskManager.Task::(prepare)
>>>>>>>> Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished:
>>>>>>>> {'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0,
>>>>>>>> 'message': '1 jobs comple
>>>>>>>> ted successfully', 'taskState': 'finished', 'taskResult': 'success',
>>>>>>>> 'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}}
>>>>>>>>
>>>>>>>>
>>>>>>>> now engine has to end the task:
>>>>>>>>
>>>>>>>> 2014-04-15 12:51:07,301 INFO
>>>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask]
>>>>>>>> (DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending
>>>>>>>> action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6):
>>>>>>>> calling endAction .
>>>>>>>> 2014-04-15 12:51:07,303 INFO
>>>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask]
>>>>>>>> (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction
>>>>>>>> [within thread] context: Attempting to endAction LiveMigrateVmDisks,
>>>>>>>> executionIndex: 0
>>>>>>>> 2014-04-15 12:51:07,315 INFO
>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully:
>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand
>>>>>>>> 2014-04-15 12:51:07,319 INFO
>>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully:
>>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand
>>>>>>>>
>>>>>>>> and than
>>>>>>>>
>>>>>>>> we see an exception:
>>>>>>>>
>>>>>>>> 2014-04-15 12:51:07,498 WARN
>>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to
>>>>>>>> error: VdcBLLException: VdcBLLException:
>>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
>>>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error =
>>>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code
>>>>>>>> 48). VM will still be configured to the new created snapshot
>>>>>>>> 2014-04-15 12:51:07,543 INFO
>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>>>>> (org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack:
>>>>>>>> org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException:
>>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
>>>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error =
>>>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118)
>>>>>>>> at
>>>>>>>> org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492)
>>>>>>>> at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 04/15/2014 05:54 PM, Maurice James wrote:
>>>>>>>>> Logs are attached.
>>>>>>>>>
>>>>>>>>> Live Migration failed
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>> Sent: Tuesday, April 15, 2014 10:46:29 AM
>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>
>>>>>>>>> yeah, but is it the same error we see before in vdsm? :)
>>>>>>>>> lets try to follow the action in both logs so we can debug it better,
>>>>>>>>> can you do the following?
>>>>>>>>>
>>>>>>>>> 1. stop the vm and clean all snapshots.
>>>>>>>>> 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >>
>>>>>>>>> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >>
>>>>>>>>> /var/log/vdsm/vdsm.log)
>>>>>>>>> 3. start the vm
>>>>>>>>> 4. try to live migrate the disk
>>>>>>>>> 5. attach the full engine and vdsm logs.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Dafna
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 04/15/2014 03:41 PM, Maurice James wrote:
>>>>>>>>>> It failed again. Its reported in the same location.
>>>>>>>>>>
>>>>>>>>>> Snip of engine.log
>>>>>>>>>>
>>>>>>>>>> 2014-04-15 10:41:17,174 ERROR
>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method
>>>>>>>>>> 2014-04-15 10:41:17,178 ERROR
>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName
>>>>>>>>>> = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195,
>>>>>>>>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception:
>>>>>>>>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
>>>>>>>>>> SnapshotVDS, error = Snapshot failed, code = 48
>>>>>>>>>> 2014-04-15 10:41:17,531 ERROR
>>>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
>>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command
>>>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll
>>>>>>>>>> exception. With error message VdcBLLException: Auto-generated live
>>>>>>>>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with
>>>>>>>>>> error imageErr and code 13)
>>>>>>>>>> 2014-04-15 10:41:17,534 ERROR
>>>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
>>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler:
>>>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:37:41 AM
>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>
>>>>>>>>>> ok.
>>>>>>>>>> so the vm was down and you deleted all the snapshots.
>>>>>>>>>> in the webadmin, when you select the disk you want to live migrate, is
>>>>>>>>>> it the same storage reported in vdsm?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Dafna
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 04/15/2014 03:25 PM, Maurice James wrote:
>>>>>>>>>>> I restarted from the VM. The VM now has 0 snapshots. All of the
>>>>>>>>>>> snapshots in the screen shots never completed, so I deleted them while
>>>>>>>>>>> the VM was powerd off from the ui
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:21:06 AM
>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>
>>>>>>>>>>> did you restart the vm from the vm internally or stop -> start the vm?
>>>>>>>>>>>
>>>>>>>>>>> how many snapshots does the vm have?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Dafna
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 04/15/2014 03:16 PM, Maurice James wrote:
>>>>>>>>>>>> I restarted the VM. Same issue. I was able to find the physical
>>>>>>>>>>>> location on disk. I attached 2 screen shots showing the location of
>>>>>>>>>>>> the disks on the file system.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:12:56 AM
>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>
>>>>>>>>>>>> it seems that the issue is the links between the engine and the vdsm
>>>>>>>>>>>>
>>>>>>>>>>>> why don't you check where the disk is physically located and than try
>>>>>>>>>>>> to
>>>>>>>>>>>> restart the vm?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Dafna
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 04/15/2014 02:57 PM, Maurice James wrote:
>>>>>>>>>>>>> yes I did see a warning about restarting. It said:
>>>>>>>>>>>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'.
>>>>>>>>>>>>> VM restart is recommended
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM
>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Maurice,
>>>>>>>>>>>>>
>>>>>>>>>>>>> are you getting any errors/alert about restarting the vm in the event
>>>>>>>>>>>>> log (in the webadmin).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe that you have a mismatch in the directory link between the
>>>>>>>>>>>>> db
>>>>>>>>>>>>> and vdsm which was created when you tried to live migrate and failed
>>>>>>>>>>>>> the
>>>>>>>>>>>>> fist time).
>>>>>>>>>>>>>
>>>>>>>>>>>>> we can see in the vdsm log that createVolume (which is part of the
>>>>>>>>>>>>> live
>>>>>>>>>>>>> migration) is failing with this error:
>>>>>>>>>>>>>
>>>>>>>>>>>>> OSError: [Errno 2] No such file or directory:
>>>>>>>>>>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b'
>>>>>>>>>>>>>
>>>>>>>>>>>>> so it seems that the engine is pointing to a link that does not exist
>>>>>>>>>>>>> and we need to find on what storage the disk actually exists on and
>>>>>>>>>>>>> where the engine is pointing to.
>>>>>>>>>>>>>
>>>>>>>>>>>>> you can look at the webadmin at the disk and see where it's suppose to
>>>>>>>>>>>>> be and than look in /rhev and see where the disk link is in vdsm (if
>>>>>>>>>>>>> you
>>>>>>>>>>>>> can see where the disk is actually is - physically in the storage,
>>>>>>>>>>>>> that
>>>>>>>>>>>>> would be good too).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dafna
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 04/15/2014 01:16 PM, Maurice James wrote:
>>>>>>>>>>>>>> As far as the disk creation , I got that sorted out. It seem that my
>>>>>>>>>>>>>> vdsm versions were out of sync on my hosts. When I updated them the
>>>>>>>>>>>>>> disk creation began working, but im still having issues with live
>>>>>>>>>>>>>> disk migrations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM
>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, before the task fails, there are ERRORs in the log which are
>>>>>>>>>>>>>> reporting problems in the DC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2014-04-14 09:56:05,161 ERROR
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-16) Command
>>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>>>>>>>>>>> failed. Exception: VDSNetworkException:
>>>>>>>>>>>>>> java.net.SocketTimeoutException:
>>>>>>>>>>>>>> connect timed out
>>>>>>>>>>>>>> 2014-04-14 09:56:10,226 ERROR
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-49) Command
>>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>>>>>>>>>>> failed. Exception: VDSNetworkException:
>>>>>>>>>>>>>> java.net.SocketTimeoutException:
>>>>>>>>>>>>>> connect timed out
>>>>>>>>>>>>>> 2014-04-14 09:56:15,283 ERROR
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-81) Command
>>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>>>>>>>>>>> failed. Exception: VDSNetworkException:
>>>>>>>>>>>>>> java.net.SocketTimeoutException:
>>>>>>>>>>>>>> connect timed out
>>>>>>>>>>>>>> 2014-04-14 09:56:20,342 ERROR
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-58) Command
>>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>>>>>>>>>>> failed. Exception: VDSNetworkException:
>>>>>>>>>>>>>> java.net.SocketTimeoutException:
>>>>>>>>>>>>>> connect timed out
>>>>>>>>>>>>>> 2014-04-14 09:56:25,409 ERROR
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-84) Command
>>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>>>>>>>>>>> failed. Exception: VDSNetworkException:
>>>>>>>>>>>>>> java.net.SocketTimeoutException:
>>>>>>>>>>>>>> connect timed out
>>>>>>>>>>>>>> 2014-04-14 09:56:30,542 INFO
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-91) START,
>>>>>>>>>>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId =
>>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id:
>>>>>>>>>>>>>> 30f312c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> than you stop the vm and fail the AddDisk.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> is there a chance you were having problems with your storage?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dafna
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 04/14/2014 04:04 PM, Maurice James wrote:
>>>>>>>>>>>>>>> Snapshot creation seems to be failing as well
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>, "Federico Simoncelli"
>>>>>>>>>>>>>>> <fsimonce at redhat.com>
>>>>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM
>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Federico,
>>>>>>>>>>>>>>> Can you please take a look?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>> From: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>>>>>> To: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM
>>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Logs attached
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>> From: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>>>>>>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM
>>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> IMHO not enough info is provided,
>>>>>>>>>>>>>>>> Can you please provide full engine.log and relevant vdsm.log?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> THanks,
>>>>>>>>>>>>>>>> Yair
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>>> From: "Maurice James" <mjames at media-node.com>
>>>>>>>>>>>>>>>>> To: users at ovirt.org
>>>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM
>>>>>>>>>>>>>>>>> Subject: [ovirt-users] Error creating Disks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> oVirt Engine Version:
>>>>>>>>>>>>>>>>> 3.4.1-0.0.master.20140412010845.git43746c6.el6
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> While attempting to create a disk on an NFS storage domain, it
>>>>>>>>>>>>>>>>> fails with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> following error in the engine.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,127 ERROR
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in
>>>>>>>>>>>>>>>>> HSMGetAllTasksStatusesVDS
>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,139 ERROR
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.SPMAsyncTask]
>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72)
>>>>>>>>>>>>>>>>> BaseAsyncTask::LogEndTaskFailure: Task
>>>>>>>>>>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk,
>>>>>>>>>>>>>>>>> Parameters
>>>>>>>>>>>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters)
>>>>>>>>>>>>>>>>> ended
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> failure:
>>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,159 ERROR
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddDiskCommand]
>>>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with
>>>>>>>>>>>>>>>>> failure:
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand
>>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,212 ERROR
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand]
>>>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with
>>>>>>>>>>>>>>>>> failure:
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>> --
>>>>>>> Dafna Ron
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>
--
Dafna Ron
More information about the Users
mailing list