[ovirt-users] Error creating Disks
Maurice James
mjames at media-node.com
Tue Apr 15 16:54:11 UTC 2014
Logs are attached.
Live Migration failed
----- Original Message -----
From: "Dafna Ron" <dron at redhat.com>
To: "Maurice James" <mjames at media-node.com>
Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
Sent: Tuesday, April 15, 2014 10:46:29 AM
Subject: Re: [ovirt-users] Error creating Disks
yeah, but is it the same error we see before in vdsm? :)
lets try to follow the action in both logs so we can debug it better,
can you do the following?
1. stop the vm and clean all snapshots.
2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >>
/var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >>
/var/log/vdsm/vdsm.log)
3. start the vm
4. try to live migrate the disk
5. attach the full engine and vdsm logs.
Thanks,
Dafna
On 04/15/2014 03:41 PM, Maurice James wrote:
> It failed again. Its reported in the same location.
>
> Snip of engine.log
>
> 2014-04-15 10:41:17,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method
> 2014-04-15 10:41:17,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48
> 2014-04-15 10:41:17,531 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-34) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13)
> 2014-04-15 10:41:17,534 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler
>
>
>
>
> ----- Original Message -----
> From: "Dafna Ron" <dron at redhat.com>
> To: "Maurice James" <mjames at media-node.com>
> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
> Sent: Tuesday, April 15, 2014 10:37:41 AM
> Subject: Re: [ovirt-users] Error creating Disks
>
> ok.
> so the vm was down and you deleted all the snapshots.
> in the webadmin, when you select the disk you want to live migrate, is
> it the same storage reported in vdsm?
>
> Thanks,
>
> Dafna
>
>
> On 04/15/2014 03:25 PM, Maurice James wrote:
>> I restarted from the VM. The VM now has 0 snapshots. All of the snapshots in the screen shots never completed, so I deleted them while the VM was powerd off from the ui
>>
>> ----- Original Message -----
>> From: "Dafna Ron" <dron at redhat.com>
>> To: "Maurice James" <mjames at media-node.com>
>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>> Sent: Tuesday, April 15, 2014 10:21:06 AM
>> Subject: Re: [ovirt-users] Error creating Disks
>>
>> did you restart the vm from the vm internally or stop -> start the vm?
>>
>> how many snapshots does the vm have?
>>
>> Thanks,
>> Dafna
>>
>>
>> On 04/15/2014 03:16 PM, Maurice James wrote:
>>> I restarted the VM. Same issue. I was able to find the physical location on disk. I attached 2 screen shots showing the location of the disks on the file system.
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Dafna Ron" <dron at redhat.com>
>>> To: "Maurice James" <mjames at media-node.com>
>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>> Sent: Tuesday, April 15, 2014 10:12:56 AM
>>> Subject: Re: [ovirt-users] Error creating Disks
>>>
>>> it seems that the issue is the links between the engine and the vdsm
>>>
>>> why don't you check where the disk is physically located and than try to
>>> restart the vm?
>>>
>>> Thanks,
>>> Dafna
>>>
>>>
>>>
>>> On 04/15/2014 02:57 PM, Maurice James wrote:
>>>> yes I did see a warning about restarting. It said:
>>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'. VM restart is recommended
>>>>
>>>> ----- Original Message -----
>>>> From: "Dafna Ron" <dron at redhat.com>
>>>> To: "Maurice James" <mjames at media-node.com>
>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM
>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>
>>>> Hi Maurice,
>>>>
>>>> are you getting any errors/alert about restarting the vm in the event
>>>> log (in the webadmin).
>>>>
>>>> I believe that you have a mismatch in the directory link between the db
>>>> and vdsm which was created when you tried to live migrate and failed the
>>>> fist time).
>>>>
>>>> we can see in the vdsm log that createVolume (which is part of the live
>>>> migration) is failing with this error:
>>>>
>>>> OSError: [Errno 2] No such file or directory:
>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b'
>>>>
>>>> so it seems that the engine is pointing to a link that does not exist
>>>> and we need to find on what storage the disk actually exists on and
>>>> where the engine is pointing to.
>>>>
>>>> you can look at the webadmin at the disk and see where it's suppose to
>>>> be and than look in /rhev and see where the disk link is in vdsm (if you
>>>> can see where the disk is actually is - physically in the storage, that
>>>> would be good too).
>>>>
>>>> Thanks,
>>>>
>>>> Dafna
>>>>
>>>>
>>>> On 04/15/2014 01:16 PM, Maurice James wrote:
>>>>> As far as the disk creation , I got that sorted out. It seem that my vdsm versions were out of sync on my hosts. When I updated them the disk creation began working, but im still having issues with live disk migrations.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Dafna Ron" <dron at redhat.com>
>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>> Cc: "Yair Zaslavsky" <yzaslavs at redhat.com>, users at ovirt.org
>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM
>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>
>>>>> Also, before the task fails, there are ERRORs in the log which are
>>>>> reporting problems in the DC
>>>>>
>>>>> 2014-04-14 09:56:05,161 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-16) Command
>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException:
>>>>> connect timed out
>>>>> 2014-04-14 09:56:10,226 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-49) Command
>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException:
>>>>> connect timed out
>>>>> 2014-04-14 09:56:15,283 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-81) Command
>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException:
>>>>> connect timed out
>>>>> 2014-04-14 09:56:20,342 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-58) Command
>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException:
>>>>> connect timed out
>>>>> 2014-04-14 09:56:25,409 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-84) Command
>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
>>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException:
>>>>> connect timed out
>>>>> 2014-04-14 09:56:30,542 INFO
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>>>> (DefaultQuartzScheduler_Worker-91) START,
>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId =
>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: 30f312c
>>>>>
>>>>>
>>>>> than you stop the vm and fail the AddDisk.
>>>>>
>>>>>
>>>>> is there a chance you were having problems with your storage?
>>>>>
>>>>> Dafna
>>>>>
>>>>>
>>>>> On 04/14/2014 04:04 PM, Maurice James wrote:
>>>>>> Snapshot creation seems to be failing as well
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>> To: "Maurice James" <mjames at media-node.com>, "Federico Simoncelli" <fsimonce at redhat.com>
>>>>>> Cc: users at ovirt.org
>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM
>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>
>>>>>> Hi Federico,
>>>>>> Can you please take a look?
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Maurice James" <mjames at media-node.com>
>>>>>>> To: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>>> Cc: users at ovirt.org
>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM
>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>
>>>>>>> Logs attached
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Yair Zaslavsky" <yzaslavs at redhat.com>
>>>>>>> To: "Maurice James" <mjames at media-node.com>
>>>>>>> Cc: users at ovirt.org
>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM
>>>>>>> Subject: Re: [ovirt-users] Error creating Disks
>>>>>>>
>>>>>>> Hi,
>>>>>>> IMHO not enough info is provided,
>>>>>>> Can you please provide full engine.log and relevant vdsm.log?
>>>>>>>
>>>>>>> THanks,
>>>>>>> Yair
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Maurice James" <mjames at media-node.com>
>>>>>>>> To: users at ovirt.org
>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM
>>>>>>>> Subject: [ovirt-users] Error creating Disks
>>>>>>>>
>>>>>>>> oVirt Engine Version: 3.4.1-0.0.master.20140412010845.git43746c6.el6
>>>>>>>>
>>>>>>>>
>>>>>>>> While attempting to create a disk on an NFS storage domain, it fails with
>>>>>>>> the
>>>>>>>> following error in the engine.log
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-04-14 09:58:12,127 ERROR
>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in HSMGetAllTasksStatusesVDS
>>>>>>>> method
>>>>>>>> 2014-04-14 09:58:12,139 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask]
>>>>>>>> (DefaultQuartzScheduler_Worker-72) BaseAsyncTask::LogEndTaskFailure: Task
>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, Parameters
>>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended
>>>>>>>> with
>>>>>>>> failure:
>>>>>>>> 2014-04-14 09:58:12,159 ERROR [org.ovirt.engine.core.bll.AddDiskCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with failure:
>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand
>>>>>>>> 2014-04-14 09:58:12,212 ERROR
>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand]
>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with failure:
>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>
--
Dafna Ron
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm(1).zip
Type: application/zip
Size: 674924 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140415/f77f747e/attachment-0002.zip>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine(1).zip
Type: application/zip
Size: 512457 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140415/f77f747e/attachment-0003.zip>
More information about the Users
mailing list