Re: [ovirt-users] Error creating Disks

Logs are attached. Live Migration failed ----- Original Message ----- From: "Dafna Ron" <dron(a)redhat.com&gt; To: "Maurice James" <mjames(a)media-node.com&gt; Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org Sent: Tuesday, April 15, 2014 10:46:29 AM Subject: Re: [ovirt-users] Error creating Disks yeah, but is it the same error we see before in vdsm? :) lets try to follow the action in both logs so we can debug it better, can you do the following? 1. stop the vm and clean all snapshots. 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >> /var/log/vdsm/vdsm.log) 3. start the vm 4. try to live migrate the disk 5. attach the full engine and vdsm logs. Thanks, Dafna On 04/15/2014 03:41 PM, Maurice James wrote: > It failed again. Its reported in the same location. > > Snip of engine.log > > 2014-04-15 10:41:17,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method > 2014-04-15 10:41:17,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = Snapshot failed, code = 48 > 2014-04-15 10:41:17,531 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-34) Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll exception. With error message VdcBLLException: Auto-generated live snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with error imageErr and code 13) > 2014-04-15 10:41:17,534 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler > > > > > ----- Original Message ----- > From: "Dafna Ron" <dron(a)redhat.com&gt; > To: "Maurice James" <mjames(a)media-node.com&gt; > Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org > Sent: Tuesday, April 15, 2014 10:37:41 AM > Subject: Re: [ovirt-users] Error creating Disks > > ok. > so the vm was down and you deleted all the snapshots. > in the webadmin, when you select the disk you want to live migrate, is > it the same storage reported in vdsm? > > Thanks, > > Dafna > > > On 04/15/2014 03:25 PM, Maurice James wrote: >> I restarted from the VM. The VM now has 0 snapshots. All of the snapshots in the screen shots never completed, so I deleted them while the VM was powerd off from the ui >> >> ----- Original Message ----- >> From: "Dafna Ron" <dron(a)redhat.com&gt; >> To: "Maurice James" <mjames(a)media-node.com&gt; >> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >> Sent: Tuesday, April 15, 2014 10:21:06 AM >> Subject: Re: [ovirt-users] Error creating Disks >> >> did you restart the vm from the vm internally or stop -> start the vm? >> >> how many snapshots does the vm have? >> >> Thanks, >> Dafna >> >> >> On 04/15/2014 03:16 PM, Maurice James wrote: >>> I restarted the VM. Same issue. I was able to find the physical location on disk. I attached 2 screen shots showing the location of the disks on the file system. >>> >>> >>> >>> ----- Original Message ----- >>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>> To: "Maurice James" <mjames(a)media-node.com&gt; >>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>> Sent: Tuesday, April 15, 2014 10:12:56 AM >>> Subject: Re: [ovirt-users] Error creating Disks >>> >>> it seems that the issue is the links between the engine and the vdsm >>> >>> why don't you check where the disk is physically located and than try to >>> restart the vm? >>> >>> Thanks, >>> Dafna >>> >>> >>> >>> On 04/15/2014 02:57 PM, Maurice James wrote: >>>> yes I did see a warning about restarting. It said: >>>> Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'. VM restart is recommended >>>> >>>> ----- Original Message ----- >>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>> Sent: Tuesday, April 15, 2014 9:49:22 AM >>>> Subject: Re: [ovirt-users] Error creating Disks >>>> >>>> Hi Maurice, >>>> >>>> are you getting any errors/alert about restarting the vm in the event >>>> log (in the webadmin). >>>> >>>> I believe that you have a mismatch in the directory link between the db >>>> and vdsm which was created when you tried to live migrate and failed the >>>> fist time). >>>> >>>> we can see in the vdsm log that createVolume (which is part of the live >>>> migration) is failing with this error: >>>> >>>> OSError: [Errno 2] No such file or directory: >>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b' >>>> >>>> so it seems that the engine is pointing to a link that does not exist >>>> and we need to find on what storage the disk actually exists on and >>>> where the engine is pointing to. >>>> >>>> you can look at the webadmin at the disk and see where it's suppose to >>>> be and than look in /rhev and see where the disk link is in vdsm (if you >>>> can see where the disk is actually is - physically in the storage, that >>>> would be good too). >>>> >>>> Thanks, >>>> >>>> Dafna >>>> >>>> >>>> On 04/15/2014 01:16 PM, Maurice James wrote: >>>>> As far as the disk creation , I got that sorted out. It seem that my vdsm versions were out of sync on my hosts. When I updated them the disk creation began working, but im still having issues with live disk migrations. >>>>> >>>>> ----- Original Message ----- >>>>> From: "Dafna Ron" <dron(a)redhat.com&gt; >>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org >>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM >>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>> >>>>> Also, before the task fails, there are ERRORs in the log which are >>>>> reporting problems in the DC >>>>> >>>>> 2014-04-14 09:56:05,161 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-16) Command >>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: >>>>> connect timed out >>>>> 2014-04-14 09:56:10,226 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-49) Command >>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: >>>>> connect timed out >>>>> 2014-04-14 09:56:15,283 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-81) Command >>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: >>>>> connect timed out >>>>> 2014-04-14 09:56:20,342 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-58) Command >>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: >>>>> connect timed out >>>>> 2014-04-14 09:56:25,409 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-84) Command >>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>> failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: >>>>> connect timed out >>>>> 2014-04-14 09:56:30,542 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>> (DefaultQuartzScheduler_Worker-91) START, >>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId = >>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: 30f312c >>>>> >>>>> >>>>> than you stop the vm and fail the AddDisk. >>>>> >>>>> >>>>> is there a chance you were having problems with your storage? >>>>> >>>>> Dafna >>>>> >>>>> >>>>> On 04/14/2014 04:04 PM, Maurice James wrote: >>>>>> Snapshot creation seems to be failing as well >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>> To: "Maurice James" <mjames(a)media-node.com&gt;, "Federico Simoncelli" <fsimonce(a)redhat.com&gt; >>>>>> Cc: users(a)ovirt.org >>>>>> Sent: Monday, April 14, 2014 11:00:55 AM >>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>> >>>>>> Hi Federico, >>>>>> Can you please take a look? >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>> Cc: users(a)ovirt.org >>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> Logs attached >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>>> To: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>> Cc: users(a)ovirt.org >>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> Hi, >>>>>>> IMHO not enough info is provided, >>>>>>> Can you please provide full engine.log and relevant vdsm.log? >>>>>>> >>>>>>> THanks, >>>>>>> Yair >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Maurice James" <mjames(a)media-node.com&gt; >>>>>>>> To: users(a)ovirt.org >>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM >>>>>>>> Subject: [ovirt-users] Error creating Disks >>>>>>>> >>>>>>>> oVirt Engine Version: 3.4.1-0.0.master.20140412010845.git43746c6.el6 >>>>>>>> >>>>>>>> >>>>>>>> While attempting to create a disk on an NFS storage domain, it fails with >>>>>>>> the >>>>>>>> following error in the engine.log >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2014-04-14 09:58:12,127 ERROR >>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in HSMGetAllTasksStatusesVDS >>>>>>>> method >>>>>>>> 2014-04-14 09:58:12,139 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>> (DefaultQuartzScheduler_Worker-72) BaseAsyncTask::LogEndTaskFailure: Task >>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, Parameters >>>>>>>> Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended >>>>>>>> with >>>>>>>> failure: >>>>>>>> 2014-04-14 09:58:12,159 ERROR [org.ovirt.engine.core.bll.AddDiskCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with failure: >>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand >>>>>>>> 2014-04-14 09:58:12,212 ERROR >>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with failure: >>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users