[Users] cant remove disks from iscsi domain

Dafna Ron dron at redhat.com
Thu Aug 22 10:14:01 UTC 2013


there are actually a lot of tables that have a reference to the image in 
the db.
I am adding Liron who might be able to give you the patch needed for 
deleting these images or tell you in what rpm they were merged to.


On 08/22/2013 11:06 AM, Yuriy Demchenko wrote:
> If by "latest" you mean head/nightly - i'm not using it yet, I'm using 
> packets from stable el6 repo, so my test env match production env.
>
> Errors i get:
>> User admin at internal failed to initiate removing of disk 
>> virt-dhcp-3_Disk1 from domain iscsi-store
>
>> Thread-1196::ERROR::2013-08-22 
>> 14:00:57,736::task::833::TaskManager.Task::(_setError) 
>> Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error
>> Traceback (most recent call last):
>>   File "/usr/share/vdsm/storage/task.py", line 840, in _run
>>     return fn(*args, **kargs)
>>   File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
>>     res = f(*args, **kwargs)
>>   File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage
>>     raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> ImageDoesNotExistInSD: Image does not exist in domain: 
>> 'image=dde47a09-81d9-4592-87e4-92398ea834ce, 
>> domain=d786e2d5-05ab-4da6-95fc-1af791a3c113'
>
> Maybe just do some sql-query to remove them from DB?
>
> Yuriy Demchenko
>
> On 08/22/2013 01:55 PM, Dafna Ron wrote:
>> deleting illegal objects from UI was added to the latest ovirt/vdsm
>> are you getting any errors when trying to delete them now?
>>
>>
>>
>> On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
>>> Thanks a lot! Patch did the job, a few more disks were deleted 
>>> successfully.
>>>
>>> One last question - how do i remove stale records of disks in 
>>> "illegal" state?
>>>
>>> Yuriy Demchenko
>>>
>>> On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
>>>>
>>>> ----- Original Message -----
>>>>> you said that the disks are deleted although an exception is given 
>>>>> but
>>>>> the engine is reporting the delete as failed.
>>>>> I think that I remember a bug reported and fixed on that as well 
>>>>> but I
>>>>> can't seem to find it.
>>>>> Adding Ayal and Eduardo
>>>>>
>>>> The log issue already solved in v4.11.0~380.
>>>>
>>>> commit ad916c79e2b0959dea20dd19a21b99bc702d65ca
>>>> Author: Eduardo Warszawski <ewarszaw at redhat.com>
>>>> Date:   Mon Dec 17 14:32:51 2012 +0200
>>>>
>>>>      Fix typo in negative flow log in blockSD.rmDCImgDir().
>>>>           Related to BZ#885489.
>>>>           Change-Id: I951e582acc86e08d709da4249084015660fc4ea0
>>>>      Signed-off-by: Eduardo <ewarszaw at redhat.com>
>>>>      Reviewed-on: http://gerrit.ovirt.org/10153
>>>>      Reviewed-by: Yeela Kaplan <ykaplan at redhat.com>
>>>>      Reviewed-by: Ayal Baron <abaron at redhat.com>
>>>>      Tested-by: Dan Kenigsberg <danken at redhat.com>
>>>>
>>>> diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py
>>>> index b5447cd..22a434b 100644
>>>> --- a/vdsm/storage/blockSD.py
>>>> +++ b/vdsm/storage/blockSD.py
>>>> @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain):
>>>>           try:
>>>>               os.rmdir(imgPath)
>>>>           except OSError:
>>>> -            self.log.warning("Can't rmdir %s. %s", imgPath, 
>>>> exc_info=True)
>>>> +            self.log.warning("Can't rmdir %s", imgPath, 
>>>> exc_info=True)
>>>>           else:
>>>>               self.log.debug("removed image dir: %s", imgPath)
>>>>           return imgPath
>>>>
>>>>>
>>>>> On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
>>>>>> I've done some more tests - and it seems quota error is not 
>>>>>> related to
>>>>>> my issue: I tried to remove another disk and this time there were no
>>>>>> quota errors in engine.log
>>>>>> New logs in attach.
>>>>>>
>>>>>> What catches my eye in logs is this errors, but maybe that's not the
>>>>>> root of case:
>>>>>>> Thread-60725::DEBUG::2013-08-22
>>>>>>> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation
>>>>>>> 'lvm invalidate operation' released the operation mutex
>>>>>>> Thread-60725::WARNING::2013-08-22
>>>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
>>>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
>>>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 
>>>>>>>
>>>>>>> 85-8900-da271affe831. [Errno 2] No such file or directory:
>>>>>>> '/rhev/data-center/mn
>>>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
>>>>>>>
>>>>>>> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
>>>>>>> Thread-60725::WARNING::2013-08-22
>>>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
>>>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
>>>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 
>>>>>>>
>>>>>>> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
>>>>>>> '/rhev/data-center/mn
>>>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
>>>>>>>
>>>>>>> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
>>>>>>> Thread-60725::DEBUG::2013-08-22
>>>>>>> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
>>>>>>> removed: []
>>>>>>> Thread-60725::ERROR::2013-08-22
>>>>>>> 10:37:45,549::task::833::TaskManager.Task::(_set Error)
>>>>>>> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
>>>>>>> Traceback (most recent call last):
>>>>>>>    File "/usr/share/vdsm/storage/task.py", line 840, in _run
>>>>>>>      return fn(*args, **kargs)
>>>>>>>    File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
>>>>>>>      res = f(*args, **kwargs)
>>>>>>>    File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage
>>>>>>>      dom.deleteImage(sdUUID, imgUUID, volsByImg)
>>>>>>>    File "/usr/share/vdsm/storage/blockSD.py", line 957, in 
>>>>>>> deleteImage
>>>>>>>      self.rmDCImgDir(imgUUID, volsImgs)
>>>>>>>    File "/usr/share/vdsm/storage/blockSD.py", line 943, in 
>>>>>>> rmDCImgDir
>>>>>>>      self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 1068, 
>>>>>>> in warning
>>>>>>>      self._log(WARNING, msg, args, **kwargs)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 1173, 
>>>>>>> in _log
>>>>>>>      self.handle(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 1183, 
>>>>>>> in handle
>>>>>>>      self.callHandlers(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in
>>>>>>> callHandlers
>>>>>>>      hdlr.handle(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 679, in 
>>>>>>> handle
>>>>>>>      self.emit(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/handlers.py", line 780, in 
>>>>>>> emit
>>>>>>>      msg = self.format(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 654, in 
>>>>>>> format
>>>>>>>      return fmt.format(record)
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 436, in 
>>>>>>> format
>>>>>>>      record.message = record.getMessage()
>>>>>>>    File "/usr/lib64/python2.6/logging/__init__.py", line 306, in
>>>>>>> getMessage
>>>>>>>      msg = msg % self.args
>>>>>>> TypeError: not enough arguments for format string
>>>>>>
>>>>>> Yuriy Demchenko
>>>>>>
>>>>>> On 08/22/2013 04:11 AM, Greg Padgett wrote:
>>>>>>> On 08/21/2013 04:10 PM, Dafna Ron wrote:
>>>>>>>> there is a is an exception in the log related to a quota 
>>>>>>>> calculation
>>>>>>>>
>>>>>>>> 2013-08-21 17:52:32,694 ERROR
>>>>>>>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl]
>>>>>>>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled 
>>>>>>>> method upd
>>>>>>>> ateQuotaCache: java.lang.reflect.InvocationTargetException
>>>>>>>>           at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown
>>>>>>>> Source)
>>>>>>>> [:1.7.0   _25]
>>>>>>>>           at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces 
>>>>>>>>
>>>>>>>> sorImpl.java:43) [rt.jar:1.7.0_25]
>>>>>>>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>           at
>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 
>>>>>>>>
>>>>>>>> 60) [scheduler.jar:]
>>>>>>>>           at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>> [quartz.jar:]
>>>>>>>>           at
>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j 
>>>>>>>>
>>>>>>>> ava:557) [quartz.jar:]
>>>>>>>> Caused by: org.springframework.jdbc.BadSqlGrammarException:
>>>>>>>> PreparedStatementCal   lback; bad SQL grammar [select * from
>>>>>>>> calculateallstorageusage()]; nested excep   tion is
>>>>>>>> org.postgresql.util.PSQLException: ERROR: column
>>>>>>>> "quota_limitation.quota
>>>>>>>> _id" must appear in the GROUP BY clause or be used in an aggregate
>>>>>>>> function
>>>>>>>>     Where: PL/pgSQL function "calculateallstorageusage" line 3 at
>>>>>>>> RETURN QUERY
>>>>>>>>
>>>>>>>>
>>>>>>>> in any case this is a bug.
>>>>>>>> I'm adding Doron to this mail, perhaps this was reported in the 
>>>>>>>> past
>>>>>>>> and
>>>>>>>> already solved in later versions.
>>>>>>>> if not it should be reported and fixed.
>>>>>>>>
>>>>>>>> Dafna
>>>>>>>>
>>>>>>>>
>>>>>>> If I'm not mistaken, it looks like this bug:
>>>>>>>
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=905891
>>>>>>>
>>>>>>> Greg
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/21/2013 05:26 PM, Yura Demchenko wrote:
>>>>>>>>> 21.08.2013 19:18, Dafna Ron пишет:
>>>>>>>>>> from the logs it appears to be a quota issue.
>>>>>>>>>> do you have quota enabled?
>>>>>>>>>>
>>>>>>>>> Yes, quota in "enforced" mode. But VMs/disks in question 
>>>>>>>>> belongs to
>>>>>>>>> "unlimited" quota (defined quota policy with no limits on
>>>>>>>>> storage/cpu/ram)
>>>>>>>>>
>>>>>>>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've recently encountered a problem with removing disks from 
>>>>>>>>>>> iscsi
>>>>>>>>>>> domain in my test lab - just cant remove any.
>>>>>>>>>>> Remove operation fails with message "User admin at internal 
>>>>>>>>>>> failed to
>>>>>>>>>>> initiate removing of disk pg-slave1_opt from domain 
>>>>>>>>>>> iscsi-store" and
>>>>>>>>>>> re-elections of SPM. After that - disk is marked as 
>>>>>>>>>>> "illegal" in
>>>>>>>>>>> ovirt
>>>>>>>>>>> webinterface, however, it is _in fact_ removed from storage -
>>>>>>>>>>> lvdisplay doesn't show it and free space is updated correctly.
>>>>>>>>>>> And this happens with just about every disk/vm i try to remove.
>>>>>>>>>>>
>>>>>>>>>>> ovirt 3.2.2-el6
>>>>>>>>>>> centos 6.4
>>>>>>>>>>> vdsm-4.10.3-17.el6
>>>>>>>>>>> lvm2-2.02.98-9.el6
>>>>>>>>>>>
>>>>>>>>>>> Any tips? logs in attach
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>
>>>>>
>>>>> -- 
>>>>> Dafna Ron
>>>>>
>>>
>>
>>
>


-- 
Dafna Ron



More information about the Users mailing list