[Users] cant remove disks from iscsi domain
Yuriy Demchenko
demchenko.ya at gmail.com
Thu Aug 22 10:06:25 UTC 2013
If by "latest" you mean head/nightly - i'm not using it yet, I'm using
packets from stable el6 repo, so my test env match production env.
Errors i get:
> User admin at internal failed to initiate removing of disk
> virt-dhcp-3_Disk1 from domain iscsi-store
> Thread-1196::ERROR::2013-08-22
> 14:00:57,736::task::833::TaskManager.Task::(_setError)
> Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/task.py", line 840, in _run
> return fn(*args, **kargs)
> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
> res = f(*args, **kwargs)
> File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage
> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
> ImageDoesNotExistInSD: Image does not exist in domain:
> 'image=dde47a09-81d9-4592-87e4-92398ea834ce,
> domain=d786e2d5-05ab-4da6-95fc-1af791a3c113'
Maybe just do some sql-query to remove them from DB?
Yuriy Demchenko
On 08/22/2013 01:55 PM, Dafna Ron wrote:
> deleting illegal objects from UI was added to the latest ovirt/vdsm
> are you getting any errors when trying to delete them now?
>
>
>
> On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
>> Thanks a lot! Patch did the job, a few more disks were deleted
>> successfully.
>>
>> One last question - how do i remove stale records of disks in
>> "illegal" state?
>>
>> Yuriy Demchenko
>>
>> On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
>>>
>>> ----- Original Message -----
>>>> you said that the disks are deleted although an exception is given but
>>>> the engine is reporting the delete as failed.
>>>> I think that I remember a bug reported and fixed on that as well but I
>>>> can't seem to find it.
>>>> Adding Ayal and Eduardo
>>>>
>>> The log issue already solved in v4.11.0~380.
>>>
>>> commit ad916c79e2b0959dea20dd19a21b99bc702d65ca
>>> Author: Eduardo Warszawski <ewarszaw at redhat.com>
>>> Date: Mon Dec 17 14:32:51 2012 +0200
>>>
>>> Fix typo in negative flow log in blockSD.rmDCImgDir().
>>> Related to BZ#885489.
>>> Change-Id: I951e582acc86e08d709da4249084015660fc4ea0
>>> Signed-off-by: Eduardo <ewarszaw at redhat.com>
>>> Reviewed-on: http://gerrit.ovirt.org/10153
>>> Reviewed-by: Yeela Kaplan <ykaplan at redhat.com>
>>> Reviewed-by: Ayal Baron <abaron at redhat.com>
>>> Tested-by: Dan Kenigsberg <danken at redhat.com>
>>>
>>> diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py
>>> index b5447cd..22a434b 100644
>>> --- a/vdsm/storage/blockSD.py
>>> +++ b/vdsm/storage/blockSD.py
>>> @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain):
>>> try:
>>> os.rmdir(imgPath)
>>> except OSError:
>>> - self.log.warning("Can't rmdir %s. %s", imgPath,
>>> exc_info=True)
>>> + self.log.warning("Can't rmdir %s", imgPath, exc_info=True)
>>> else:
>>> self.log.debug("removed image dir: %s", imgPath)
>>> return imgPath
>>>
>>>>
>>>> On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
>>>>> I've done some more tests - and it seems quota error is not
>>>>> related to
>>>>> my issue: I tried to remove another disk and this time there were no
>>>>> quota errors in engine.log
>>>>> New logs in attach.
>>>>>
>>>>> What catches my eye in logs is this errors, but maybe that's not the
>>>>> root of case:
>>>>>> Thread-60725::DEBUG::2013-08-22
>>>>>> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation
>>>>>> 'lvm invalidate operation' released the operation mutex
>>>>>> Thread-60725::WARNING::2013-08-22
>>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
>>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
>>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40
>>>>>>
>>>>>> 85-8900-da271affe831. [Errno 2] No such file or directory:
>>>>>> '/rhev/data-center/mn
>>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
>>>>>>
>>>>>> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
>>>>>> Thread-60725::WARNING::2013-08-22
>>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
>>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
>>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43
>>>>>>
>>>>>> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
>>>>>> '/rhev/data-center/mn
>>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
>>>>>>
>>>>>> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
>>>>>> Thread-60725::DEBUG::2013-08-22
>>>>>> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
>>>>>> removed: []
>>>>>> Thread-60725::ERROR::2013-08-22
>>>>>> 10:37:45,549::task::833::TaskManager.Task::(_set Error)
>>>>>> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
>>>>>> Traceback (most recent call last):
>>>>>> File "/usr/share/vdsm/storage/task.py", line 840, in _run
>>>>>> return fn(*args, **kargs)
>>>>>> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
>>>>>> res = f(*args, **kwargs)
>>>>>> File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage
>>>>>> dom.deleteImage(sdUUID, imgUUID, volsByImg)
>>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 957, in
>>>>>> deleteImage
>>>>>> self.rmDCImgDir(imgUUID, volsImgs)
>>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 943, in
>>>>>> rmDCImgDir
>>>>>> self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in
>>>>>> warning
>>>>>> self._log(WARNING, msg, args, **kwargs)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in
>>>>>> _log
>>>>>> self.handle(record)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in
>>>>>> handle
>>>>>> self.callHandlers(record)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in
>>>>>> callHandlers
>>>>>> hdlr.handle(record)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 679, in
>>>>>> handle
>>>>>> self.emit(record)
>>>>>> File "/usr/lib64/python2.6/logging/handlers.py", line 780, in
>>>>>> emit
>>>>>> msg = self.format(record)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 654, in
>>>>>> format
>>>>>> return fmt.format(record)
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 436, in
>>>>>> format
>>>>>> record.message = record.getMessage()
>>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 306, in
>>>>>> getMessage
>>>>>> msg = msg % self.args
>>>>>> TypeError: not enough arguments for format string
>>>>>
>>>>> Yuriy Demchenko
>>>>>
>>>>> On 08/22/2013 04:11 AM, Greg Padgett wrote:
>>>>>> On 08/21/2013 04:10 PM, Dafna Ron wrote:
>>>>>>> there is a is an exception in the log related to a quota
>>>>>>> calculation
>>>>>>>
>>>>>>> 2013-08-21 17:52:32,694 ERROR
>>>>>>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl]
>>>>>>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled
>>>>>>> method upd
>>>>>>> ateQuotaCache: java.lang.reflect.InvocationTargetException
>>>>>>> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown
>>>>>>> Source)
>>>>>>> [:1.7.0 _25]
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>>>>>>
>>>>>>> sorImpl.java:43) [rt.jar:1.7.0_25]
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> [rt.jar:1.7.0_25]
>>>>>>> at
>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:
>>>>>>>
>>>>>>> 60) [scheduler.jar:]
>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>> [quartz.jar:]
>>>>>>> at
>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
>>>>>>>
>>>>>>> ava:557) [quartz.jar:]
>>>>>>> Caused by: org.springframework.jdbc.BadSqlGrammarException:
>>>>>>> PreparedStatementCal lback; bad SQL grammar [select * from
>>>>>>> calculateallstorageusage()]; nested excep tion is
>>>>>>> org.postgresql.util.PSQLException: ERROR: column
>>>>>>> "quota_limitation.quota
>>>>>>> _id" must appear in the GROUP BY clause or be used in an aggregate
>>>>>>> function
>>>>>>> Where: PL/pgSQL function "calculateallstorageusage" line 3 at
>>>>>>> RETURN QUERY
>>>>>>>
>>>>>>>
>>>>>>> in any case this is a bug.
>>>>>>> I'm adding Doron to this mail, perhaps this was reported in the
>>>>>>> past
>>>>>>> and
>>>>>>> already solved in later versions.
>>>>>>> if not it should be reported and fixed.
>>>>>>>
>>>>>>> Dafna
>>>>>>>
>>>>>>>
>>>>>> If I'm not mistaken, it looks like this bug:
>>>>>>
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=905891
>>>>>>
>>>>>> Greg
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/21/2013 05:26 PM, Yura Demchenko wrote:
>>>>>>>> 21.08.2013 19:18, Dafna Ron пишет:
>>>>>>>>> from the logs it appears to be a quota issue.
>>>>>>>>> do you have quota enabled?
>>>>>>>>>
>>>>>>>> Yes, quota in "enforced" mode. But VMs/disks in question
>>>>>>>> belongs to
>>>>>>>> "unlimited" quota (defined quota policy with no limits on
>>>>>>>> storage/cpu/ram)
>>>>>>>>
>>>>>>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I've recently encountered a problem with removing disks from
>>>>>>>>>> iscsi
>>>>>>>>>> domain in my test lab - just cant remove any.
>>>>>>>>>> Remove operation fails with message "User admin at internal
>>>>>>>>>> failed to
>>>>>>>>>> initiate removing of disk pg-slave1_opt from domain
>>>>>>>>>> iscsi-store" and
>>>>>>>>>> re-elections of SPM. After that - disk is marked as "illegal" in
>>>>>>>>>> ovirt
>>>>>>>>>> webinterface, however, it is _in fact_ removed from storage -
>>>>>>>>>> lvdisplay doesn't show it and free space is updated correctly.
>>>>>>>>>> And this happens with just about every disk/vm i try to remove.
>>>>>>>>>>
>>>>>>>>>> ovirt 3.2.2-el6
>>>>>>>>>> centos 6.4
>>>>>>>>>> vdsm-4.10.3-17.el6
>>>>>>>>>> lvm2-2.02.98-9.el6
>>>>>>>>>>
>>>>>>>>>> Any tips? logs in attach
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at ovirt.org
>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Dafna Ron
>>>>
>>
>
>
More information about the Users
mailing list