Re: [Users] cant remove disks from iscsi domain

If by "latest" you mean head/nightly - i'm not using it yet, I'm using packets from stable el6 repo, so my test env match production env. Errors i get: > User admin@internal failed to initiate removing of disk > virt-dhcp-3_Disk1 from domain iscsi-store > Thread-1196::ERROR::2013-08-22 > 14:00:57,736::task::833::TaskManager.Task::(_setError) > Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error > Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 42, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage > raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) > ImageDoesNotExistInSD: Image does not exist in domain: > 'image=dde47a09-81d9-4592-87e4-92398ea834ce, > domain=d786e2d5-05ab-4da6-95fc-1af791a3c113' Maybe just do some sql-query to remove them from DB? Yuriy Demchenko On 08/22/2013 01:55 PM, Dafna Ron wrote: > deleting illegal objects from UI was added to the latest ovirt/vdsm > are you getting any errors when trying to delete them now? > > > > On 08/22/2013 10:46 AM, Yuriy Demchenko wrote: >> Thanks a lot! Patch did the job, a few more disks were deleted >> successfully. >> >> One last question - how do i remove stale records of disks in >> "illegal" state? >> >> Yuriy Demchenko >> >> On 08/22/2013 01:25 PM, Eduardo Warszawski wrote: >>> >>> ----- Original Message ----- >>>> you said that the disks are deleted although an exception is given >>>> but >>>> the engine is reporting the delete as failed. >>>> I think that I remember a bug reported and fixed on that as well >>>> but I >>>> can't seem to find it. >>>> Adding Ayal and Eduardo >>>> >>> The log issue already solved in v4.11.0~380. >>> >>> commit ad916c79e2b0959dea20dd19a21b99bc702d65ca >>> Author: Eduardo Warszawski <ewarszaw(a)redhat.com&gt; >>> Date: Mon Dec 17 14:32:51 2012 +0200 >>> >>> Fix typo in negative flow log in blockSD.rmDCImgDir(). >>> Related to BZ#885489. >>> Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 >>> Signed-off-by: Eduardo <ewarszaw(a)redhat.com&gt; >>> Reviewed-on: http://gerrit.ovirt.org/10153 >>> Reviewed-by: Yeela Kaplan <ykaplan(a)redhat.com&gt; >>> Reviewed-by: Ayal Baron <abaron(a)redhat.com&gt; >>> Tested-by: Dan Kenigsberg <danken(a)redhat.com&gt; >>> >>> diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py >>> index b5447cd..22a434b 100644 >>> --- a/vdsm/storage/blockSD.py >>> +++ b/vdsm/storage/blockSD.py >>> @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): >>> try: >>> os.rmdir(imgPath) >>> except OSError: >>> - self.log.warning("Can't rmdir %s. %s", imgPath, >>> exc_info=True) >>> + self.log.warning("Can't rmdir %s", imgPath, >>> exc_info=True) >>> else: >>> self.log.debug("removed image dir: %s", imgPath) >>> return imgPath >>> >>>> >>>> On 08/22/2013 07:55 AM, Yuriy Demchenko wrote: >>>>> I've done some more tests - and it seems quota error is not >>>>> related to >>>>> my issue: I tried to remove another disk and this time there were no >>>>> quota errors in engine.log >>>>> New logs in attach. >>>>> >>>>> What catches my eye in logs is this errors, but maybe that's not the >>>>> root of case: >>>>>> Thread-60725::DEBUG::2013-08-22 >>>>>> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation >>>>>> 'lvm invalidate operation' released the operation mutex >>>>>> Thread-60725::WARNING::2013-08-22 >>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 >>>>>> >>>>>> 85-8900-da271affe831. [Errno 2] No such file or directory: >>>>>> '/rhev/data-center/mn >>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>>>> >>>>>> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' >>>>>> Thread-60725::WARNING::2013-08-22 >>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 >>>>>> >>>>>> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: >>>>>> '/rhev/data-center/mn >>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>>>> >>>>>> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' >>>>>> Thread-60725::DEBUG::2013-08-22 >>>>>> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) >>>>>> removed: [] >>>>>> Thread-60725::ERROR::2013-08-22 >>>>>> 10:37:45,549::task::833::TaskManager.Task::(_set Error) >>>>>> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error >>>>>> Traceback (most recent call last): >>>>>> File "/usr/share/vdsm/storage/task.py", line 840, in _run >>>>>> return fn(*args, **kargs) >>>>>> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper >>>>>> res = f(*args, **kwargs) >>>>>> File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage >>>>>> dom.deleteImage(sdUUID, imgUUID, volsByImg) >>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 957, in >>>>>> deleteImage >>>>>> self.rmDCImgDir(imgUUID, volsImgs) >>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 943, in >>>>>> rmDCImgDir >>>>>> self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1068, >>>>>> in warning >>>>>> self._log(WARNING, msg, args, **kwargs) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1173, >>>>>> in _log >>>>>> self.handle(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1183, >>>>>> in handle >>>>>> self.callHandlers(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in >>>>>> callHandlers >>>>>> hdlr.handle(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 679, in >>>>>> handle >>>>>> self.emit(record) >>>>>> File "/usr/lib64/python2.6/logging/handlers.py", line 780, in >>>>>> emit >>>>>> msg = self.format(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 654, in >>>>>> format >>>>>> return fmt.format(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 436, in >>>>>> format >>>>>> record.message = record.getMessage() >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 306, in >>>>>> getMessage >>>>>> msg = msg % self.args >>>>>> TypeError: not enough arguments for format string >>>>> >>>>> Yuriy Demchenko >>>>> >>>>> On 08/22/2013 04:11 AM, Greg Padgett wrote: >>>>>> On 08/21/2013 04:10 PM, Dafna Ron wrote: >>>>>>> there is a is an exception in the log related to a quota >>>>>>> calculation >>>>>>> >>>>>>> 2013-08-21 17:52:32,694 ERROR >>>>>>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] >>>>>>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled >>>>>>> method upd >>>>>>> ateQuotaCache: java.lang.reflect.InvocationTargetException >>>>>>> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown >>>>>>> Source) >>>>>>> [:1.7.0 _25] >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >>>>>>> >>>>>>> sorImpl.java:43) [rt.jar:1.7.0_25] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: >>>>>>> >>>>>>> 60) [scheduler.jar:] >>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>> [quartz.jar:] >>>>>>> at >>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j >>>>>>> >>>>>>> ava:557) [quartz.jar:] >>>>>>> Caused by: org.springframework.jdbc.BadSqlGrammarException: >>>>>>> PreparedStatementCal lback; bad SQL grammar [select * from >>>>>>> calculateallstorageusage()]; nested excep tion is >>>>>>> org.postgresql.util.PSQLException: ERROR: column >>>>>>> "quota_limitation.quota >>>>>>> _id" must appear in the GROUP BY clause or be used in an aggregate >>>>>>> function >>>>>>> Where: PL/pgSQL function "calculateallstorageusage" line 3 at >>>>>>> RETURN QUERY >>>>>>> >>>>>>> >>>>>>> in any case this is a bug. >>>>>>> I'm adding Doron to this mail, perhaps this was reported in the >>>>>>> past >>>>>>> and >>>>>>> already solved in later versions. >>>>>>> if not it should be reported and fixed. >>>>>>> >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>> If I'm not mistaken, it looks like this bug: >>>>>> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=905891 >>>>>> >>>>>> Greg >>>>>> >>>>>>> >>>>>>> >>>>>>> On 08/21/2013 05:26 PM, Yura Demchenko wrote: >>>>>>>> 21.08.2013 19:18, Dafna Ron пишет: >>>>>>>>> from the logs it appears to be a quota issue. >>>>>>>>> do you have quota enabled? >>>>>>>>> >>>>>>>> Yes, quota in "enforced" mode. But VMs/disks in question >>>>>>>> belongs to >>>>>>>> "unlimited" quota (defined quota policy with no limits on >>>>>>>> storage/cpu/ram) >>>>>>>> >>>>>>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I've recently encountered a problem with removing disks from >>>>>>>>>> iscsi >>>>>>>>>> domain in my test lab - just cant remove any. >>>>>>>>>> Remove operation fails with message "User admin@internal >>>>>>>>>> failed to >>>>>>>>>> initiate removing of disk pg-slave1_opt from domain >>>>>>>>>> iscsi-store" and >>>>>>>>>> re-elections of SPM. After that - disk is marked as >>>>>>>>>> "illegal" in >>>>>>>>>> ovirt >>>>>>>>>> webinterface, however, it is _in fact_ removed from storage - >>>>>>>>>> lvdisplay doesn't show it and free space is updated correctly. >>>>>>>>>> And this happens with just about every disk/vm i try to remove. >>>>>>>>>> >>>>>>>>>> ovirt 3.2.2-el6 >>>>>>>>>> centos 6.4 >>>>>>>>>> vdsm-4.10.3-17.el6 >>>>>>>>>> lvm2-2.02.98-9.el6 >>>>>>>>>> >>>>>>>>>> Any tips? logs in attach >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>> >>>> >>>> -- >>>> Dafna Ron >>>> >> > >