you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo

On 08/22/2013 07:55 AM, Yuriy Demchenko wrote: > I've done some more tests - and it seems quota error is not related to > my issue: I tried to remove another disk and this time there were no > quota errors in engine.log > New logs in attach. > > What catches my eye in logs is this errors, but maybe that's not the > root of case: >> Thread-60725::DEBUG::2013-08-22 >> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation >> 'lvm invalidate operation' released the operation mutex >> Thread-60725::WARNING::2013-08-22 >> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 >> 85-8900-da271affe831. [Errno 2] No such file or directory: >> '/rhev/data-center/mn >> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' >> Thread-60725::WARNING::2013-08-22 >> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 >> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: >> '/rhev/data-center/mn >> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' >> Thread-60725::DEBUG::2013-08-22 >> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) >> removed: [] >> Thread-60725::ERROR::2013-08-22 >> 10:37:45,549::task::833::TaskManager.Task::(_set Error) >> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error >> Traceback (most recent call last): >> File "/usr/share/vdsm/storage/task.py", line 840, in _run >> return fn(*args, **kargs) >> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper >> res = f(*args, **kwargs) >> File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage >> dom.deleteImage(sdUUID, imgUUID, volsByImg) >> File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage >> self.rmDCImgDir(imgUUID, volsImgs) >> File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir >> self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) >> File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning >> self._log(WARNING, msg, args, **kwargs) >> File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log >> self.handle(record) >> File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle >> self.callHandlers(record) >> File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in >> callHandlers >> hdlr.handle(record) >> File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle >> self.emit(record) >> File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit >> msg = self.format(record) >> File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format >> return fmt.format(record) >> File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format >> record.message = record.getMessage() >> File "/usr/lib64/python2.6/logging/__init__.py", line 306, in >> getMessage >> msg = msg % self.args >> TypeError: not enough arguments for format string > > > Yuriy Demchenko > > On 08/22/2013 04:11 AM, Greg Padgett wrote: >> On 08/21/2013 04:10 PM, Dafna Ron wrote: >>> there is a is an exception in the log related to a quota calculation >>> >>> 2013-08-21 17:52:32,694 ERROR >>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] >>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd >>> ateQuotaCache: java.lang.reflect.InvocationTargetException >>> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown >>> Source) >>> [:1.7.0 _25] >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >>> sorImpl.java:43) [rt.jar:1.7.0_25] >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> [rt.jar:1.7.0_25] >>> at >>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: >>> 60) [scheduler.jar:] >>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>> [quartz.jar:] >>> at >>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j >>> ava:557) [quartz.jar:] >>> Caused by: org.springframework.jdbc.BadSqlGrammarException: >>> PreparedStatementCal lback; bad SQL grammar [select * from >>> calculateallstorageusage()]; nested excep tion is >>> org.postgresql.util.PSQLException: ERROR: column >>> "quota_limitation.quota >>> _id" must appear in the GROUP BY clause or be used in an aggregate >>> function >>> Where: PL/pgSQL function "calculateallstorageusage" line 3 at >>> RETURN QUERY >>> >>> >>> in any case this is a bug. >>> I'm adding Doron to this mail, perhaps this was reported in the past >>> and >>> already solved in later versions. >>> if not it should be reported and fixed. >>> >>> Dafna >>> >>> >> >> If I'm not mistaken, it looks like this bug: >> >> https://bugzilla.redhat.com/show_bug.cgi?id=905891 >> >> Greg >> >>> >>> >>> >>> On 08/21/2013 05:26 PM, Yura Demchenko wrote: >>>> 21.08.2013 19:18, Dafna Ron пишет: >>>>> from the logs it appears to be a quota issue. >>>>> do you have quota enabled? >>>>> >>>> Yes, quota in "enforced" mode. But VMs/disks in question belongs to >>>> "unlimited" quota (defined quota policy with no limits on >>>> storage/cpu/ram) >>>> >>>>> >>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>>>> Hi, >>>>>> >>>>>> I've recently encountered a problem with removing disks from iscsi >>>>>> domain in my test lab - just cant remove any. >>>>>> Remove operation fails with message "User admin@internal failed to >>>>>> initiate removing of disk pg-slave1_opt from domain iscsi-store" and >>>>>> re-elections of SPM. After that - disk is marked as "illegal" in >>>>>> ovirt >>>>>> webinterface, however, it is _in fact_ removed from storage - >>>>>> lvdisplay doesn't show it and free space is updated correctly. >>>>>> And this happens with just about every disk/vm i try to remove. >>>>>> >>>>>> ovirt 3.2.2-el6 >>>>>> centos 6.4 >>>>>> vdsm-4.10.3-17.el6 >>>>>> lvm2-2.02.98-9.el6 >>>>>> >>>>>> Any tips? logs in attach >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>> >>> >> > -- Dafna Ron

Thanks a lot! Patch did the job, a few more disks were deleted successfully. One last question - how do i remove stale records of disks in "illegal" state? Yuriy Demchenko On 08/22/2013 01:25 PM, Eduardo Warszawski wrote: > > ----- Original Message ----- >> you said that the disks are deleted although an exception is given but >> the engine is reporting the delete as failed. >> I think that I remember a bug reported and fixed on that as well but I >> can't seem to find it. >> Adding Ayal and Eduardo >> > The log issue already solved in v4.11.0~380. > > commit ad916c79e2b0959dea20dd19a21b99bc702d65ca > Author: Eduardo Warszawski <ewarszaw(a)redhat.com&gt; > Date: Mon Dec 17 14:32:51 2012 +0200 > > Fix typo in negative flow log in blockSD.rmDCImgDir(). > Related to BZ#885489. > Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 > Signed-off-by: Eduardo <ewarszaw(a)redhat.com&gt; > Reviewed-on: http://gerrit.ovirt.org/10153 > Reviewed-by: Yeela Kaplan <ykaplan(a)redhat.com&gt; > Reviewed-by: Ayal Baron <abaron(a)redhat.com&gt; > Tested-by: Dan Kenigsberg <danken(a)redhat.com&gt; > > diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py > index b5447cd..22a434b 100644 > --- a/vdsm/storage/blockSD.py > +++ b/vdsm/storage/blockSD.py > @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): > try: > os.rmdir(imgPath) > except OSError: > - self.log.warning("Can't rmdir %s. %s", imgPath, > exc_info=True) > + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) > else: > self.log.debug("removed image dir: %s", imgPath) > return imgPath > >> >> On 08/22/2013 07:55 AM, Yuriy Demchenko wrote: >>> I've done some more tests - and it seems quota error is not related to >>> my issue: I tried to remove another disk and this time there were no >>> quota errors in engine.log >>> New logs in attach. >>> >>> What catches my eye in logs is this errors, but maybe that's not the >>> root of case: >>>> Thread-60725::DEBUG::2013-08-22 >>>> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation >>>> 'lvm invalidate operation' released the operation mutex >>>> Thread-60725::WARNING::2013-08-22 >>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 >>>> >>>> 85-8900-da271affe831. [Errno 2] No such file or directory: >>>> '/rhev/data-center/mn >>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>> >>>> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' >>>> Thread-60725::WARNING::2013-08-22 >>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 >>>> >>>> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: >>>> '/rhev/data-center/mn >>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>> >>>> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' >>>> Thread-60725::DEBUG::2013-08-22 >>>> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) >>>> removed: [] >>>> Thread-60725::ERROR::2013-08-22 >>>> 10:37:45,549::task::833::TaskManager.Task::(_set Error) >>>> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error >>>> Traceback (most recent call last): >>>> File "/usr/share/vdsm/storage/task.py", line 840, in _run >>>> return fn(*args, **kargs) >>>> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper >>>> res = f(*args, **kwargs) >>>> File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage >>>> dom.deleteImage(sdUUID, imgUUID, volsByImg) >>>> File "/usr/share/vdsm/storage/blockSD.py", line 957, in >>>> deleteImage >>>> self.rmDCImgDir(imgUUID, volsImgs) >>>> File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir >>>> self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in >>>> warning >>>> self._log(WARNING, msg, args, **kwargs) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in >>>> _log >>>> self.handle(record) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in >>>> handle >>>> self.callHandlers(record) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in >>>> callHandlers >>>> hdlr.handle(record) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 679, in >>>> handle >>>> self.emit(record) >>>> File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit >>>> msg = self.format(record) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 654, in >>>> format >>>> return fmt.format(record) >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 436, in >>>> format >>>> record.message = record.getMessage() >>>> File "/usr/lib64/python2.6/logging/__init__.py", line 306, in >>>> getMessage >>>> msg = msg % self.args >>>> TypeError: not enough arguments for format string >>> >>> Yuriy Demchenko >>> >>> On 08/22/2013 04:11 AM, Greg Padgett wrote: >>>> On 08/21/2013 04:10 PM, Dafna Ron wrote: >>>>> there is a is an exception in the log related to a quota calculation >>>>> >>>>> 2013-08-21 17:52:32,694 ERROR >>>>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] >>>>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled >>>>> method upd >>>>> ateQuotaCache: java.lang.reflect.InvocationTargetException >>>>> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown >>>>> Source) >>>>> [:1.7.0 _25] >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >>>>> >>>>> sorImpl.java:43) [rt.jar:1.7.0_25] >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> [rt.jar:1.7.0_25] >>>>> at >>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: >>>>> >>>>> 60) [scheduler.jar:] >>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> [quartz.jar:] >>>>> at >>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j >>>>> >>>>> ava:557) [quartz.jar:] >>>>> Caused by: org.springframework.jdbc.BadSqlGrammarException: >>>>> PreparedStatementCal lback; bad SQL grammar [select * from >>>>> calculateallstorageusage()]; nested excep tion is >>>>> org.postgresql.util.PSQLException: ERROR: column >>>>> "quota_limitation.quota >>>>> _id" must appear in the GROUP BY clause or be used in an aggregate >>>>> function >>>>> Where: PL/pgSQL function "calculateallstorageusage" line 3 at >>>>> RETURN QUERY >>>>> >>>>> >>>>> in any case this is a bug. >>>>> I'm adding Doron to this mail, perhaps this was reported in the past >>>>> and >>>>> already solved in later versions. >>>>> if not it should be reported and fixed. >>>>> >>>>> Dafna >>>>> >>>>> >>>> If I'm not mistaken, it looks like this bug: >>>> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=905891 >>>> >>>> Greg >>>> >>>>> >>>>> >>>>> On 08/21/2013 05:26 PM, Yura Demchenko wrote: >>>>>> 21.08.2013 19:18, Dafna Ron пишет: >>>>>>> from the logs it appears to be a quota issue. >>>>>>> do you have quota enabled? >>>>>>> >>>>>> Yes, quota in "enforced" mode. But VMs/disks in question belongs to >>>>>> "unlimited" quota (defined quota policy with no limits on >>>>>> storage/cpu/ram) >>>>>> >>>>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I've recently encountered a problem with removing disks from >>>>>>>> iscsi >>>>>>>> domain in my test lab - just cant remove any. >>>>>>>> Remove operation fails with message "User admin@internal >>>>>>>> failed to >>>>>>>> initiate removing of disk pg-slave1_opt from domain >>>>>>>> iscsi-store" and >>>>>>>> re-elections of SPM. After that - disk is marked as "illegal" in >>>>>>>> ovirt >>>>>>>> webinterface, however, it is _in fact_ removed from storage - >>>>>>>> lvdisplay doesn't show it and free space is updated correctly. >>>>>>>> And this happens with just about every disk/vm i try to remove. >>>>>>>> >>>>>>>> ovirt 3.2.2-el6 >>>>>>>> centos 6.4 >>>>>>>> vdsm-4.10.3-17.el6 >>>>>>>> lvm2-2.02.98-9.el6 >>>>>>>> >>>>>>>> Any tips? logs in attach >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>> >> >> -- >> Dafna Ron >>

If by "latest" you mean head/nightly - i'm not using it yet, I'm using packets from stable el6 repo, so my test env match production env. Errors i get: > User admin@internal failed to initiate removing of disk > virt-dhcp-3_Disk1 from domain iscsi-store > Thread-1196::ERROR::2013-08-22 > 14:00:57,736::task::833::TaskManager.Task::(_setError) > Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error > Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 42, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage > raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) > ImageDoesNotExistInSD: Image does not exist in domain: > 'image=dde47a09-81d9-4592-87e4-92398ea834ce, > domain=d786e2d5-05ab-4da6-95fc-1af791a3c113' Maybe just do some sql-query to remove them from DB? Yuriy Demchenko On 08/22/2013 01:55 PM, Dafna Ron wrote: > deleting illegal objects from UI was added to the latest ovirt/vdsm > are you getting any errors when trying to delete them now? > > > > On 08/22/2013 10:46 AM, Yuriy Demchenko wrote: >> Thanks a lot! Patch did the job, a few more disks were deleted >> successfully. >> >> One last question - how do i remove stale records of disks in >> "illegal" state? >> >> Yuriy Demchenko >> >> On 08/22/2013 01:25 PM, Eduardo Warszawski wrote: >>> >>> ----- Original Message ----- >>>> you said that the disks are deleted although an exception is given >>>> but >>>> the engine is reporting the delete as failed. >>>> I think that I remember a bug reported and fixed on that as well >>>> but I >>>> can't seem to find it. >>>> Adding Ayal and Eduardo >>>> >>> The log issue already solved in v4.11.0~380. >>> >>> commit ad916c79e2b0959dea20dd19a21b99bc702d65ca >>> Author: Eduardo Warszawski <ewarszaw(a)redhat.com&gt; >>> Date: Mon Dec 17 14:32:51 2012 +0200 >>> >>> Fix typo in negative flow log in blockSD.rmDCImgDir(). >>> Related to BZ#885489. >>> Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 >>> Signed-off-by: Eduardo <ewarszaw(a)redhat.com&gt; >>> Reviewed-on: http://gerrit.ovirt.org/10153 >>> Reviewed-by: Yeela Kaplan <ykaplan(a)redhat.com&gt; >>> Reviewed-by: Ayal Baron <abaron(a)redhat.com&gt; >>> Tested-by: Dan Kenigsberg <danken(a)redhat.com&gt; >>> >>> diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py >>> index b5447cd..22a434b 100644 >>> --- a/vdsm/storage/blockSD.py >>> +++ b/vdsm/storage/blockSD.py >>> @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): >>> try: >>> os.rmdir(imgPath) >>> except OSError: >>> - self.log.warning("Can't rmdir %s. %s", imgPath, >>> exc_info=True) >>> + self.log.warning("Can't rmdir %s", imgPath, >>> exc_info=True) >>> else: >>> self.log.debug("removed image dir: %s", imgPath) >>> return imgPath >>> >>>> >>>> On 08/22/2013 07:55 AM, Yuriy Demchenko wrote: >>>>> I've done some more tests - and it seems quota error is not >>>>> related to >>>>> my issue: I tried to remove another disk and this time there were no >>>>> quota errors in engine.log >>>>> New logs in attach. >>>>> >>>>> What catches my eye in logs is this errors, but maybe that's not the >>>>> root of case: >>>>>> Thread-60725::DEBUG::2013-08-22 >>>>>> 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation >>>>>> 'lvm invalidate operation' released the operation mutex >>>>>> Thread-60725::WARNING::2013-08-22 >>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 >>>>>> >>>>>> 85-8900-da271affe831. [Errno 2] No such file or directory: >>>>>> '/rhev/data-center/mn >>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>>>> >>>>>> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' >>>>>> Thread-60725::WARNING::2013-08-22 >>>>>> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) >>>>>> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da >>>>>> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 >>>>>> >>>>>> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: >>>>>> '/rhev/data-center/mn >>>>>> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da >>>>>> >>>>>> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' >>>>>> Thread-60725::DEBUG::2013-08-22 >>>>>> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) >>>>>> removed: [] >>>>>> Thread-60725::ERROR::2013-08-22 >>>>>> 10:37:45,549::task::833::TaskManager.Task::(_set Error) >>>>>> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error >>>>>> Traceback (most recent call last): >>>>>> File "/usr/share/vdsm/storage/task.py", line 840, in _run >>>>>> return fn(*args, **kargs) >>>>>> File "/usr/share/vdsm/logUtils.py", line 42, in wrapper >>>>>> res = f(*args, **kwargs) >>>>>> File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage >>>>>> dom.deleteImage(sdUUID, imgUUID, volsByImg) >>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 957, in >>>>>> deleteImage >>>>>> self.rmDCImgDir(imgUUID, volsImgs) >>>>>> File "/usr/share/vdsm/storage/blockSD.py", line 943, in >>>>>> rmDCImgDir >>>>>> self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1068, >>>>>> in warning >>>>>> self._log(WARNING, msg, args, **kwargs) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1173, >>>>>> in _log >>>>>> self.handle(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1183, >>>>>> in handle >>>>>> self.callHandlers(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in >>>>>> callHandlers >>>>>> hdlr.handle(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 679, in >>>>>> handle >>>>>> self.emit(record) >>>>>> File "/usr/lib64/python2.6/logging/handlers.py", line 780, in >>>>>> emit >>>>>> msg = self.format(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 654, in >>>>>> format >>>>>> return fmt.format(record) >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 436, in >>>>>> format >>>>>> record.message = record.getMessage() >>>>>> File "/usr/lib64/python2.6/logging/__init__.py", line 306, in >>>>>> getMessage >>>>>> msg = msg % self.args >>>>>> TypeError: not enough arguments for format string >>>>> >>>>> Yuriy Demchenko >>>>> >>>>> On 08/22/2013 04:11 AM, Greg Padgett wrote: >>>>>> On 08/21/2013 04:10 PM, Dafna Ron wrote: >>>>>>> there is a is an exception in the log related to a quota >>>>>>> calculation >>>>>>> >>>>>>> 2013-08-21 17:52:32,694 ERROR >>>>>>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] >>>>>>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled >>>>>>> method upd >>>>>>> ateQuotaCache: java.lang.reflect.InvocationTargetException >>>>>>> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown >>>>>>> Source) >>>>>>> [:1.7.0 _25] >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >>>>>>> >>>>>>> sorImpl.java:43) [rt.jar:1.7.0_25] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: >>>>>>> >>>>>>> 60) [scheduler.jar:] >>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>> [quartz.jar:] >>>>>>> at >>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j >>>>>>> >>>>>>> ava:557) [quartz.jar:] >>>>>>> Caused by: org.springframework.jdbc.BadSqlGrammarException: >>>>>>> PreparedStatementCal lback; bad SQL grammar [select * from >>>>>>> calculateallstorageusage()]; nested excep tion is >>>>>>> org.postgresql.util.PSQLException: ERROR: column >>>>>>> "quota_limitation.quota >>>>>>> _id" must appear in the GROUP BY clause or be used in an aggregate >>>>>>> function >>>>>>> Where: PL/pgSQL function "calculateallstorageusage" line 3 at >>>>>>> RETURN QUERY >>>>>>> >>>>>>> >>>>>>> in any case this is a bug. >>>>>>> I'm adding Doron to this mail, perhaps this was reported in the >>>>>>> past >>>>>>> and >>>>>>> already solved in later versions. >>>>>>> if not it should be reported and fixed. >>>>>>> >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>> If I'm not mistaken, it looks like this bug: >>>>>> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=905891 >>>>>> >>>>>> Greg >>>>>> >>>>>>> >>>>>>> >>>>>>> On 08/21/2013 05:26 PM, Yura Demchenko wrote: >>>>>>>> 21.08.2013 19:18, Dafna Ron пишет: >>>>>>>>> from the logs it appears to be a quota issue. >>>>>>>>> do you have quota enabled? >>>>>>>>> >>>>>>>> Yes, quota in "enforced" mode. But VMs/disks in question >>>>>>>> belongs to >>>>>>>> "unlimited" quota (defined quota policy with no limits on >>>>>>>> storage/cpu/ram) >>>>>>>> >>>>>>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I've recently encountered a problem with removing disks from >>>>>>>>>> iscsi >>>>>>>>>> domain in my test lab - just cant remove any. >>>>>>>>>> Remove operation fails with message "User admin@internal >>>>>>>>>> failed to >>>>>>>>>> initiate removing of disk pg-slave1_opt from domain >>>>>>>>>> iscsi-store" and >>>>>>>>>> re-elections of SPM. After that - disk is marked as >>>>>>>>>> "illegal" in >>>>>>>>>> ovirt >>>>>>>>>> webinterface, however, it is _in fact_ removed from storage - >>>>>>>>>> lvdisplay doesn't show it and free space is updated correctly. >>>>>>>>>> And this happens with just about every disk/vm i try to remove. >>>>>>>>>> >>>>>>>>>> ovirt 3.2.2-el6 >>>>>>>>>> centos 6.4 >>>>>>>>>> vdsm-4.10.3-17.el6 >>>>>>>>>> lvm2-2.02.98-9.el6 >>>>>>>>>> >>>>>>>>>> Any tips? logs in attach >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>> >>>> >>>> -- >>>> Dafna Ron >>>> >> > >