I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach. What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users