[Users] cant remove disks from iscsi domain

Thu Aug 22 02:55:19 EDT 2013

I've done some more tests - and it seems quota error is not related to 
my issue: I tried to remove another disk and this time there were no 
quota errors in engine.log
New logs in attach.

What catches my eye in logs is this errors, but maybe that's not the 
root of case:
> Thread-60725::DEBUG::2013-08-22 
> 10:37:45,549::lvm::485::OperationMutex::(_invali  datevgs) Operation 
> 'lvm invalidate operation' released the operation mutex
> Thread-60725::WARNING::2013-08-22 
> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) 
> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 
> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 
> 85-8900-da271affe831. [Errno 2] No such file or directory: 
> '/rhev/data-center/mn 
> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
> 7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
> Thread-60725::WARNING::2013-08-22 
> 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) 
> Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 
> 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 
> 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: 
> '/rhev/data-center/mn 
> t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
> 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
> Thread-60725::DEBUG::2013-08-22 
> 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) 
> removed: []
> Thread-60725::ERROR::2013-08-22 
> 10:37:45,549::task::833::TaskManager.Task::(_set  Error) 
> Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
> Traceback (most recent call last):
>   File "/usr/share/vdsm/storage/task.py", line 840, in _run
>     return fn(*args, **kargs)
>   File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
>     res = f(*args, **kwargs)
>   File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage
>     dom.deleteImage(sdUUID, imgUUID, volsByImg)
>   File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage
>     self.rmDCImgDir(imgUUID, volsImgs)
>   File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir
>     self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning
>     self._log(WARNING, msg, args, **kwargs)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log
>     self.handle(record)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle
>     self.callHandlers(record)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in 
> callHandlers
>     hdlr.handle(record)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle
>     self.emit(record)
>   File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit
>     msg = self.format(record)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format
>     return fmt.format(record)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format
>     record.message = record.getMessage()
>   File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage
>     msg = msg % self.args
> TypeError: not enough arguments for format string

Yuriy Demchenko

On 08/22/2013 04:11 AM, Greg Padgett wrote:
> On 08/21/2013 04:10 PM, Dafna Ron wrote:
>> there is a is an exception in the log related to a quota calculation
>>
>> 2013-08-21 17:52:32,694 ERROR
>> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu   artzImpl]
>> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd
>> ateQuotaCache: java.lang.reflect.InvocationTargetException
>>          at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown 
>> Source)
>> [:1.7.0   _25]
>>          at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:43) [rt.jar:1.7.0_25]
>>          at java.lang.reflect.Method.invoke(Method.java:606)
>> [rt.jar:1.7.0_25]
>>          at
>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:
>> 60) [scheduler.jar:]
>>          at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>> [quartz.jar:]
>>          at
>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
>> ava:557) [quartz.jar:]
>> Caused by: org.springframework.jdbc.BadSqlGrammarException:
>> PreparedStatementCal   lback; bad SQL grammar [select * from
>> calculateallstorageusage()]; nested excep   tion is
>> org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota
>> _id" must appear in the GROUP BY clause or be used in an aggregate 
>> function
>>    Where: PL/pgSQL function "calculateallstorageusage" line 3 at 
>> RETURN QUERY
>>
>>
>> in any case this is a bug.
>> I'm adding Doron to this mail, perhaps this was reported in the past and
>> already solved in later versions.
>> if not it should be reported and fixed.
>>
>> Dafna
>>
>>
>
> If I'm not mistaken, it looks like this bug:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=905891
>
> Greg
>
>>
>>
>>
>> On 08/21/2013 05:26 PM, Yura Demchenko wrote:
>>> 21.08.2013 19:18, Dafna Ron пишет:
>>>> from the logs it appears to be a quota issue.
>>>> do you have quota enabled?
>>>>
>>> Yes, quota in "enforced" mode. But VMs/disks in question belongs to
>>> "unlimited" quota (defined quota policy with no limits on 
>>> storage/cpu/ram)
>>>
>>>>
>>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
>>>>> Hi,
>>>>>
>>>>> I've recently encountered a problem with removing disks from iscsi
>>>>> domain in my test lab - just cant remove any.
>>>>> Remove operation fails with message "User admin at internal failed to
>>>>> initiate removing of disk pg-slave1_opt from domain iscsi-store" and
>>>>> re-elections of SPM. After that - disk is marked as "illegal" in 
>>>>> ovirt
>>>>> webinterface, however, it is _in fact_ removed from storage -
>>>>> lvdisplay doesn't show it and free space is updated correctly.
>>>>> And this happens with just about every disk/vm i try to remove.
>>>>>
>>>>> ovirt 3.2.2-el6
>>>>> centos 6.4
>>>>> vdsm-4.10.3-17.el6
>>>>> lvm2-2.02.98-9.el6
>>>>>
>>>>> Any tips? logs in attach
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine.log
Type: text/x-log
Size: 9651 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130822/99587a27/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.log
Type: text/x-log
Size: 60962 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130822/99587a27/attachment-0003.bin>