[Users] cant remove disks from iscsi domain
Hi, I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove. ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6 Any tips? logs in attach -- Yuriy Demchenko
from the logs it appears to be a quota issue. do you have quota enabled? On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dafna Ron
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
there is a is an exception in the log related to a quota calculation 2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed. Dafna On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dafna Ron
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug: https://bugzilla.redhat.com/show_bug.cgi?id=905891 Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach. What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message ----- | From: "Yuriy Demchenko" <demchenko.ya@gmail.com> | To: "Greg Padgett" <gpadgett@redhat.com> | Cc: users@ovirt.org | Sent: Thursday, August 22, 2013 9:55:19 AM | Subject: Re: [Users] cant remove disks from iscsi domain | | I've done some more tests - and it seems quota error is not related to | my issue: I tried to remove another disk and this time there were no | quota errors in engine.log | New logs in attach. | | What catches my eye in logs is this errors, but maybe that's not the | root of case: | > Thread-60725::DEBUG::2013-08-22 | > 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation | > 'lvm invalidate operation' released the operation mutex | > Thread-60725::WARNING::2013-08-22 | > 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) | > Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da | > 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 | > 85-8900-da271affe831. [Errno 2] No such file or directory: | > '/rhev/data-center/mn | > t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da | > 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' | > Thread-60725::WARNING::2013-08-22 | > 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) | > Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da | > 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 | > 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: | > '/rhev/data-center/mn | > t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da | > 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' | > Thread-60725::DEBUG::2013-08-22 | > 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) | > removed: [] | > Thread-60725::ERROR::2013-08-22 | > 10:37:45,549::task::833::TaskManager.Task::(_set Error) | > Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error | > Traceback (most recent call last): | > File "/usr/share/vdsm/storage/task.py", line 840, in _run | > return fn(*args, **kargs) | > File "/usr/share/vdsm/logUtils.py", line 42, in wrapper | > res = f(*args, **kwargs) | > File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage | > dom.deleteImage(sdUUID, imgUUID, volsByImg) | > File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage | > self.rmDCImgDir(imgUUID, volsImgs) | > File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir | > self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) | > File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning | > self._log(WARNING, msg, args, **kwargs) | > File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log | > self.handle(record) | > File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle | > self.callHandlers(record) | > File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in | > callHandlers | > hdlr.handle(record) | > File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle | > self.emit(record) | > File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit | > msg = self.format(record) | > File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format | > return fmt.format(record) | > File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format | > record.message = record.getMessage() | > File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage | > msg = msg % self.args | > TypeError: not enough arguments for format string | | | Yuriy Demchenko Yuri, just to clarify the quota part, the stack trace you provided was resolved as the bz Greg described- https://bugzilla.redhat.com/show_bug.cgi?id=905891 This is unrelated to the storage issue, as the quota caching is an independent procedure running in parallel. So what we now need to focus on is this part: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-4085-8900-da271affe831' What we need to understand is how did we get into this state; Did you have network issues or relevant crashes?
On 08/22/2013 11:17 AM, Doron Fediuck wrote:
Yuri, just to clarify the quota part, the stack trace you provided was resolved as the bz Greg described- https://bugzilla.redhat.com/show_bug.cgi?id=905891
This is unrelated to the storage issue, as the quota caching is an independent procedure running in parallel.
So what we now need to focus on is this part: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
What we need to understand is how did we get into this state; Did you have network issues or relevant crashes? Well, I had a few power loss events (its my test env, so no ups installed yet) with storage, engine and nodes powered off unclean; my storage is kinda slow - just raid5 from 3 sata hdd. But there were no network issues i'm aware of, iscsi network is in separate vlan with 1gbps physical. Apart from power loss events I had no crashes of engine or nodes.
Yuriy Demchenko
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:
Hi,
I've recently encountered a problem with removing disks from iscsi domain in my test lab - just cant remove any. Remove operation fails with message "User admin@internal failed to initiate removing of disk pg-slave1_opt from domain iscsi-store" and re-elections of SPM. After that - disk is marked as "illegal" in ovirt webinterface, however, it is _in fact_ removed from storage - lvdisplay doesn't show it and free space is updated correctly. And this happens with just about every disk/vm i try to remove.
ovirt 3.2.2-el6 centos 6.4 vdsm-4.10.3-17.el6 lvm2-2.02.98-9.el6
Any tips? logs in attach
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dafna Ron
----- Original Message -----
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo
The log issue already solved in v4.11.0~380. commit ad916c79e2b0959dea20dd19a21b99bc702d65ca Author: Eduardo Warszawski <ewarszaw@redhat.com> Date: Mon Dec 17 14:32:51 2012 +0200 Fix typo in negative flow log in blockSD.rmDCImgDir(). Related to BZ#885489. Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 Signed-off-by: Eduardo <ewarszaw@redhat.com> Reviewed-on: http://gerrit.ovirt.org/10153 Reviewed-by: Yeela Kaplan <ykaplan@redhat.com> Reviewed-by: Ayal Baron <abaron@redhat.com> Tested-by: Dan Kenigsberg <danken@redhat.com> diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index b5447cd..22a434b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): try: os.rmdir(imgPath) except OSError: - self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) else: self.log.debug("removed image dir: %s", imgPath) return imgPath
On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет:
from the logs it appears to be a quota issue. do you have quota enabled?
Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: > Hi, > > I've recently encountered a problem with removing disks from iscsi > domain in my test lab - just cant remove any. > Remove operation fails with message "User admin@internal failed to > initiate removing of disk pg-slave1_opt from domain iscsi-store" and > re-elections of SPM. After that - disk is marked as "illegal" in > ovirt > webinterface, however, it is _in fact_ removed from storage - > lvdisplay doesn't show it and free space is updated correctly. > And this happens with just about every disk/vm i try to remove. > > ovirt 3.2.2-el6 > centos 6.4 > vdsm-4.10.3-17.el6 > lvm2-2.02.98-9.el6 > > Any tips? logs in attach > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
-- Dafna Ron
Thanks a lot! Patch did the job, a few more disks were deleted successfully. One last question - how do i remove stale records of disks in "illegal" state? Yuriy Demchenko On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
----- Original Message -----
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo
The log issue already solved in v4.11.0~380.
commit ad916c79e2b0959dea20dd19a21b99bc702d65ca Author: Eduardo Warszawski <ewarszaw@redhat.com> Date: Mon Dec 17 14:32:51 2012 +0200
Fix typo in negative flow log in blockSD.rmDCImgDir().
Related to BZ#885489.
Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 Signed-off-by: Eduardo <ewarszaw@redhat.com> Reviewed-on: http://gerrit.ovirt.org/10153 Reviewed-by: Yeela Kaplan <ykaplan@redhat.com> Reviewed-by: Ayal Baron <abaron@redhat.com> Tested-by: Dan Kenigsberg <danken@redhat.com>
diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index b5447cd..22a434b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): try: os.rmdir(imgPath) except OSError: - self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) else: self.log.debug("removed image dir: %s", imgPath) return imgPath
On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: 60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote:
21.08.2013 19:18, Dafna Ron пишет: > from the logs it appears to be a quota issue. > do you have quota enabled? > Yes, quota in "enforced" mode. But VMs/disks in question belongs to "unlimited" quota (defined quota policy with no limits on storage/cpu/ram)
> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >> Hi, >> >> I've recently encountered a problem with removing disks from iscsi >> domain in my test lab - just cant remove any. >> Remove operation fails with message "User admin@internal failed to >> initiate removing of disk pg-slave1_opt from domain iscsi-store" and >> re-elections of SPM. After that - disk is marked as "illegal" in >> ovirt >> webinterface, however, it is _in fact_ removed from storage - >> lvdisplay doesn't show it and free space is updated correctly. >> And this happens with just about every disk/vm i try to remove. >> >> ovirt 3.2.2-el6 >> centos 6.4 >> vdsm-4.10.3-17.el6 >> lvm2-2.02.98-9.el6 >> >> Any tips? logs in attach >> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >
-- Dafna Ron
deleting illegal objects from UI was added to the latest ovirt/vdsm are you getting any errors when trying to delete them now? On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
Thanks a lot! Patch did the job, a few more disks were deleted successfully.
One last question - how do i remove stale records of disks in "illegal" state?
Yuriy Demchenko
On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
----- Original Message -----
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo
The log issue already solved in v4.11.0~380.
commit ad916c79e2b0959dea20dd19a21b99bc702d65ca Author: Eduardo Warszawski <ewarszaw@redhat.com> Date: Mon Dec 17 14:32:51 2012 +0200
Fix typo in negative flow log in blockSD.rmDCImgDir(). Related to BZ#885489. Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 Signed-off-by: Eduardo <ewarszaw@redhat.com> Reviewed-on: http://gerrit.ovirt.org/10153 Reviewed-by: Yeela Kaplan <ykaplan@redhat.com> Reviewed-by: Ayal Baron <abaron@redhat.com> Tested-by: Dan Kenigsberg <danken@redhat.com>
diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index b5447cd..22a434b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): try: os.rmdir(imgPath) except OSError: - self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) else: self.log.debug("removed image dir: %s", imgPath) return imgPath
On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40
85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43
5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote:
there is a is an exception in the log related to a quota calculation
2013-08-21 17:52:32,694 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd ateQuotaCache: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.7.0 _25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:
60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
ava:557) [quartz.jar:] Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal lback; bad SQL grammar [select * from calculateallstorageusage()]; nested excep tion is org.postgresql.util.PSQLException: ERROR: column "quota_limitation.quota _id" must appear in the GROUP BY clause or be used in an aggregate function Where: PL/pgSQL function "calculateallstorageusage" line 3 at RETURN QUERY
in any case this is a bug. I'm adding Doron to this mail, perhaps this was reported in the past and already solved in later versions. if not it should be reported and fixed.
Dafna
If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
On 08/21/2013 05:26 PM, Yura Demchenko wrote: > 21.08.2013 19:18, Dafna Ron пишет: >> from the logs it appears to be a quota issue. >> do you have quota enabled? >> > Yes, quota in "enforced" mode. But VMs/disks in question belongs to > "unlimited" quota (defined quota policy with no limits on > storage/cpu/ram) > >> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>> Hi, >>> >>> I've recently encountered a problem with removing disks from >>> iscsi >>> domain in my test lab - just cant remove any. >>> Remove operation fails with message "User admin@internal >>> failed to >>> initiate removing of disk pg-slave1_opt from domain >>> iscsi-store" and >>> re-elections of SPM. After that - disk is marked as "illegal" in >>> ovirt >>> webinterface, however, it is _in fact_ removed from storage - >>> lvdisplay doesn't show it and free space is updated correctly. >>> And this happens with just about every disk/vm i try to remove. >>> >>> ovirt 3.2.2-el6 >>> centos 6.4 >>> vdsm-4.10.3-17.el6 >>> lvm2-2.02.98-9.el6 >>> >>> Any tips? logs in attach >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>
-- Dafna Ron
-- Dafna Ron
If by "latest" you mean head/nightly - i'm not using it yet, I'm using packets from stable el6 repo, so my test env match production env. Errors i get:
User admin@internal failed to initiate removing of disk virt-dhcp-3_Disk1 from domain iscsi-store
Thread-1196::ERROR::2013-08-22 14:00:57,736::task::833::TaskManager.Task::(_setError) Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) ImageDoesNotExistInSD: Image does not exist in domain: 'image=dde47a09-81d9-4592-87e4-92398ea834ce, domain=d786e2d5-05ab-4da6-95fc-1af791a3c113'
Maybe just do some sql-query to remove them from DB? Yuriy Demchenko On 08/22/2013 01:55 PM, Dafna Ron wrote:
deleting illegal objects from UI was added to the latest ovirt/vdsm are you getting any errors when trying to delete them now?
On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
Thanks a lot! Patch did the job, a few more disks were deleted successfully.
One last question - how do i remove stale records of disks in "illegal" state?
Yuriy Demchenko
On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
----- Original Message -----
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo
The log issue already solved in v4.11.0~380.
commit ad916c79e2b0959dea20dd19a21b99bc702d65ca Author: Eduardo Warszawski <ewarszaw@redhat.com> Date: Mon Dec 17 14:32:51 2012 +0200
Fix typo in negative flow log in blockSD.rmDCImgDir(). Related to BZ#885489. Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 Signed-off-by: Eduardo <ewarszaw@redhat.com> Reviewed-on: http://gerrit.ovirt.org/10153 Reviewed-by: Yeela Kaplan <ykaplan@redhat.com> Reviewed-by: Ayal Baron <abaron@redhat.com> Tested-by: Dan Kenigsberg <danken@redhat.com>
diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index b5447cd..22a434b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): try: os.rmdir(imgPath) except OSError: - self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) else: self.log.debug("removed image dir: %s", imgPath) return imgPath
On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case:
Thread-60725::DEBUG::2013-08-22 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40
85-8900-da271affe831. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/dfefc573-de85-4085-8900-da271affe831' Thread-60725::WARNING::2013-08-22 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43
5d-bdc7-713b445a2326. [Errno 2] No such file or directory: '/rhev/data-center/mn t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' Thread-60725::DEBUG::2013-08-22 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) removed: [] Thread-60725::ERROR::2013-08-22 10:37:45,549::task::833::TaskManager.Task::(_set Error) Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage dom.deleteImage(sdUUID, imgUUID, volsByImg) File "/usr/share/vdsm/storage/blockSD.py", line 957, in deleteImage self.rmDCImgDir(imgUUID, volsImgs) File "/usr/share/vdsm/storage/blockSD.py", line 943, in rmDCImgDir self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1068, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote:
On 08/21/2013 04:10 PM, Dafna Ron wrote: > there is a is an exception in the log related to a quota > calculation > > 2013-08-21 17:52:32,694 ERROR > [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] > (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled > method upd > ateQuotaCache: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown > Source) > [:1.7.0 _25] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > > sorImpl.java:43) [rt.jar:1.7.0_25] > at java.lang.reflect.Method.invoke(Method.java:606) > [rt.jar:1.7.0_25] > at > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: > > 60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > [quartz.jar:] > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j > > ava:557) [quartz.jar:] > Caused by: org.springframework.jdbc.BadSqlGrammarException: > PreparedStatementCal lback; bad SQL grammar [select * from > calculateallstorageusage()]; nested excep tion is > org.postgresql.util.PSQLException: ERROR: column > "quota_limitation.quota > _id" must appear in the GROUP BY clause or be used in an aggregate > function > Where: PL/pgSQL function "calculateallstorageusage" line 3 at > RETURN QUERY > > > in any case this is a bug. > I'm adding Doron to this mail, perhaps this was reported in the > past > and > already solved in later versions. > if not it should be reported and fixed. > > Dafna > > If I'm not mistaken, it looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=905891
Greg
> > > On 08/21/2013 05:26 PM, Yura Demchenko wrote: >> 21.08.2013 19:18, Dafna Ron пишет: >>> from the logs it appears to be a quota issue. >>> do you have quota enabled? >>> >> Yes, quota in "enforced" mode. But VMs/disks in question >> belongs to >> "unlimited" quota (defined quota policy with no limits on >> storage/cpu/ram) >> >>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>> Hi, >>>> >>>> I've recently encountered a problem with removing disks from >>>> iscsi >>>> domain in my test lab - just cant remove any. >>>> Remove operation fails with message "User admin@internal >>>> failed to >>>> initiate removing of disk pg-slave1_opt from domain >>>> iscsi-store" and >>>> re-elections of SPM. After that - disk is marked as "illegal" in >>>> ovirt >>>> webinterface, however, it is _in fact_ removed from storage - >>>> lvdisplay doesn't show it and free space is updated correctly. >>>> And this happens with just about every disk/vm i try to remove. >>>> >>>> ovirt 3.2.2-el6 >>>> centos 6.4 >>>> vdsm-4.10.3-17.el6 >>>> lvm2-2.02.98-9.el6 >>>> >>>> Any tips? logs in attach >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >
-- Dafna Ron
there are actually a lot of tables that have a reference to the image in the db. I am adding Liron who might be able to give you the patch needed for deleting these images or tell you in what rpm they were merged to. On 08/22/2013 11:06 AM, Yuriy Demchenko wrote:
If by "latest" you mean head/nightly - i'm not using it yet, I'm using packets from stable el6 repo, so my test env match production env.
Errors i get:
User admin@internal failed to initiate removing of disk virt-dhcp-3_Disk1 from domain iscsi-store
Thread-1196::ERROR::2013-08-22 14:00:57,736::task::833::TaskManager.Task::(_setError) Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1434, in deleteImage raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) ImageDoesNotExistInSD: Image does not exist in domain: 'image=dde47a09-81d9-4592-87e4-92398ea834ce, domain=d786e2d5-05ab-4da6-95fc-1af791a3c113'
Maybe just do some sql-query to remove them from DB?
Yuriy Demchenko
On 08/22/2013 01:55 PM, Dafna Ron wrote:
deleting illegal objects from UI was added to the latest ovirt/vdsm are you getting any errors when trying to delete them now?
On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
Thanks a lot! Patch did the job, a few more disks were deleted successfully.
One last question - how do i remove stale records of disks in "illegal" state?
Yuriy Demchenko
On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:
----- Original Message -----
you said that the disks are deleted although an exception is given but the engine is reporting the delete as failed. I think that I remember a bug reported and fixed on that as well but I can't seem to find it. Adding Ayal and Eduardo
The log issue already solved in v4.11.0~380.
commit ad916c79e2b0959dea20dd19a21b99bc702d65ca Author: Eduardo Warszawski <ewarszaw@redhat.com> Date: Mon Dec 17 14:32:51 2012 +0200
Fix typo in negative flow log in blockSD.rmDCImgDir(). Related to BZ#885489. Change-Id: I951e582acc86e08d709da4249084015660fc4ea0 Signed-off-by: Eduardo <ewarszaw@redhat.com> Reviewed-on: http://gerrit.ovirt.org/10153 Reviewed-by: Yeela Kaplan <ykaplan@redhat.com> Reviewed-by: Ayal Baron <abaron@redhat.com> Tested-by: Dan Kenigsberg <danken@redhat.com>
diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index b5447cd..22a434b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain): try: os.rmdir(imgPath) except OSError: - self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) else: self.log.debug("removed image dir: %s", imgPath) return imgPath
On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to my issue: I tried to remove another disk and this time there were no quota errors in engine.log New logs in attach.
What catches my eye in logs is this errors, but maybe that's not the root of case: > Thread-60725::DEBUG::2013-08-22 > 10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation > 'lvm invalidate operation' released the operation mutex > Thread-60725::WARNING::2013-08-22 > 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) > Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da > 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 > > 85-8900-da271affe831. [Errno 2] No such file or directory: > '/rhev/data-center/mn > t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da > > 7937fbdfad/dfefc573-de85-4085-8900-da271affe831' > Thread-60725::WARNING::2013-08-22 > 10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) > Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da > 6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 > > 5d-bdc7-713b445a2326. [Errno 2] No such file or directory: > '/rhev/data-center/mn > t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da > > 7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326' > Thread-60725::DEBUG::2013-08-22 > 10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) > removed: [] > Thread-60725::ERROR::2013-08-22 > 10:37:45,549::task::833::TaskManager.Task::(_set Error) > Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error > Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 42, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1460, in deleteImage > dom.deleteImage(sdUUID, imgUUID, volsByImg) > File "/usr/share/vdsm/storage/blockSD.py", line 957, in > deleteImage > self.rmDCImgDir(imgUUID, volsImgs) > File "/usr/share/vdsm/storage/blockSD.py", line 943, in > rmDCImgDir > self.log.warning("Can't rmdir %s. %s", imgPath, exc_info=True) > File "/usr/lib64/python2.6/logging/__init__.py", line 1068, > in warning > self._log(WARNING, msg, args, **kwargs) > File "/usr/lib64/python2.6/logging/__init__.py", line 1173, > in _log > self.handle(record) > File "/usr/lib64/python2.6/logging/__init__.py", line 1183, > in handle > self.callHandlers(record) > File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in > callHandlers > hdlr.handle(record) > File "/usr/lib64/python2.6/logging/__init__.py", line 679, in > handle > self.emit(record) > File "/usr/lib64/python2.6/logging/handlers.py", line 780, in > emit > msg = self.format(record) > File "/usr/lib64/python2.6/logging/__init__.py", line 654, in > format > return fmt.format(record) > File "/usr/lib64/python2.6/logging/__init__.py", line 436, in > format > record.message = record.getMessage() > File "/usr/lib64/python2.6/logging/__init__.py", line 306, in > getMessage > msg = msg % self.args > TypeError: not enough arguments for format string
Yuriy Demchenko
On 08/22/2013 04:11 AM, Greg Padgett wrote: > On 08/21/2013 04:10 PM, Dafna Ron wrote: >> there is a is an exception in the log related to a quota >> calculation >> >> 2013-08-21 17:52:32,694 ERROR >> [org.ovirt.engine.core.utils.timer.SchedulerUtilQu artzImpl] >> (DefaultQuartzScheduler_Worker-7) failed to invoke sceduled >> method upd >> ateQuotaCache: java.lang.reflect.InvocationTargetException >> at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown >> Source) >> [:1.7.0 _25] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >> >> sorImpl.java:43) [rt.jar:1.7.0_25] >> at java.lang.reflect.Method.invoke(Method.java:606) >> [rt.jar:1.7.0_25] >> at >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java: >> >> 60) [scheduler.jar:] >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> [quartz.jar:] >> at >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j >> >> ava:557) [quartz.jar:] >> Caused by: org.springframework.jdbc.BadSqlGrammarException: >> PreparedStatementCal lback; bad SQL grammar [select * from >> calculateallstorageusage()]; nested excep tion is >> org.postgresql.util.PSQLException: ERROR: column >> "quota_limitation.quota >> _id" must appear in the GROUP BY clause or be used in an aggregate >> function >> Where: PL/pgSQL function "calculateallstorageusage" line 3 at >> RETURN QUERY >> >> >> in any case this is a bug. >> I'm adding Doron to this mail, perhaps this was reported in the >> past >> and >> already solved in later versions. >> if not it should be reported and fixed. >> >> Dafna >> >> > If I'm not mistaken, it looks like this bug: > > https://bugzilla.redhat.com/show_bug.cgi?id=905891 > > Greg > >> >> >> On 08/21/2013 05:26 PM, Yura Demchenko wrote: >>> 21.08.2013 19:18, Dafna Ron пишет: >>>> from the logs it appears to be a quota issue. >>>> do you have quota enabled? >>>> >>> Yes, quota in "enforced" mode. But VMs/disks in question >>> belongs to >>> "unlimited" quota (defined quota policy with no limits on >>> storage/cpu/ram) >>> >>>> On 08/21/2013 03:20 PM, Yuriy Demchenko wrote: >>>>> Hi, >>>>> >>>>> I've recently encountered a problem with removing disks from >>>>> iscsi >>>>> domain in my test lab - just cant remove any. >>>>> Remove operation fails with message "User admin@internal >>>>> failed to >>>>> initiate removing of disk pg-slave1_opt from domain >>>>> iscsi-store" and >>>>> re-elections of SPM. After that - disk is marked as >>>>> "illegal" in >>>>> ovirt >>>>> webinterface, however, it is _in fact_ removed from storage - >>>>> lvdisplay doesn't show it and free space is updated correctly. >>>>> And this happens with just about every disk/vm i try to remove. >>>>> >>>>> ovirt 3.2.2-el6 >>>>> centos 6.4 >>>>> vdsm-4.10.3-17.el6 >>>>> lvm2-2.02.98-9.el6 >>>>> >>>>> Any tips? logs in attach >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>
-- Dafna Ron
-- Dafna Ron
participants (6)
-
Dafna Ron -
Doron Fediuck -
Eduardo Warszawski -
Greg Padgett -
Yura Demchenko -
Yuriy Demchenko