[Users] vdsm live migration errors in latest master
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4 Thread-1306::ERROR::2013-09-23 16:02:42,422::BindingXMLRPC::993::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 211, in vmDestroy return vm.destroy() File "/usr/share/vdsm/API.py", line 323, in destroy res = v.destroy() File "/usr/share/vdsm/vm.py", line 4326, in destroy response = self.releaseVm() File "/usr/share/vdsm/vm.py", line 4292, in releaseVm self._cleanup() File "/usr/share/vdsm/vm.py", line 2750, in _cleanup self._cleanupDrives() File "/usr/share/vdsm/vm.py", line 2482, in _cleanupDrives drive, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1329, in error self.logger.error(msg, *args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args File "/usr/share/vdsm/vm.py", line 107, in __str__ if not a.startswith('__')] File "/usr/share/vdsm/vm.py", line 1344, in hasVolumeLeases if self.shared != DRIVE_SHARED_TYPE.EXCLUSIVE: AttributeError: 'Drive' object has no attribute 'shared' - DHC
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report. The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm? Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel. Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values. Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
Thread-1306::ERROR::2013-09-23 16:02:42,422::BindingXMLRPC::993::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 211, in vmDestroy return vm.destroy() File "/usr/share/vdsm/API.py", line 323, in destroy res = v.destroy() File "/usr/share/vdsm/vm.py", line 4326, in destroy response = self.releaseVm() File "/usr/share/vdsm/vm.py", line 4292, in releaseVm self._cleanup() File "/usr/share/vdsm/vm.py", line 2750, in _cleanup self._cleanupDrives() File "/usr/share/vdsm/vm.py", line 2482, in _cleanupDrives drive, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1329, in error self.logger.error(msg, *args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args File "/usr/share/vdsm/vm.py", line 107, in __str__ if not a.startswith('__')] File "/usr/share/vdsm/vm.py", line 1344, in hasVolumeLeases if self.shared != DRIVE_SHARED_TYPE.EXCLUSIVE: AttributeError: 'Drive' object has no attribute 'shared'
- DHC
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at: https://bugzilla.redhat.com/show_bug.cgi?id=1011608 A possible fix could be: http://gerrit.ovirt.org/#/c/19509 -- Federico
On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at:
https://bugzilla.redhat.com/show_bug.cgi?id=1011608
A possible fix could be:
Beyond this, we must make sure that on Engine side, the extended shared values would be used only for clusterLevel 3.4 and above. Are the extended shared values already used by Engine?
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 1:38:15 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at:
https://bugzilla.redhat.com/show_bug.cgi?id=1011608
A possible fix could be:
Beyond this, we must make sure that on Engine side, the extended shared values would be used only for clusterLevel 3.4 and above.
Are the extended shared values already used by Engine?
Yes. That's the idea. Actually to be fair, the second case you mentioned (migrating from extended shared property to old vdsm) it wouldn't have been possible I suppose (the issue here is that Dead Horse has one or more hosts running on master instead of 3.3). The extended shared property would have appeared only in 3.4 and to allow the migration you would have had to upgrade all the nodes. But anyway since we were also talking about a new 3.3.1 barnch I just went ahead and covered all cases. -- Federico
On Thu, Sep 26, 2013 at 05:35:46AM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 1:38:15 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at:
https://bugzilla.redhat.com/show_bug.cgi?id=1011608
A possible fix could be:
Beyond this, we must make sure that on Engine side, the extended shared values would be used only for clusterLevel 3.4 and above.
Are the extended shared values already used by Engine?
Yes. That's the idea. Actually to be fair, the second case you mentioned (migrating from extended shared property to old vdsm) it wouldn't have been possible I suppose (the issue here is that Dead Horse has one or more hosts running on master instead of 3.3). The extended shared property would have appeared only in 3.4 and to allow the migration you would have had to upgrade all the nodes.
But anyway since we were also talking about a new 3.3.1 barnch I just went ahead and covered all cases.
I do not see how the 3.3.1 branch is relevant to the discussion, as its Vdsm is NOT going to support clusterLevel 3.4. Pardon my slowliness, but would you confirm that this feature is to be used only on clusterLevel 3.4 and above? If so, I'm +2ing your patch. Dan.
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 2:09:15 PM Subject: Re: [Users] vdsm live migration errors in latest master
On Thu, Sep 26, 2013 at 05:35:46AM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 1:38:15 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote:
Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at:
https://bugzilla.redhat.com/show_bug.cgi?id=1011608
A possible fix could be:
Beyond this, we must make sure that on Engine side, the extended shared values would be used only for clusterLevel 3.4 and above.
Are the extended shared values already used by Engine?
Yes. That's the idea. Actually to be fair, the second case you mentioned (migrating from extended shared property to old vdsm) it wouldn't have been possible I suppose (the issue here is that Dead Horse has one or more hosts running on master instead of 3.3). The extended shared property would have appeared only in 3.4 and to allow the migration you would have had to upgrade all the nodes.
But anyway since we were also talking about a new 3.3.1 barnch I just went ahead and covered all cases.
I do not see how the 3.3.1 branch is relevant to the discussion, as its Vdsm is NOT going to support clusterLevel 3.4.
That is what I was referring to. If 3.3.1 was 3.3.0 + backported patches then we just wouldn't backport the extended shared attributes patch and that's it. But from what I understood 3.3.1 will be rebased on master (where instead we have the extended shared attributes) and that is why we have to cover both migration direction cases (instead of just the simple getattr one).
Pardon my slowliness, but would you confirm that this feature is to be used only on clusterLevel 3.4 and above? If so, I'm +2ing your patch.
Yes, the extended attributes will be used in the hosted engine and cluster level 3.4. But what the engine does is not relevant to +2ing correct vdsm patches. -- Federico
On Thu, Sep 26, 2013 at 12:14:06PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 2:09:15 PM Subject: Re: [Users] vdsm live migration errors in latest master
On Thu, Sep 26, 2013 at 05:35:46AM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" <users@ovirt.org>, vdsm-devel@fedorahosted.org, abaron@redhat.com Sent: Thursday, September 26, 2013 1:38:15 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Dead Horse" <deadhorseconsulting@gmail.com> Cc: "<users@ovirt.org>" <users@ovirt.org>, vdsm-devel@fedorahosted.org, fsimonce@redhat.com, abaron@redhat.com Sent: Tuesday, September 24, 2013 11:44:48 AM Subject: Re: [Users] vdsm live migration errors in latest master
On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote: > Seeing failed live migrations and these errors in the vdsm logs > with > latest > VDSM/Engine master. > Hosts are EL6.4
Thanks for posting this report.
The log is from the source of migration, right? Could you trace the history of the hosts of this VM? Could it be that it was started on an older version of vdsm (say ovirt-3.3.0) and then (due to migration or vdsm upgrade) got into a host with a much newer vdsm?
Would you share the vmCreate (or vmMigrationCreate) line for this Vm in your log? I smells like an unintended regression of http://gerrit.ovirt.org/17714 vm: extend shared property to support locking
solving it may not be trivial, as we should not call _normalizeDriveSharedAttribute() automatically on migration destination, as it may well still be apart of a 3.3 clusterLevel.
Also, migration from vdsm with extended shared property, to an ovirt 3.3 vdsm is going to explode (in a different way), since the destination does not expect the extended values.
Federico, do we have a choice but to revert that patch, and use something like "shared3" property instead?
I filed a bug at:
https://bugzilla.redhat.com/show_bug.cgi?id=1011608
A possible fix could be:
Beyond this, we must make sure that on Engine side, the extended shared values would be used only for clusterLevel 3.4 and above.
Are the extended shared values already used by Engine?
Yes. That's the idea. Actually to be fair, the second case you mentioned (migrating from extended shared property to old vdsm) it wouldn't have been possible I suppose (the issue here is that Dead Horse has one or more hosts running on master instead of 3.3). The extended shared property would have appeared only in 3.4 and to allow the migration you would have had to upgrade all the nodes.
But anyway since we were also talking about a new 3.3.1 barnch I just went ahead and covered all cases.
I do not see how the 3.3.1 branch is relevant to the discussion, as its Vdsm is NOT going to support clusterLevel 3.4.
That is what I was referring to.
If 3.3.1 was 3.3.0 + backported patches then we just wouldn't backport the extended shared attributes patch and that's it. But from what I understood 3.3.1 will be rebased on master (where instead we have the extended shared attributes) and that is why we have to cover both migration direction cases (instead of just the simple getattr one).
Pardon my slowliness, but would you confirm that this feature is to be used only on clusterLevel 3.4 and above? If so, I'm +2ing your patch.
Yes, the extended attributes will be used in the hosted engine and cluster level 3.4. But what the engine does is not relevant to +2ing correct vdsm patches.
IMHO the patch is correct only if all clients understned that this is a clusterLevel 3.4 feature. Now that that's clear to me. Ack by me.
participants (3)
-
Dan Kenigsberg -
Dead Horse -
Federico Simoncelli