Seeing failed live migrations and these errors in the vdsm logs with latest VDSM/Engine master. Hosts are EL6.4

Thread-1306::ERROR::2013-09-23 16:02:42,422::BindingXMLRPC::993::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 211, in vmDestroy return vm.destroy() File "/usr/share/vdsm/API.py", line 323, in destroy res = v.destroy() File "/usr/share/vdsm/vm.py", line 4326, in destroy response = self.releaseVm() File "/usr/share/vdsm/vm.py", line 4292, in releaseVm self._cleanup() File "/usr/share/vdsm/vm.py", line 2750, in _cleanup self._cleanupDrives() File "/usr/share/vdsm/vm.py", line 2482, in _cleanupDrives drive, exc_info=True) File "/usr/lib64/python2.6/logging/__init__.py", line 1329, in error self.logger.error(msg, *args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1082, in error self._log(ERROR, msg, args, **kwargs) File "/usr/lib64/python2.6/logging/__init__.py", line 1173, in _log self.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1183, in handle self.callHandlers(record) File "/usr/lib64/python2.6/logging/__init__.py", line 1220, in callHandlers hdlr.handle(record) File "/usr/lib64/python2.6/logging/__init__.py", line 679, in handle self.emit(record) File "/usr/lib64/python2.6/logging/handlers.py", line 780, in emit msg = self.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 654, in format return fmt.format(record) File "/usr/lib64/python2.6/logging/__init__.py", line 436, in format record.message = record.getMessage() File "/usr/lib64/python2.6/logging/__init__.py", line 306, in getMessage msg = msg % self.args File "/usr/share/vdsm/vm.py", line 107, in __str__ if not a.startswith('__')] File "/usr/share/vdsm/vm.py", line 1344, in hasVolumeLeases if self.shared != DRIVE_SHARED_TYPE.EXCLUSIVE: AttributeError: 'Drive' object has no attribute 'shared' - DHC

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

----- Original Message ----- > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > To: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt; > Cc: &quot;&lt;users(a)ovirt.org&gt;&quot; <users(a)ovirt.org&gt;, vdsm-devel(a)fedorahosted.org, fsimonce(a)redhat.com, abaron(a)redhat.com > Sent: Tuesday, September 24, 2013 11:44:48 AM > Subject: Re: [Users] vdsm live migration errors in latest master > > On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote: > > Seeing failed live migrations and these errors in the vdsm logs with latest > > VDSM/Engine master. > > Hosts are EL6.4 > > Thanks for posting this report. > > The log is from the source of migration, right? > Could you trace the history of the hosts of this VM? Could it be that it > was started on an older version of vdsm (say ovirt-3.3.0) and then (due > to migration or vdsm upgrade) got into a host with a much newer vdsm? > > Would you share the vmCreate (or vmMigrationCreate) line for this Vm in > your log? I smells like an unintended regression of > http://gerrit.ovirt.org/17714 > vm: extend shared property to support locking > > solving it may not be trivial, as we should not call > _normalizeDriveSharedAttribute() automatically on migration destination, > as it may well still be apart of a 3.3 clusterLevel. > > Also, migration from vdsm with extended shared property, to an ovirt 3.3 > vdsm is going to explode (in a different way), since the destination > does not expect the extended values. > > Federico, do we have a choice but to revert that patch, and use > something like "shared3" property instead? I filed a bug at: https://bugzilla.redhat.com/show_bug.cgi?id=1011608 A possible fix could be: http://gerrit.ovirt.org/#/c/19509

----- Original Message ----- > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt; > Cc: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt;, "users" <users(a)ovirt.org&gt;, vdsm-devel(a)fedorahosted.org, > abaron(a)redhat.com > Sent: Thursday, September 26, 2013 1:38:15 AM > Subject: Re: [Users] vdsm live migration errors in latest master > > On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote: > > ----- Original Message ----- > > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > > > To: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt; > > > Cc: &quot;&lt;users(a)ovirt.org&gt;&quot; <users(a)ovirt.org&gt;, vdsm-devel(a)fedorahosted.org, > > > fsimonce(a)redhat.com, abaron(a)redhat.com > > > Sent: Tuesday, September 24, 2013 11:44:48 AM > > > Subject: Re: [Users] vdsm live migration errors in latest master > > > > > > On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote: > > > > Seeing failed live migrations and these errors in the vdsm logs with > > > > latest > > > > VDSM/Engine master. > > > > Hosts are EL6.4 > > > > > > Thanks for posting this report. > > > > > > The log is from the source of migration, right? > > > Could you trace the history of the hosts of this VM? Could it be that it > > > was started on an older version of vdsm (say ovirt-3.3.0) and then (due > > > to migration or vdsm upgrade) got into a host with a much newer vdsm? > > > > > > Would you share the vmCreate (or vmMigrationCreate) line for this Vm in > > > your log? I smells like an unintended regression of > > > http://gerrit.ovirt.org/17714 > > > vm: extend shared property to support locking > > > > > > solving it may not be trivial, as we should not call > > > _normalizeDriveSharedAttribute() automatically on migration destination, > > > as it may well still be apart of a 3.3 clusterLevel. > > > > > > Also, migration from vdsm with extended shared property, to an ovirt 3.3 > > > vdsm is going to explode (in a different way), since the destination > > > does not expect the extended values. > > > > > > Federico, do we have a choice but to revert that patch, and use > > > something like "shared3" property instead? > > > > I filed a bug at: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1011608 > > > > A possible fix could be: > > > > http://gerrit.ovirt.org/#/c/19509 > > Beyond this, we must make sure that on Engine side, the extended shared > values would be used only for clusterLevel 3.4 and above. > > Are the extended shared values already used by Engine? Yes. That's the idea. Actually to be fair, the second case you mentioned (migrating from extended shared property to old vdsm) it wouldn't have been possible I suppose (the issue here is that Dead Horse has one or more hosts running on master instead of 3.3). The extended shared property would have appeared only in 3.4 and to allow the migration you would have had to upgrade all the nodes. But anyway since we were also talking about a new 3.3.1 barnch I just went ahead and covered all cases.

----- Original Message ----- > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt; > Cc: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt;, "users" <users(a)ovirt.org&gt;, vdsm-devel(a)fedorahosted.org, > abaron(a)redhat.com > Sent: Thursday, September 26, 2013 2:09:15 PM > Subject: Re: [Users] vdsm live migration errors in latest master > > On Thu, Sep 26, 2013 at 05:35:46AM -0400, Federico Simoncelli wrote: > > ----- Original Message ----- > > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > > > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt; > > > Cc: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt;, "users" > > > <users(a)ovirt.org&gt;, vdsm-devel(a)fedorahosted.org, > > > abaron(a)redhat.com > > > Sent: Thursday, September 26, 2013 1:38:15 AM > > > Subject: Re: [Users] vdsm live migration errors in latest master > > > > > > On Tue, Sep 24, 2013 at 12:04:14PM -0400, Federico Simoncelli wrote: > > > > ----- Original Message ----- > > > > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > > > > > To: "Dead Horse" <deadhorseconsulting(a)gmail.com&gt; > > > > > Cc: &quot;&lt;users(a)ovirt.org&gt;&quot; <users(a)ovirt.org&gt;, > > > > > vdsm-devel(a)fedorahosted.org, > > > > > fsimonce(a)redhat.com, abaron(a)redhat.com > > > > > Sent: Tuesday, September 24, 2013 11:44:48 AM > > > > > Subject: Re: [Users] vdsm live migration errors in latest master > > > > > > > > > > On Mon, Sep 23, 2013 at 04:05:34PM -0500, Dead Horse wrote: > > > > > > Seeing failed live migrations and these errors in the vdsm logs > > > > > > with > > > > > > latest > > > > > > VDSM/Engine master. > > > > > > Hosts are EL6.4 > > > > > > > > > > Thanks for posting this report. > > > > > > > > > > The log is from the source of migration, right? > > > > > Could you trace the history of the hosts of this VM? Could it be that > > > > > it > > > > > was started on an older version of vdsm (say ovirt-3.3.0) and then > > > > > (due > > > > > to migration or vdsm upgrade) got into a host with a much newer vdsm? > > > > > > > > > > Would you share the vmCreate (or vmMigrationCreate) line for this Vm > > > > > in > > > > > your log? I smells like an unintended regression of > > > > > http://gerrit.ovirt.org/17714 > > > > > vm: extend shared property to support locking > > > > > > > > > > solving it may not be trivial, as we should not call > > > > > _normalizeDriveSharedAttribute() automatically on migration > > > > > destination, > > > > > as it may well still be apart of a 3.3 clusterLevel. > > > > > > > > > > Also, migration from vdsm with extended shared property, to an ovirt > > > > > 3.3 > > > > > vdsm is going to explode (in a different way), since the destination > > > > > does not expect the extended values. > > > > > > > > > > Federico, do we have a choice but to revert that patch, and use > > > > > something like "shared3" property instead? > > > > > > > > I filed a bug at: > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1011608 > > > > > > > > A possible fix could be: > > > > > > > > http://gerrit.ovirt.org/#/c/19509 > > > > > > Beyond this, we must make sure that on Engine side, the extended shared > > > values would be used only for clusterLevel 3.4 and above. > > > > > > Are the extended shared values already used by Engine? > > > > Yes. That's the idea. Actually to be fair, the second case you mentioned > > (migrating from extended shared property to old vdsm) it wouldn't have been > > possible I suppose (the issue here is that Dead Horse has one or more > > hosts running on master instead of 3.3). The extended shared property would > > have appeared only in 3.4 and to allow the migration you would have had to > > upgrade all the nodes. > > > > But anyway since we were also talking about a new 3.3.1 barnch I just went > > ahead and covered all cases. > > I do not see how the 3.3.1 branch is relevant to the discussion, as its > Vdsm is NOT going to support clusterLevel 3.4. That is what I was referring to. If 3.3.1 was 3.3.0 + backported patches then we just wouldn't backport the extended shared attributes patch and that's it. But from what I understood 3.3.1 will be rebased on master (where instead we have the extended shared attributes) and that is why we have to cover both migration direction cases (instead of just the simple getattr one). > Pardon my slowliness, but would you confirm that this feature is to be > used only on clusterLevel 3.4 and above? If so, I'm +2ing your patch. Yes, the extended attributes will be used in the hosted engine and cluster level 3.4. But what the engine does is not relevant to +2ing correct vdsm patches.