3:30 p.m.
VM migrations are failing with latest master engine and vdsm
logs attached from engine and both hosts
Hosts are EL 6.4 with latest master VDSM
Additionally the hosts seem to lose connection with their storage domains
(new behavior), are offline then recovered (even though not is physically
wrong).
- DHC
5:30 p.m.
On Fri, Oct 11, 2013 at 03:30:35PM -0500, Dead Horse wrote:
VM migrations are failing with latest master engine and vdsm
logs attached from engine and both hosts
Hosts are EL 6.4 with latest master VDSM
Thanks for your report!
Thread-195::ERROR::2013-10-11 15:22:39,508::vm::304::vm.Vm::(run)
vmId=`4bad94ad-c338-4ec5-8e5b-9910d58c1854`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 291, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 369, in _startUnderlyingMigration
self._abortOnError else 0),
AttributeError: 'module' object has no attribute
'VIR_MIGRATE_ABORT_ON_ERROR'
Peter, Michal, I think that VIR_MIGRATE_ABORT_ON_ERROR is expected only in
el6.5, which is still not public
Bug 972675 - Fail migration when VM get paused due to EIO
This must be reverted in vdsm or hacked in Engine (do not set abortOnError=True if libvirt
< libvirt-0.10.2-20.el6)
Additionally the hosts seem to lose connection with their storage domains
(new behavior), are offline then recovered (even though not is physically
wrong).
9:05 a.m.
On Oct 12, 2013, at 00:30 , Dan Kenigsberg <danken(a)redhat.com> wrote:
On Fri, Oct 11, 2013 at 03:30:35PM -0500, Dead Horse wrote:
> VM migrations are failing with latest master engine and vdsm
>
> logs attached from engine and both hosts
>
> Hosts are EL 6.4 with latest master VDSM
Thanks for your report!
Thread-195::ERROR::2013-10-11 15:22:39,508::vm::304::vm.Vm::(run)
vmId=`4bad94ad-c338-4ec5-8e5b-9910d58c1854`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 291, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 369, in _startUnderlyingMigration
self._abortOnError else 0),
AttributeError: 'module' object has no attribute
'VIR_MIGRATE_ABORT_ON_ERROR'
Peter, Michal, I think that VIR_MIGRATE_ABORT_ON_ERROR is expected only in
el6.5, which is still not public
Bug 972675 - Fail migration when VM get paused due to EIO
This must be reverted in vdsm or hacked in Engine (do not set abortOnError=True if
libvirt < libvirt-0.10.2-20.el6)
as of http://gerrit.ovirt.org/#/c/19312/ the flag is sent for 3.3 clusters, which is
correct
I thought you're not supposed to have EL 6.4 host in 3.3 cluster. 6.5 should work..
Thanks,
michal
>
> Additionally the hosts seem to lose connection with their storage domains
> (new behavior), are offline then recovered (even though not is physically
> wrong).
9:36 a.m.
I have been running EL 6.4 hosts in 3.3 mode for quite some time, I only
noticed this breakage in the latest master VDSM 4.13.x. Tagged vdsm
versions:
ovirt-3.3.0<http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=shortlog;h=ref...
ovirt-3.3<http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=shortlog;h=refs/...
work (if not using master which one of these should be used with 3.3
btw?)
The running version of libvirt on the hosts is:
libvirt-0.10.2-18.0.1.el6_4.14.x86_64
- DHC
On Tue, Oct 15, 2013 at 9:05 AM, Michal Skrivanek <mskrivan(a)redhat.com>wrote:
On Oct 12, 2013, at 00:30 , Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Fri, Oct 11, 2013 at 03:30:35PM -0500, Dead Horse wrote:
>> VM migrations are failing with latest master engine and vdsm
>>
>> logs attached from engine and both hosts
>>
>> Hosts are EL 6.4 with latest master VDSM
>
> Thanks for your report!
>
> Thread-195::ERROR::2013-10-11 15:22:39,508::vm::304::vm.Vm::(run)
vmId=`4bad94ad-c338-4ec5-8e5b-9910d58c1854`::Failed to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line 291, in run
> self._startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 369, in _startUnderlyingMigration
> self._abortOnError else 0),
> AttributeError: 'module' object has no attribute
'VIR_MIGRATE_ABORT_ON_ERROR'
>
>
> Peter, Michal, I think that VIR_MIGRATE_ABORT_ON_ERROR is expected only
in
> el6.5, which is still not public
> Bug 972675 - Fail migration when VM get paused due to EIO
>
> This must be reverted in vdsm or hacked in Engine (do not set
abortOnError=True if libvirt < libvirt-0.10.2-20.el6)
as of http://gerrit.ovirt.org/#/c/19312/ the flag is sent for 3.3
clusters, which is correct
I thought you're not supposed to have EL 6.4 host in 3.3 cluster. 6.5
should work..
Thanks,
michal
>
>>
>> Additionally the hosts seem to lose connection with their storage
domains
>> (new behavior), are offline then recovered (even though not is
physically
>> wrong).
>
9:42 a.m.
On Oct 15, 2013, at 16:36 , Dead Horse <deadhorseconsulting(a)gmail.com> wrote:
I have been running EL 6.4 hosts in 3.3 mode for quite some time, I
only noticed this breakage in the latest master VDSM 4.13.x. Tagged vdsm versions:
ovirt-3.3.0 and ovirt-3.3 do work (if not using master which one of these should be used
with 3.3 btw?)
it is correct
I suppose as a workaround you can use engine-config and set AbortMigrationOnError to false
for now…
the default should change to false till EL 6.5 comes out, I guess…
Thanks,
michal
The running version of libvirt on the hosts is:
libvirt-0.10.2-18.0.1.el6_4.14.x86_64
- DHC
On Tue, Oct 15, 2013 at 9:05 AM, Michal Skrivanek <mskrivan(a)redhat.com> wrote:
On Oct 12, 2013, at 00:30 , Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Fri, Oct 11, 2013 at 03:30:35PM -0500, Dead Horse wrote:
>> VM migrations are failing with latest master engine and vdsm
>>
>> logs attached from engine and both hosts
>>
>> Hosts are EL 6.4 with latest master VDSM
>
> Thanks for your report!
>
> Thread-195::ERROR::2013-10-11 15:22:39,508::vm::304::vm.Vm::(run)
vmId=`4bad94ad-c338-4ec5-8e5b-9910d58c1854`::Failed to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line 291, in run
> self._startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 369, in _startUnderlyingMigration
> self._abortOnError else 0),
> AttributeError: 'module' object has no attribute
'VIR_MIGRATE_ABORT_ON_ERROR'
>
>
> Peter, Michal, I think that VIR_MIGRATE_ABORT_ON_ERROR is expected only in
> el6.5, which is still not public
> Bug 972675 - Fail migration when VM get paused due to EIO
>
> This must be reverted in vdsm or hacked in Engine (do not set abortOnError=True if
libvirt < libvirt-0.10.2-20.el6)
as of http://gerrit.ovirt.org/#/c/19312/ the flag is sent for 3.3 clusters, which is
correct
I thought you're not supposed to have EL 6.4 host in 3.3 cluster. 6.5 should work..
Thanks,
michal
>
>>
>> Additionally the hosts seem to lose connection with their storage domains
>> (new behavior), are offline then recovered (even though not is physically
>> wrong).
>
10:55 a.m.
On Tue, Oct 15, 2013 at 04:42:11PM +0200, Michal Skrivanek wrote:
On Oct 15, 2013, at 16:36 , Dead Horse <deadhorseconsulting(a)gmail.com> wrote:
> I have been running EL 6.4 hosts in 3.3 mode for quite some time, I only noticed
this breakage in the latest master VDSM 4.13.x. Tagged vdsm versions: ovirt-3.3.0 and
ovirt-3.3 do work (if not using master which one of these should be used with 3.3 btw?)
it is correct
I suppose as a workaround you can use engine-config and set AbortMigrationOnError to
false for now…
the default should change to false till EL 6.5 comes out, I guess…
Global configurable are evil, but yes. Please set it to False by
default, until we can require a libvirt version that has it.
Dan.
3:22 p.m.
I just took a look through the vdc_options keys and I only found key
entries of "AbortMigrationOnError" for cluster levels 3.0, 3.1, and 3.2 but
not 3.3.
- DHC
On Tue, Oct 15, 2013 at 10:55 AM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Tue, Oct 15, 2013 at 04:42:11PM +0200, Michal Skrivanek wrote:
>
> On Oct 15, 2013, at 16:36 , Dead Horse <deadhorseconsulting(a)gmail.com>
wrote:
>
> > I have been running EL 6.4 hosts in 3.3 mode for quite some time, I
only noticed this breakage in the latest master VDSM 4.13.x. Tagged vdsm
versions: ovirt-3.3.0 and ovirt-3.3 do work (if not using master which one
of these should be used with 3.3 btw?)
>
> it is correct
> I suppose as a workaround you can use engine-config and set
AbortMigrationOnError to false for now…
> the default should change to false till EL 6.5 comes out, I guess…
Global configurable are evil, but yes. Please set it to False by
default, until we can require a libvirt version that has it.
Dan.
12:12 a.m.
--Apple-Mail-B5D6075B-ED03-4C00-A2E0-515CD9178B4C
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: base64
DQoNCk9uIDE1IE9jdCAyMDEzLCBhdCAyMjoyMiwgRGVhZCBIb3JzZSA8ZGVhZGhvcnNlY29uc3Vs
dGluZ0BnbWFpbC5jb20+IHdyb3RlOg0KDQo+IEkganVzdCB0b29rIGEgbG9vayB0aHJvdWdoIHRo
ZSB2ZGNfb3B0aW9ucyBrZXlzIGFuZCBJIG9ubHkgZm91bmQga2V5IGVudHJpZXMgb2YgIkFib3J0
TWlncmF0aW9uT25FcnJvciIgZm9yIGNsdXN0ZXIgbGV2ZWxzIDMuMCwgMy4xLCBhbmQgMy4yIGJ1
dCBub3QgMy4zLiANCg0KWWVzLCBzZXQgYSBuZXcgb25lIGZvciAzLjMgdG8gZmFsc2UNCg0KPiAN
Cj4gLSBESEMNCj4gDQo+IA0KPiBPbiBUdWUsIE9jdCAxNSwgMjAxMyBhdCAxMDo1NSBBTSwgRGFu
IEtlbmlnc2JlcmcgPGRhbmtlbkByZWRoYXQuY29tPiB3cm90ZToNCj4+IE9uIFR1ZSwgT2N0IDE1
LCAyMDEzIGF0IDA0OjQyOjExUE0gKzAyMDAsIE1pY2hhbCBTa3JpdmFuZWsgd3JvdGU6DQo+PiA+
DQo+PiA+IE9uIE9jdCAxNSwgMjAxMywgYXQgMTY6MzYgLCBEZWFkIEhvcnNlIDxkZWFkaG9yc2Vj
b25zdWx0aW5nQGdtYWlsLmNvbT4gd3JvdGU6DQo+PiA+DQo+PiA+ID4gSSBoYXZlIGJlZW4gcnVu
bmluZyBFTCA2LjQgaG9zdHMgaW4gMy4zIG1vZGUgZm9yIHF1aXRlIHNvbWUgdGltZSwgSSBvbmx5
IG5vdGljZWQgdGhpcyBicmVha2FnZSBpbiB0aGUgbGF0ZXN0IG1hc3RlciBWRFNNIDQuMTMueC4g
VGFnZ2VkIHZkc20gdmVyc2lvbnM6IG92aXJ0LTMuMy4wIGFuZCBvdmlydC0zLjMgZG8gd29yayAo
aWYgbm90IHVzaW5nIG1hc3RlciB3aGljaCBvbmUgb2YgdGhlc2Ugc2hvdWxkIGJlIHVzZWQgd2l0
aCAzLjMgYnR3PykNCj4+ID4NCj4+ID4gaXQgaXMgY29ycmVjdA0KPj4gPiBJIHN1cHBvc2UgYXMg
YSB3b3JrYXJvdW5kIHlvdSBjYW4gdXNlIGVuZ2luZS1jb25maWcgYW5kIHNldCBBYm9ydE1pZ3Jh
dGlvbk9uRXJyb3IgdG8gZmFsc2UgZm9yIG5vd+KApg0KPj4gPiB0aGUgZGVmYXVsdCBzaG91bGQg
Y2hhbmdlIHRvIGZhbHNlIHRpbGwgRUwgNi41IGNvbWVzIG91dCwgSSBndWVzc+KApg0KPj4gDQo+
PiBHbG9iYWwgY29uZmlndXJhYmxlIGFyZSBldmlsLCBidXQgeWVzLiBQbGVhc2Ugc2V0IGl0IHRv
IEZhbHNlIGJ5DQo+PiBkZWZhdWx0LCB1bnRpbCB3ZSBjYW4gcmVxdWlyZSBhIGxpYnZpcnQgdmVy
c2lvbiB0aGF0IGhhcyBpdC4NCj4+IA0KPj4gRGFuLg0KPiANCg==
--Apple-Mail-B5D6075B-ED03-4C00-A2E0-515CD9178B4C
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0
L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPjwvaGVhZD48Ym9keSBkaXI9ImF1dG8iPjxkaXY+PGJyPjxi
cj5PbiAxNSBPY3QgMjAxMywgYXQgMjI6MjIsIERlYWQgSG9yc2UgJmx0OzxhIGhyZWY9Im1haWx0
bzpkZWFkaG9yc2Vjb25zdWx0aW5nQGdtYWlsLmNvbSI+ZGVhZGhvcnNlY29uc3VsdGluZ0BnbWFp
bC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+PGJyPjwvZGl2PjxibG9ja3F1b3RlIHR5cGU9ImNpdGUi
PjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj5JIGp1c3QgdG9vayBhIGxvb2sgdGhyb3VnaCB0aGUg
dmRjX29wdGlvbnMga2V5cyBhbmQgSSBvbmx5IGZvdW5kIGtleSBlbnRyaWVzIG9mICJBYm9ydE1p
Z3JhdGlvbk9uRXJyb3IiIGZvciBjbHVzdGVyIGxldmVscyAzLjAsIDMuMSwgYW5kIDMuMiBidXQg
bm90IDMuMy4gPGJyPjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rp
dj5ZZXMsIHNldCBhIG5ldyBvbmUgZm9yIDMuMyB0byBmYWxzZTxkaXY+PGJyPjxibG9ja3F1b3Rl
IHR5cGU9ImNpdGUiPjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48YnI+PC9kaXY+LSBESEM8YnI+
PC9kaXY+DQo8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPjxicj48ZGl2IGNsYXNzPSJnbWFp
bF9xdW90ZSI+T24gVHVlLCBPY3QgMTUsIDIwMTMgYXQgMTA6NTUgQU0sIERhbiBLZW5pZ3NiZXJn
IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmRhbmtlbkByZWRoYXQuY29tIiB0
YXJnZXQ9Il9ibGFuayI+ZGFua2VuQHJlZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJy
Pg0KPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44
ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBjbGFz
cz0iaW0iPk9uIFR1ZSwgT2N0IDE1LCAyMDEzIGF0IDA0OjQyOjExUE0gKzAyMDAsIE1pY2hhbCBT
a3JpdmFuZWsgd3JvdGU6PGJyPg0KJmd0Ozxicj4NCiZndDsgT24gT2N0IDE1LCAyMDEzLCBhdCAx
NjozNiAsIERlYWQgSG9yc2UgJmx0OzxhIGhyZWY9Im1haWx0bzpkZWFkaG9yc2Vjb25zdWx0aW5n
QGdtYWlsLmNvbSI+ZGVhZGhvcnNlY29uc3VsdGluZ0BnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8
YnI+DQomZ3Q7PGJyPg0KJmd0OyAmZ3Q7IEkgaGF2ZSBiZWVuIHJ1bm5pbmcgRUwgNi40IGhvc3Rz
IGluIDMuMyBtb2RlIGZvciBxdWl0ZSBzb21lIHRpbWUsIEkgb25seSBub3RpY2VkIHRoaXMgYnJl
YWthZ2UgaW4gdGhlIGxhdGVzdCBtYXN0ZXIgVkRTTSA0LjEzLnguIFRhZ2dlZCB2ZHNtIHZlcnNp
b25zOiBvdmlydC0zLjMuMCBhbmQgb3ZpcnQtMy4zIGRvIHdvcmsgKGlmIG5vdCB1c2luZyBtYXN0
ZXIgd2hpY2ggb25lIG9mIHRoZXNlIHNob3VsZCBiZSB1c2VkIHdpdGggMy4zIGJ0dz8pPGJyPg0K
DQomZ3Q7PGJyPg0KJmd0OyBpdCBpcyBjb3JyZWN0PGJyPg0KJmd0OyBJIHN1cHBvc2UgYXMgYSB3
b3JrYXJvdW5kIHlvdSBjYW4gdXNlIGVuZ2luZS1jb25maWcgYW5kIHNldCBBYm9ydE1pZ3JhdGlv
bk9uRXJyb3IgdG8gZmFsc2UgZm9yIG5vd+KApjxicj4NCiZndDsgdGhlIGRlZmF1bHQgc2hvdWxk
IGNoYW5nZSB0byBmYWxzZSB0aWxsIEVMIDYuNSBjb21lcyBvdXQsIEkgZ3Vlc3PigKY8YnI+DQo8
YnI+DQo8L2Rpdj5HbG9iYWwgY29uZmlndXJhYmxlIGFyZSBldmlsLCBidXQgeWVzLiBQbGVhc2Ug
c2V0IGl0IHRvIEZhbHNlIGJ5PGJyPg0KZGVmYXVsdCwgdW50aWwgd2UgY2FuIHJlcXVpcmUgYSBs
aWJ2aXJ0IHZlcnNpb24gdGhhdCBoYXMgaXQuPGJyPg0KPHNwYW4gY2xhc3M9IkhPRW5aYiI+PGZv
bnQgY29sb3I9IiM4ODg4ODgiPjxicj4NCkRhbi48YnI+DQo8L2ZvbnQ+PC9zcGFuPjwvYmxvY2tx
dW90ZT48L2Rpdj48YnI+PC9kaXY+DQo8L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+PC9ib2R5Pjwv
aHRtbD4=
--Apple-Mail-B5D6075B-ED03-4C00-A2E0-515CD9178B4C--
3868
days inactive
3873
days old
7 comments
3 participants
participants (3)
-
Dan Kenigsberg -
Dead Horse -
Michal Skrivanek