9:40 p.m.
------=_NextPartTM-000-842b7e16-7dd5-497a-b341-090e5ac35743
Content-Type: multipart/alternative;
boundary="_000_354c6d82d06343f59dfebdb779762820emailandroidcom_"
--_000_354c6d82d06343f59dfebdb779762820emailandroidcom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
RG8geW91IGhhdmUgYSBjaGFuY2UgdG8gaW5zdGFsbCBxZW11LWRlYnVnPyBJZiB5ZXMgSSB3b3Vs
ZCB0cnkgYSBiYWNrdHJhY2UuDQoNCmdkYiAtcCA8cWVtdS1waWQ+DQojIGJ0DQoNCk1hcmt1cw0K
DQpBbSAxNS4wOS4yMDE1IDQ6MTUgbmFjaG0uIHNjaHJpZWIgRGFuaWVsIEhlbGdlbmJlcmdlciA8
ZGFuaWVsLmhlbGdlbmJlcmdlckBtLWJveC5kZT46DQpIZWxsbywNCg0KSSBkbyBub3Qgd2FudCB0
byBoaWphY2sgdGhlIHRocmVhZCBidXQgbWF5YmUgbXkgaXNzdWUgaXMgcmVsYXRlZD8NCg0KSXQg
bWlnaHQgaGF2ZSBzdGFydGVkIHdpdGggb3ZpcnQgMy41LjM7IGJ1dCBJIGNhbm5vdCB0ZWxsIGZv
ciBzdXJlLg0KDQpGb3IgbWUsIG9uZSB2bSAoZm9yZW1hbikgaXMgYWZmZWN0ZWQ7IHRoZSBzZWNv
bmQgdGltZSBpbiAxNCBkYXlzLiBJIGNhbiBjb25maXJtIHRoaXMgYXMgSSBhbHNvIGxvb3NlIGFu
eSBuZXR3b3JrIGNvbm5lY3Rpb24gdG8gdGhlIFZNIGFuZA0KdGhlIGFiaWxpdHkgdG8gY29ubmVj
dCBhIGNvbnNvbGUuDQpBbHNvLCB0aGUgb25seSB0aGluZyB3aXRjaCAnZml4ZXMnIHRoZSBpc3N1
ZSBpcyByaWdodCBub3cgJ2tpbGwgLTkgPHBpZCBvZiBxZW11LWt2bSBwcm9jZXNzPicNCg0KQXMg
ZmFyIGFzIEkgY2FuIHRlbGwgdGhlIFZNIGJlY2FtZSB1bnJlc3BvbnNpdmUgYXQgYXJvdW5kIFNl
cCAxNSAxMjozMDowMTsgZW5naW5lIGxvZ2dlZCB0aGlzIGF0IDEyOjM0LiBOb3RoaW5nIG9idmlv
dXMgaW4gVkRTTSBsb2dzIChzZWUNCmF0dGFjaGVkKS4NCg0KQmVsb3cgdGhlIGVuZ2luZS5sb2cg
cGFydC4NCg0KVmVyc2lvbnM6DQpvdmlydC1lbmdpbmUtMy41LjQuMi0xLmVsNy5jZW50b3Mubm9h
cmNoDQoNCnZkc20tNC4xNi4yNi0wLmVsNy5jZW50b3MNCmxpYnZpcnQtMS4yLjgtMTYuZWw3XzEu
Mw0KDQplbmdpbmUubG9nICgxMjAwIC0gMTMwMDoNCjIwMTUtMDktMTUgMTI6MDM6NDcsOTQ5IElO
Rk8gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLnNjaGVkdWxpbmcuSGFSZXNlcnZhdGlvbkhh
bmRsaW5nXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNTYpIFsyNjRkNTAyYV0gSEEN
CnJlc2VydmF0aW9uIHN0YXR1cyBmb3IgY2x1c3RlciBEZWZhdWx0IGlzIE9LDQoyMDE1LTA5LTE1
IDEyOjA4OjAyLDcwOCBJTkZPICBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5PdmZEYXRhVXBk
YXRlcl0gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTg5KSBbMmU3YmY1NmVdIEF0dGVt
cHRpbmcgdG8gdXBkYXRlDQpWTXMvVGVtcGxhdGVzIE92Zi4NCjIwMTUtMDktMTUgMTI6MDg6MDIs
NzA5IElORk8gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLlByb2Nlc3NPdmZVcGRhdGVGb3JT
dG9yYWdlUG9vbENvbW1hbmRdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci04OSkNCls1
ZTlmNGJhNl0gUnVubmluZyBjb21tYW5kOiBQcm9jZXNzT3ZmVXBkYXRlRm9yU3RvcmFnZVBvb2xD
b21tYW5kIGludGVybmFsOiB0cnVlLiBFbnRpdGllcyBhZmZlY3RlZCA6ICBJRDoNCjAwMDAwMDAy
LTAwMDItMDAwMi0wMDAyLTAwMDAwMDAwMDA4OCBUeXBlOiBsDQoyMDE1LTA5LTE1IDEyOjA4OjAy
LDc4MCBJTkZPICBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5Qcm9jZXNzT3ZmVXBkYXRlRm9y
U3RvcmFnZVBvb2xDb21tYW5kXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItODkpDQpb
NWU5ZjRiYTZdIExvY2sgZnJlZWQgdG8gb2JqZWN0IEVuZ2luZUxvY2sgW2V4Y2x1c2l2ZUxvY2tz
PSBrZXk6IDAwMDAwMDAyLTAwMDItMDAwMi0wMDAyLTAwMDAwMDAwMDA4OCB2YWx1ZTogT1ZGX1VQ
REFURQ0KMjAxNS0wOS0xNSAxMjowODo0Nyw5OTcgSU5GTyAgW29yZy5vdmlydC5lbmdpbmUuY29y
ZS5ibGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2No
ZWR1bGVyX1dvcmtlci0yMSkgWzNmYzg1NGEyXSBIQQ0KcmVzZXJ2YXRpb24gc3RhdHVzIGZvciBj
bHVzdGVyIERlZmF1bHQgaXMgT0sNCjIwMTUtMDktMTUgMTI6MTM6MDYsOTk4IElORk8gIFtvcmcu
b3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLnZkc2Jyb2tlci5HZXRGaWxlU3RhdHNWRFNDb21t
YW5kXSAob3JnLm92aXJ0LnRocmVhZC5wb29sLTgtdGhyZWFkLTQ4KQ0KWzUwMjIxY2RjXSBTVEFS
VCwgR2V0RmlsZVN0YXRzVkRTQ29tbWFuZCggc3RvcmFnZVBvb2xJZCA9IDAwMDAwMDAyLTAwMDIt
MDAwMi0wMDAyLTAwMDAwMDAwMDA4OCwgaWdub3JlRmFpbG92ZXJMaW1pdCA9IGZhbHNlKSwgbG9n
IGlkOiAxNTAzOTY4DQoyMDE1LTA5LTE1IDEyOjEzOjA3LDEzNyBJTkZPICBbb3JnLm92aXJ0LmVu
Z2luZS5jb3JlLnZkc2Jyb2tlci52ZHNicm9rZXIuR2V0RmlsZVN0YXRzVkRTQ29tbWFuZF0gKG9y
Zy5vdmlydC50aHJlYWQucG9vbC04LXRocmVhZC00OCkNCls1MDIyMWNkY10gRklOSVNILCBHZXRG
aWxlU3RhdHNWRFNDb21tYW5kLCByZXR1cm46IHtwZlNlbnNlLTIuMC1SRUxFQVNFLWkzODYuaXNv
PXtzdGF0dXM9MCwgY3RpbWU9MTQzMjI4Njg4Ny4wLCBzaXplPTExNTcwOTk1Mn0sDQpGZWRvcmEt
MTUtaTY4Ni1MaXZlOA0KMjAxNS0wOS0xNSAxMjoxMzowNywxNzggSU5GTyAgW29yZy5vdmlydC5l
bmdpbmUuY29yZS5ibGwuSXNvRG9tYWluTGlzdFN5bmNyb25pemVyXSAob3JnLm92aXJ0LnRocmVh
ZC5wb29sLTgtdGhyZWFkLTQ4KSBbNTAyMjFjZGNdIEZpbmlzaGVkDQphdXRvbWF0aWMgcmVmcmVz
aCBwcm9jZXNzIGZvciBJU08gZmlsZSB0eXBlIHdpdGggc3VjY2VzcywgZm9yIHN0b3JhZ2UgZG9t
YWluIGlkIDg0ZGNiMmZjLWZiNjMtNDQyZi1hYTc3LTNlODRkYzdkNWE3Mi4NCjIwMTUtMDktMTUg
MTI6MTM6NDgsMDQzIElORk8gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLnNjaGVkdWxpbmcu
SGFSZXNlcnZhdGlvbkhhbmRsaW5nXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItODcp
IFs0ZmExYmIxNl0gSEENCnJlc2VydmF0aW9uIHN0YXR1cyBmb3IgY2x1c3RlciBEZWZhdWx0IGlz
IE9LDQoyMDE1LTA5LTE1IDEyOjE4OjQ4LDA4OCBJTkZPICBbb3JnLm92aXJ0LmVuZ2luZS5jb3Jl
LmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hl
ZHVsZXJfV29ya2VyLTQ0KSBbNjM0NWU2OThdIEhBDQpyZXNlcnZhdGlvbiBzdGF0dXMgZm9yIGNs
dXN0ZXIgRGVmYXVsdCBpcyBPSw0KMjAxNS0wOS0xNSAxMjoyMzo0OCwxMzcgSU5GTyAgW29yZy5v
dmlydC5lbmdpbmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChE
ZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0xMykgSEEgcmVzZXJ2YXRpb24NCnN0YXR1cyBm
b3IgY2x1c3RlciBEZWZhdWx0IGlzIE9LDQoyMDE1LTA5LTE1IDEyOjI4OjQ4LDE4MyBJTkZPICBb
b3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGlu
Z10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTc2KSBbMTU0YzkxZDVdIEhBDQpyZXNl
cnZhdGlvbiBzdGF0dXMgZm9yIGNsdXN0ZXIgRGVmYXVsdCBpcyBPSw0KMjAxNS0wOS0xNSAxMjoz
Mzo0OCwyMjkgSU5GTyAgW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJl
c2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0zNikgWzI3
YzczYWM2XSBIQQ0KcmVzZXJ2YXRpb24gc3RhdHVzIGZvciBjbHVzdGVyIERlZmF1bHQgaXMgT0sN
CjIwMTUtMDktMTUgMTI6MzQ6NDksNDMyIElORk8gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRz
YnJva2VyLlZkc1VwZGF0ZVJ1blRpbWVJbmZvXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3Jr
ZXItNDEpIFs1ZjJhNGI2OF0gVk0NCmZvcmVtYW4gOGI1N2ZmMWQtMjgwMC00OGFkLWIyNjctZmQ4
ZTllMmY2ZmIyIG1vdmVkIGZyb20gVXAgLS0+IE5vdFJlc3BvbmRpbmcNCjIwMTUtMDktMTUgMTI6
MzQ6NDksNTc4IFdBUk4gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJva2VyLmF1ZGl0
bG9naGFuZGxpbmcuQXVkaXRMb2dEaXJlY3Rvcl0gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29y
a2VyLTQxKQ0KWzVmMmE0YjY4XSBDb3JyZWxhdGlvbiBJRDogbnVsbCwgQ2FsbCBTdGFjazogbnVs
bCwgQ3VzdG9tIEV2ZW50IElEOiAtMSwgTWVzc2FnZTogVk0gZm9yZW1hbiBpcyBub3QgcmVzcG9u
ZGluZy4NCjIwMTUtMDktMTUgMTI6Mzg6NDgsMjczIElORk8gIFtvcmcub3ZpcnQuZW5naW5lLmNv
cmUuYmxsLnNjaGVkdWxpbmcuSGFSZXNlcnZhdGlvbkhhbmRsaW5nXSAoRGVmYXVsdFF1YXJ0elNj
aGVkdWxlcl9Xb3JrZXItMTApIFs3YTgwMDc2Nl0gSEENCnJlc2VydmF0aW9uIHN0YXR1cyBmb3Ig
Y2x1c3RlciBEZWZhdWx0IGlzIE9LDQoyMDE1LTA5LTE1IDEyOjQzOjQ4LDMyMCBJTkZPICBbb3Jn
Lm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGluZ10g
KERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTQyKSBbNDQwZjFjNDBdIEhBDQpyZXNlcnZh
dGlvbiBzdGF0dXMgZm9yIGNsdXN0ZXIgRGVmYXVsdCBpcyBPSw0KMjAxNS0wOS0xNSAxMjo0ODo0
OCwzNjYgSU5GTyAgW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJlc2Vy
dmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci03MCkgSEEgcmVz
ZXJ2YXRpb24NCnN0YXR1cyBmb3IgY2x1c3RlciBEZWZhdWx0IGlzIE9LDQoyMDE1LTA5LTE1IDEy
OjUzOjQ4LDQxMiBJTkZPICBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhh
UmVzZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTEyKSBb
NTAyMjFjZGNdIEhBDQpyZXNlcnZhdGlvbiBzdGF0dXMgZm9yIGNsdXN0ZXIgRGVmYXVsdCBpcyBP
Sw0KMjAxNS0wOS0xNSAxMjo1ODo0OCw0NTkgSU5GTyAgW29yZy5vdmlydC5lbmdpbmUuY29yZS5i
bGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2NoZWR1
bGVyX1dvcmtlci0zKSBIQSByZXNlcnZhdGlvbg0Kc3RhdHVzIGZvciBjbHVzdGVyIERlZmF1bHQg
aXMgT0sNCg0KDQoNCk9uIDI5LjA4LjIwMTUgMjI6NDgsIENocmlzdGlhbiBIYWlsZXIgd3JvdGU6
DQo+IEhlbGxvLA0KPg0KPiBsYXN0IFdlZG5lc2RheSBJIHdhbnRlZCB0byB1cGRhdGUgbXkgb1Zp
cnQgMy41IGh5cGVydmlzb3IuIEl0IGlzIGEgc2luZ2xlIENlbnRvcw0KPiA3IHNlcnZlciwgc28g
SSBzdGFydGVkIGJ5IHN1c3BlbmRpbmcgdGhlIFZNcyBpbiBvcmRlciB0byBzZXQgdGhlIG9WaXJ0
IGVuZ2luZQ0KPiBob3N0IHRvIG1haW50ZW5hbmNlIG1vZGUuIER1cmluZyB0aGUgcHJvY2VzcyBv
ZiBzdXNwZW5kaW5nIHRoZSBWTXMgdGhlIHNlcnZlcg0KPiBjcmFzaGVkLCBrZXJuZWwgcGFuaWPi
gKYNCj4NCj4gQWZ0ZXIgcmVzdGFydGluZyB0aGUgc2VydmVyIEkgaW5zdGFsbGVkIHRoZSB1cGRh
dGVzIHZpYSB5dW0gYW4gcmVzdGFydGVkIHRoZQ0KPiBzZXJ2ZXIgYWdhaW4uIEFmdGVyd2FyZHMs
IGFsbCB0aGUgVk1zIGNvdWxkIGJlIHN0YXJ0ZWQgYWdhaW4uIFNvbWUgaG91cnMgbGF0ZXINCj4g
bXkgbW9uaXRvcmluZyBzeXN0ZW0gcmVnaXN0ZXJlZCBzb21lIHVucmVzcG9uc2l2ZSBob3N0cywg
SSBoYWQgYSBsb29rIGluIHRoZQ0KPiBvVmlydCBpbnRlcmZhY2UsIDMgb2YgdGhlIFZNcyB3ZXJl
IGluIHRoZSBzdGF0ZSDigJxub3QgcmVzcG9uZGluZ+KAnSwgbWFya2VkIGJ5IGENCj4gcXVlc3Rp
b24gbWFyay4NCj4NCj4gSSB0cmllZCB0byBzaHV0IGRvd24gdGhlIFZNcywgYnV0IG9WaXJ0IHdh
c27igJl0IGFibGUgdG8gZG8gc28uIEkgdHJpZWQgdG8gcmVzZXQNCj4gdGhlIHN0YXR1cyBpbiB0
aGUgZGF0YWJhc2Ugd2l0aCB0aGUgc3FsIHN0YXRlbWVudA0KPg0KPiB1cGRhdGUgdm1fZHluYW1p
YyBzZXQgc3RhdHVzID0gMCB3aGVyZSB2bV9ndWlkID0gKHNlbGVjdCB2bV9ndWlkIGZyb20gdm1f
c3RhdGljDQo+IHdoZXJlIHZtX25hbWUgPSAnTVlWTU5BTUUnKTsNCj4NCj4gYnV0IHRoYXQgZGlk
buKAmXQgaGVscCwgZWl0aGVyLiBPbmx5IHJlYm9vdGluZyB0aGUgd2hvbGUgaHlwZXJ2aXNvciBo
ZWxwZWTigKYNCj4gYWZ0ZXJ3YXJkcyBldmVyeXRoaW5nIHdvcmtlZCBhZ2Fpbi4gQnV0IG9ubHkg
Zm9yIGEgZmV3IGhvdXJzLCB0aGVuIG9uZSBvZiB0aGUNCj4gVk1zIGVudGVyZWQgdGhlIOKAnG5v
dCByZXNwb25kaW5n4oCdIHN0YXRlIGFnYWlu4oCmIGFnYWluIG9ubHkgYSByZWJvb3QgaGVscGVk
Lg0KPiBZZXN0ZXJkYXkgaXQgaGFwcGVuZWQgYWdhaW46DQo+DQo+IDIwMTUtMDgtMjggMTc6NDQ6
MjIsNjY0IElORk8NCj4gW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuVmRzVXBkYXRl
UnVuVGltZUluZm9dDQo+IChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci02MCkgWzRlZjkw
YjEyXSBWTSBEQw0KPiAwZjNkMWYwNi1lNTE2LTQ4Y2UtYWE2Zi03MjczYzMzZDM0OTEgbW92ZWQg
ZnJvbSBVcCAtLT4gTm90UmVzcG9uZGluZw0KPg0KPiAyMDE1LTA4LTI4IDE3OjQ0OjIyLDY5MiBX
QVJODQo+IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJva2VyLmF1ZGl0bG9naGFuZGxp
bmcuQXVkaXRMb2dEaXJlY3Rvcl0NCj4gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTYw
KSBbNGVmOTBiMTJdIENvcnJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOg0KPiBudWxsLCBD
dXN0b20gRXZlbnQgSUQ6IC0xLCBNZXNzYWdlOiBWTSBEQyBpcyBub3QgcmVzcG9uZGluZy4NCj4N
Cj4gRG9lcyBhbnlib2R5IGtub3cgd2hhdCBJIGNhbiBkbz8gV2hlcmUgc2hvdWxkIEkgaGF2ZSBh
IGxvb2s/IEhpbnRzIGFyZSBncmVhdGx5DQo+IGFwcHJlY2lhdGVkIQ0KPg0KPiBUaGFua3MsDQo+
DQo+IENocmlzdGlhbg0KPg0KDQotLQ0KRGFuaWVsIEhlbGdlbmJlcmdlcg0KbSBib3ggYmV3ZWd0
YmlsZCBHbWJIDQoNClA6ICs0OS8zMC8yNDA4NzgxLTIyDQpGOiArNDkvMzAvMjQwODc4MS0xMA0K
DQpBQ0tFUlNUUi4gMTkNCkQtMTAxMTUgQkVSTElODQoNCg0Kd3d3Lm0tYm94LmRlPGh0dHA6Ly93
d3cubS1ib3guZGU+ICB3d3cubW9ua2V5bWVuLnR2PGh0dHA6Ly93d3cubW9ua2V5bWVuLnR2Pg0K
DQpHZXNjaMOkZnRzZsO8aHJlcjogTWFydGluIFJldHNjaGl0emVnZ2VyIC8gTWljaGFlbGEgR8O2
bGxuZXINCkhhbmRlc2xyZWdpc3RlcjogQW10c2dlcmljaHQgQ2hhcmxvdHRlbmJ1cmcgLyBIUkIg
MTEyNzY3DQo=
--_000_354c6d82d06343f59dfebdb779762820emailandroidcom_
Content-Type: text/html; charset="utf-8"
Content-ID: <BB39A1C46290544BB19A578F6A0B74C4(a)collogia.de>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5Pg0KPHAgZGlyPSJsdHIi
PkRvIHlvdSBoYXZlIGEgY2hhbmNlIHRvIGluc3RhbGwgcWVtdS1kZWJ1Zz8gSWYgeWVzIEkgd291
bGQgdHJ5IGEgYmFja3RyYWNlLjwvcD4NCjxwIGRpcj0ibHRyIj5nZGIgLXAgJmx0O3FlbXUtcGlk
Jmd0Ozxicj4NCiMgYnQ8L3A+DQo8cCBkaXI9Imx0ciI+TWFya3VzPC9wPg0KPGRpdiBjbGFzcz0i
Z21haWxfcXVvdGUiPkFtIDE1LjA5LjIwMTUgNDoxNSBuYWNobS4gc2NocmllYiBEYW5pZWwgSGVs
Z2VuYmVyZ2VyICZsdDtkYW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94LmRlJmd0Ozo8YnIgdHlwZT0i
YXR0cmlidXRpb24iPg0KPGJsb2NrcXVvdGUgY2xhc3M9InF1b3RlIiBzdHlsZT0ibWFyZ2luOjAg
MCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+DQo8
ZGl2Pjxmb250IHNpemU9IjIiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTBwdCI+PC9zcGFuPjwv
Zm9udD4NCjxkaXY+SGVsbG8sPGJyPg0KPGJyPg0KSSBkbyBub3Qgd2FudCB0byBoaWphY2sgdGhl
IHRocmVhZCBidXQgbWF5YmUgbXkgaXNzdWUgaXMgcmVsYXRlZD88YnI+DQo8YnI+DQpJdCBtaWdo
dCBoYXZlIHN0YXJ0ZWQgd2l0aCBvdmlydCAzLjUuMzsgYnV0IEkgY2Fubm90IHRlbGwgZm9yIHN1
cmUuPGJyPg0KPGJyPg0KRm9yIG1lLCBvbmUgdm0gKGZvcmVtYW4pIGlzIGFmZmVjdGVkOyB0aGUg
c2Vjb25kIHRpbWUgaW4gMTQgZGF5cy4gSSBjYW4gY29uZmlybSB0aGlzIGFzIEkgYWxzbyBsb29z
ZSBhbnkgbmV0d29yayBjb25uZWN0aW9uIHRvIHRoZSBWTSBhbmQ8YnI+DQp0aGUgYWJpbGl0eSB0
byBjb25uZWN0IGEgY29uc29sZS48YnI+DQpBbHNvLCB0aGUgb25seSB0aGluZyB3aXRjaCAnZml4
ZXMnIHRoZSBpc3N1ZSBpcyByaWdodCBub3cgJ2tpbGwgLTkgJmx0O3BpZCBvZiBxZW11LWt2bSBw
cm9jZXNzJmd0Oyc8YnI+DQo8YnI+DQpBcyBmYXIgYXMgSSBjYW4gdGVsbCB0aGUgVk0gYmVjYW1l
IHVucmVzcG9uc2l2ZSBhdCBhcm91bmQgU2VwIDE1IDEyOjMwOjAxOyBlbmdpbmUgbG9nZ2VkIHRo
aXMgYXQgMTI6MzQuIE5vdGhpbmcgb2J2aW91cyBpbiBWRFNNIGxvZ3MgKHNlZTxicj4NCmF0dGFj
aGVkKS48YnI+DQo8YnI+DQpCZWxvdyB0aGUgZW5naW5lLmxvZyBwYXJ0Ljxicj4NCjxicj4NClZl
cnNpb25zOjxicj4NCm92aXJ0LWVuZ2luZS0zLjUuNC4yLTEuZWw3LmNlbnRvcy5ub2FyY2g8YnI+
DQo8YnI+DQp2ZHNtLTQuMTYuMjYtMC5lbDcuY2VudG9zPGJyPg0KbGlidmlydC0xLjIuOC0xNi5l
bDdfMS4zPGJyPg0KPGJyPg0KZW5naW5lLmxvZyAoMTIwMCAtIDEzMDA6PGJyPg0KMjAxNS0wOS0x
NSAxMjowMzo0Nyw5NDkgSU5GTyZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hl
ZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29y
a2VyLTU2KSBbMjY0ZDUwMmFdIEhBPGJyPg0KcmVzZXJ2YXRpb24gc3RhdHVzIGZvciBjbHVzdGVy
IERlZmF1bHQgaXMgT0s8YnI+DQoyMDE1LTA5LTE1IDEyOjA4OjAyLDcwOCBJTkZPJm5ic3A7IFtv
cmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLk92ZkRhdGFVcGRhdGVyXSAoRGVmYXVsdFF1YXJ0elNj
aGVkdWxlcl9Xb3JrZXItODkpIFsyZTdiZjU2ZV0gQXR0ZW1wdGluZyB0byB1cGRhdGU8YnI+DQpW
TXMvVGVtcGxhdGVzIE92Zi48YnI+DQoyMDE1LTA5LTE1IDEyOjA4OjAyLDcwOSBJTkZPJm5ic3A7
IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLlByb2Nlc3NPdmZVcGRhdGVGb3JTdG9yYWdlUG9v
bENvbW1hbmRdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci04OSk8YnI+DQpbNWU5ZjRi
YTZdIFJ1bm5pbmcgY29tbWFuZDogUHJvY2Vzc092ZlVwZGF0ZUZvclN0b3JhZ2VQb29sQ29tbWFu
ZCBpbnRlcm5hbDogdHJ1ZS4gRW50aXRpZXMgYWZmZWN0ZWQgOiZuYnNwOyBJRDo8YnI+DQowMDAw
MDAwMi0wMDAyLTAwMDItMDAwMi0wMDAwMDAwMDAwODggVHlwZTogbDxicj4NCjIwMTUtMDktMTUg
MTI6MDg6MDIsNzgwIElORk8mbmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuUHJvY2Vz
c092ZlVwZGF0ZUZvclN0b3JhZ2VQb29sQ29tbWFuZF0gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJf
V29ya2VyLTg5KTxicj4NCls1ZTlmNGJhNl0gTG9jayBmcmVlZCB0byBvYmplY3QgRW5naW5lTG9j
ayBbZXhjbHVzaXZlTG9ja3M9IGtleTogMDAwMDAwMDItMDAwMi0wMDAyLTAwMDItMDAwMDAwMDAw
MDg4IHZhbHVlOiBPVkZfVVBEQVRFPGJyPg0KMjAxNS0wOS0xNSAxMjowODo0Nyw5OTcgSU5GTyZu
YnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25I
YW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTIxKSBbM2ZjODU0YTJdIEhB
PGJyPg0KcmVzZXJ2YXRpb24gc3RhdHVzIGZvciBjbHVzdGVyIERlZmF1bHQgaXMgT0s8YnI+DQoy
MDE1LTA5LTE1IDEyOjEzOjA2LDk5OCBJTkZPJm5ic3A7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUu
dmRzYnJva2VyLnZkc2Jyb2tlci5HZXRGaWxlU3RhdHNWRFNDb21tYW5kXSAob3JnLm92aXJ0LnRo
cmVhZC5wb29sLTgtdGhyZWFkLTQ4KTxicj4NCls1MDIyMWNkY10gU1RBUlQsIEdldEZpbGVTdGF0
c1ZEU0NvbW1hbmQoIHN0b3JhZ2VQb29sSWQgPSAwMDAwMDAwMi0wMDAyLTAwMDItMDAwMi0wMDAw
MDAwMDAwODgsIGlnbm9yZUZhaWxvdmVyTGltaXQgPSBmYWxzZSksIGxvZyBpZDogMTUwMzk2ODxi
cj4NCjIwMTUtMDktMTUgMTI6MTM6MDcsMTM3IElORk8mbmJzcDsgW29yZy5vdmlydC5lbmdpbmUu
Y29yZS52ZHNicm9rZXIudmRzYnJva2VyLkdldEZpbGVTdGF0c1ZEU0NvbW1hbmRdIChvcmcub3Zp
cnQudGhyZWFkLnBvb2wtOC10aHJlYWQtNDgpPGJyPg0KWzUwMjIxY2RjXSBGSU5JU0gsIEdldEZp
bGVTdGF0c1ZEU0NvbW1hbmQsIHJldHVybjoge3BmU2Vuc2UtMi4wLVJFTEVBU0UtaTM4Ni5pc289
e3N0YXR1cz0wLCBjdGltZT0xNDMyMjg2ODg3LjAsIHNpemU9MTE1NzA5OTUyfSw8YnI+DQpGZWRv
cmEtMTUtaTY4Ni1MaXZlODxicj4NCjIwMTUtMDktMTUgMTI6MTM6MDcsMTc4IElORk8mbmJzcDsg
W29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuSXNvRG9tYWluTGlzdFN5bmNyb25pemVyXSAob3Jn
Lm92aXJ0LnRocmVhZC5wb29sLTgtdGhyZWFkLTQ4KSBbNTAyMjFjZGNdIEZpbmlzaGVkPGJyPg0K
YXV0b21hdGljIHJlZnJlc2ggcHJvY2VzcyBmb3IgSVNPIGZpbGUgdHlwZSB3aXRoIHN1Y2Nlc3Ms
IGZvciBzdG9yYWdlIGRvbWFpbiBpZCA4NGRjYjJmYy1mYjYzLTQ0MmYtYWE3Ny0zZTg0ZGM3ZDVh
NzIuPGJyPg0KMjAxNS0wOS0xNSAxMjoxMzo0OCwwNDMgSU5GTyZuYnNwOyBbb3JnLm92aXJ0LmVu
Z2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRR
dWFydHpTY2hlZHVsZXJfV29ya2VyLTg3KSBbNGZhMWJiMTZdIEhBPGJyPg0KcmVzZXJ2YXRpb24g
c3RhdHVzIGZvciBjbHVzdGVyIERlZmF1bHQgaXMgT0s8YnI+DQoyMDE1LTA5LTE1IDEyOjE4OjQ4
LDA4OCBJTkZPJm5ic3A7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLnNjaGVkdWxpbmcuSGFS
ZXNlcnZhdGlvbkhhbmRsaW5nXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNDQpIFs2
MzQ1ZTY5OF0gSEE8YnI+DQpyZXNlcnZhdGlvbiBzdGF0dXMgZm9yIGNsdXN0ZXIgRGVmYXVsdCBp
cyBPSzxicj4NCjIwMTUtMDktMTUgMTI6MjM6NDgsMTM3IElORk8mbmJzcDsgW29yZy5vdmlydC5l
bmdpbmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0
UXVhcnR6U2NoZWR1bGVyX1dvcmtlci0xMykgSEEgcmVzZXJ2YXRpb248YnI+DQpzdGF0dXMgZm9y
IGNsdXN0ZXIgRGVmYXVsdCBpcyBPSzxicj4NCjIwMTUtMDktMTUgMTI6Mjg6NDgsMTgzIElORk8m
bmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9u
SGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci03NikgWzE1NGM5MWQ1XSBI
QTxicj4NCnJlc2VydmF0aW9uIHN0YXR1cyBmb3IgY2x1c3RlciBEZWZhdWx0IGlzIE9LPGJyPg0K
MjAxNS0wOS0xNSAxMjozMzo0OCwyMjkgSU5GTyZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3Jl
LmJsbC5zY2hlZHVsaW5nLkhhUmVzZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hl
ZHVsZXJfV29ya2VyLTM2KSBbMjdjNzNhYzZdIEhBPGJyPg0KcmVzZXJ2YXRpb24gc3RhdHVzIGZv
ciBjbHVzdGVyIERlZmF1bHQgaXMgT0s8YnI+DQoyMDE1LTA5LTE1IDEyOjM0OjQ5LDQzMiBJTkZP
Jm5ic3A7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLlZkc1VwZGF0ZVJ1blRpbWVJ
bmZvXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNDEpIFs1ZjJhNGI2OF0gVk08YnI+
DQpmb3JlbWFuIDhiNTdmZjFkLTI4MDAtNDhhZC1iMjY3LWZkOGU5ZTJmNmZiMiBtb3ZlZCBmcm9t
IFVwIC0tJmd0OyBOb3RSZXNwb25kaW5nPGJyPg0KMjAxNS0wOS0xNSAxMjozNDo0OSw1NzggV0FS
TiZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmRhbC5kYmJyb2tlci5hdWRpdGxvZ2hhbmRs
aW5nLkF1ZGl0TG9nRGlyZWN0b3JdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci00MSk8
YnI+DQpbNWYyYTRiNjhdIENvcnJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOiBudWxsLCBD
dXN0b20gRXZlbnQgSUQ6IC0xLCBNZXNzYWdlOiBWTSBmb3JlbWFuIGlzIG5vdCByZXNwb25kaW5n
Ljxicj4NCjIwMTUtMDktMTUgMTI6Mzg6NDgsMjczIElORk8mbmJzcDsgW29yZy5vdmlydC5lbmdp
bmUuY29yZS5ibGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVh
cnR6U2NoZWR1bGVyX1dvcmtlci0xMCkgWzdhODAwNzY2XSBIQTxicj4NCnJlc2VydmF0aW9uIHN0
YXR1cyBmb3IgY2x1c3RlciBEZWZhdWx0IGlzIE9LPGJyPg0KMjAxNS0wOS0xNSAxMjo0Mzo0OCwz
MjAgSU5GTyZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5zY2hlZHVsaW5nLkhhUmVz
ZXJ2YXRpb25IYW5kbGluZ10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTQyKSBbNDQw
ZjFjNDBdIEhBPGJyPg0KcmVzZXJ2YXRpb24gc3RhdHVzIGZvciBjbHVzdGVyIERlZmF1bHQgaXMg
T0s8YnI+DQoyMDE1LTA5LTE1IDEyOjQ4OjQ4LDM2NiBJTkZPJm5ic3A7IFtvcmcub3ZpcnQuZW5n
aW5lLmNvcmUuYmxsLnNjaGVkdWxpbmcuSGFSZXNlcnZhdGlvbkhhbmRsaW5nXSAoRGVmYXVsdFF1
YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzApIEhBIHJlc2VydmF0aW9uPGJyPg0Kc3RhdHVzIGZvciBj
bHVzdGVyIERlZmF1bHQgaXMgT0s8YnI+DQoyMDE1LTA5LTE1IDEyOjUzOjQ4LDQxMiBJTkZPJm5i
c3A7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLnNjaGVkdWxpbmcuSGFSZXNlcnZhdGlvbkhh
bmRsaW5nXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItMTIpIFs1MDIyMWNkY10gSEE8
YnI+DQpyZXNlcnZhdGlvbiBzdGF0dXMgZm9yIGNsdXN0ZXIgRGVmYXVsdCBpcyBPSzxicj4NCjIw
MTUtMDktMTUgMTI6NTg6NDgsNDU5IElORk8mbmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29yZS5i
bGwuc2NoZWR1bGluZy5IYVJlc2VydmF0aW9uSGFuZGxpbmddIChEZWZhdWx0UXVhcnR6U2NoZWR1
bGVyX1dvcmtlci0zKSBIQSByZXNlcnZhdGlvbjxicj4NCnN0YXR1cyBmb3IgY2x1c3RlciBEZWZh
dWx0IGlzIE9LPGJyPg0KPGJyPg0KPGJyPg0KPGJyPg0KT24gMjkuMDguMjAxNSAyMjo0OCwgQ2hy
aXN0aWFuIEhhaWxlciB3cm90ZTo8YnI+DQomZ3Q7IEhlbGxvLDxicj4NCiZndDsgPGJyPg0KJmd0
OyBsYXN0IFdlZG5lc2RheSBJIHdhbnRlZCB0byB1cGRhdGUgbXkgb1ZpcnQgMy41IGh5cGVydmlz
b3IuIEl0IGlzIGEgc2luZ2xlIENlbnRvcw0KPGJyPg0KJmd0OyA3IHNlcnZlciwgc28gSSBzdGFy
dGVkIGJ5IHN1c3BlbmRpbmcgdGhlIFZNcyBpbiBvcmRlciB0byBzZXQgdGhlIG9WaXJ0IGVuZ2lu
ZSA8YnI+DQomZ3Q7IGhvc3QgdG8gbWFpbnRlbmFuY2UgbW9kZS4gRHVyaW5nIHRoZSBwcm9jZXNz
IG9mIHN1c3BlbmRpbmcgdGhlIFZNcyB0aGUgc2VydmVyIDxicj4NCiZndDsgY3Jhc2hlZCwga2Vy
bmVsIHBhbmlj4oCmPGJyPg0KJmd0OyA8YnI+DQomZ3Q7IEFmdGVyIHJlc3RhcnRpbmcgdGhlIHNl
cnZlciBJIGluc3RhbGxlZCB0aGUgdXBkYXRlcyB2aWEgeXVtIGFuIHJlc3RhcnRlZCB0aGUgPGJy
Pg0KJmd0OyBzZXJ2ZXIgYWdhaW4uIEFmdGVyd2FyZHMsIGFsbCB0aGUgVk1zIGNvdWxkIGJlIHN0
YXJ0ZWQgYWdhaW4uIFNvbWUgaG91cnMgbGF0ZXIgPGJyPg0KJmd0OyBteSBtb25pdG9yaW5nIHN5
c3RlbSByZWdpc3RlcmVkIHNvbWUgdW5yZXNwb25zaXZlIGhvc3RzLCBJIGhhZCBhIGxvb2sgaW4g
dGhlIDxicj4NCiZndDsgb1ZpcnQgaW50ZXJmYWNlLCAzIG9mIHRoZSBWTXMgd2VyZSBpbiB0aGUg
c3RhdGUg4oCcbm90IHJlc3BvbmRpbmfigJ0sIG1hcmtlZCBieSBhIDxicj4NCiZndDsgcXVlc3Rp
b24gbWFyay48YnI+DQomZ3Q7IDxicj4NCiZndDsgSSB0cmllZCB0byBzaHV0IGRvd24gdGhlIFZN
cywgYnV0IG9WaXJ0IHdhc27igJl0IGFibGUgdG8gZG8gc28uIEkgdHJpZWQgdG8gcmVzZXQgPGJy
Pg0KJmd0OyB0aGUgc3RhdHVzIGluIHRoZSBkYXRhYmFzZSB3aXRoIHRoZSBzcWwgc3RhdGVtZW50
PGJyPg0KJmd0OyA8YnI+DQomZ3Q7IHVwZGF0ZSB2bV9keW5hbWljIHNldCBzdGF0dXMgPSAwIHdo
ZXJlIHZtX2d1aWQgPSAoc2VsZWN0IHZtX2d1aWQgZnJvbSB2bV9zdGF0aWMNCjxicj4NCiZndDsg
d2hlcmUgdm1fbmFtZSA9ICdNWVZNTkFNRScpOzxicj4NCiZndDsgPGJyPg0KJmd0OyBidXQgdGhh
dCBkaWRu4oCZdCBoZWxwLCBlaXRoZXIuIE9ubHkgcmVib290aW5nIHRoZSB3aG9sZSBoeXBlcnZp
c29yIGhlbHBlZOKApiA8YnI+DQomZ3Q7IGFmdGVyd2FyZHMgZXZlcnl0aGluZyB3b3JrZWQgYWdh
aW4uIEJ1dCBvbmx5IGZvciBhIGZldyBob3VycywgdGhlbiBvbmUgb2YgdGhlIDxicj4NCiZndDsg
Vk1zIGVudGVyZWQgdGhlIOKAnG5vdCByZXNwb25kaW5n4oCdIHN0YXRlIGFnYWlu4oCmIGFnYWlu
IG9ubHkgYSByZWJvb3QgaGVscGVkLiA8YnI+DQomZ3Q7IFllc3RlcmRheSBpdCBoYXBwZW5lZCBh
Z2Fpbjo8YnI+DQomZ3Q7IDxicj4NCiZndDsgMjAxNS0wOC0yOCAxNzo0NDoyMiw2NjQgSU5GTyZu
YnNwOyA8YnI+DQomZ3Q7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLlZkc1VwZGF0
ZVJ1blRpbWVJbmZvXSA8YnI+DQomZ3Q7IChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci02
MCkgWzRlZjkwYjEyXSBWTSBEQyA8YnI+DQomZ3Q7IDBmM2QxZjA2LWU1MTYtNDhjZS1hYTZmLTcy
NzNjMzNkMzQ5MSBtb3ZlZCBmcm9tIFVwIC0tJmd0OyBOb3RSZXNwb25kaW5nPGJyPg0KJmd0OyA8
YnI+DQomZ3Q7IDIwMTUtMDgtMjggMTc6NDQ6MjIsNjkyIFdBUk4mbmJzcDsgPGJyPg0KJmd0OyBb
b3JnLm92aXJ0LmVuZ2luZS5jb3JlLmRhbC5kYmJyb2tlci5hdWRpdGxvZ2hhbmRsaW5nLkF1ZGl0
TG9nRGlyZWN0b3JdIDxicj4NCiZndDsgKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTYw
KSBbNGVmOTBiMTJdIENvcnJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOg0KPGJyPg0KJmd0
OyBudWxsLCBDdXN0b20gRXZlbnQgSUQ6IC0xLCBNZXNzYWdlOiBWTSBEQyBpcyBub3QgcmVzcG9u
ZGluZy48YnI+DQomZ3Q7IDxicj4NCiZndDsgRG9lcyBhbnlib2R5IGtub3cgd2hhdCBJIGNhbiBk
bz8gV2hlcmUgc2hvdWxkIEkgaGF2ZSBhIGxvb2s/IEhpbnRzIGFyZSBncmVhdGx5IDxicj4NCiZn
dDsgYXBwcmVjaWF0ZWQhPGJyPg0KJmd0OyA8YnI+DQomZ3Q7IFRoYW5rcyw8YnI+DQomZ3Q7IDxi
cj4NCiZndDsgQ2hyaXN0aWFuPGJyPg0KJmd0OyA8YnI+DQo8YnI+DQotLSA8YnI+DQpEYW5pZWwg
SGVsZ2VuYmVyZ2VyPGJyPg0KbSBib3ggYmV3ZWd0YmlsZCBHbWJIPGJyPg0KPGJyPg0KUDogJiM0
Mzs0OS8zMC8yNDA4NzgxLTIyPGJyPg0KRjogJiM0Mzs0OS8zMC8yNDA4NzgxLTEwPGJyPg0KPGJy
Pg0KQUNLRVJTVFIuIDE5PGJyPg0KRC0xMDExNSBCRVJMSU48YnI+DQo8YnI+DQo8YnI+DQo8YSBo
cmVmPSJodHRwOi8vd3d3Lm0tYm94LmRlIj53d3cubS1ib3guZGU8L2E+Jm5ic3A7IDxhIGhyZWY9
Imh0dHA6Ly93d3cubW9ua2V5bWVuLnR2Ij4NCnd3dy5tb25rZXltZW4udHY8L2E+PGJyPg0KPGJy
Pg0KR2VzY2jDpGZ0c2bDvGhyZXI6IE1hcnRpbiBSZXRzY2hpdHplZ2dlciAvIE1pY2hhZWxhIEfD
tmxsbmVyPGJyPg0KSGFuZGVzbHJlZ2lzdGVyOiBBbXRzZ2VyaWNodCBDaGFybG90dGVuYnVyZyAv
IEhSQiAxMTI3Njc8YnI+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPC9kaXY+DQo8
L2JvZHk+DQo8L2h0bWw+DQo=
--_000_354c6d82d06343f59dfebdb779762820emailandroidcom_--
------=_NextPartTM-000-842b7e16-7dd5-497a-b341-090e5ac35743
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-842b7e16-7dd5-497a-b341-090e5ac35743--
10:24 p.m.
New subject: Some VMs in status "not responding" in oVirt interface
This is a multipart message in MIME format.
------=_NextPart_000_0479_01D0EFFC.E11214C0
Content-Type: text/plain;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Hi Markus,
=20
gdb is available on CentOS 7, but what do you mean by qemu-debug? I =
Installed qemu-kvm-tools, maybe this is the pendant for CentOS?
=20
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools
=20
Regards, Christian
=20
Von: Markus Stockhausen [mailto:stockhausen@collogia.de]=20
Gesendet: Dienstag, 15. September 2015 20:40
An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>
Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com; =
users(a)ovirt.org
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in oVirt =
interface
=20
Do you have a chance to install qemu-debug? If yes I would try a =
backtrace.
gdb -p <qemu-pid>
# bt
Markus
Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger =
<daniel.helgenberger(a)m-box.de <mailto:daniel.helgenberger@m-box.de> >:
Hello,
I do not want to hijack the thread but maybe my issue is related?
It might have started with ovirt 3.5.3; but I cannot tell for sure.
For me, one vm (foreman) is affected; the second time in 14 days. I can =
confirm this as I also loose any network connection to the VM and
the ability to connect a console.
Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid =
of qemu-kvm process>'
As far as I can tell the VM became unresponsive at around Sep 15 =
12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs (see
attached).
Below the engine.log part.
Versions:
ovirt-engine-3.5.4.2-1.el7.centos.noarch
vdsm-4.16.26-0.el7.centos
libvirt-1.2.8-16.el7_1.3
engine.log (1200 - 1300:
2015-09-15 12:03:47,949 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-56) [264d502a] HA
reservation status for cluster Default is OK
2015-09-15 12:08:02,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] =
(DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update
VMs/Templates Ovf.
2015-09-15 12:08:02,709 INFO =
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] =
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand =
internal: true. Entities affected : ID:
00000002-0002-0002-0002-000000000088 Type: l
2015-09-15 12:08:02,780 INFO =
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] =
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks=3D key: =
00000002-0002-0002-0002-000000000088 value: OVF_UPDATE
2015-09-15 12:08:47,997 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-21) [3fc854a2] HA
reservation status for cluster Default is OK
2015-09-15 12:13:06,998 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] =
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] START, GetFileStatsVDSCommand( storagePoolId =3D =
00000002-0002-0002-0002-000000000088, ignoreFailoverLimit =3D false), =
log id: 1503968
2015-09-15 12:13:07,137 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] =
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] FINISH, GetFileStatsVDSCommand, return: =
{pfSense-2.0-RELEASE-i386.iso=3D{status=3D0, ctime=3D1432286887.0, =
size=3D115709952},
Fedora-15-i686-Live8
2015-09-15 12:13:07,178 INFO =
[org.ovirt.engine.core.bll.IsoDomainListSyncronizer] =
(org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished
automatic refresh process for ISO file type with success, for storage =
domain id 84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.
2015-09-15 12:13:48,043 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA
reservation status for cluster Default is OK
2015-09-15 12:18:48,088 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-44) [6345e698] HA
reservation status for cluster Default is OK
2015-09-15 12:23:48,137 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-13) HA reservation
status for cluster Default is OK
2015-09-15 12:28:48,183 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-76) [154c91d5] HA
reservation status for cluster Default is OK
2015-09-15 12:33:48,229 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-36) [27c73ac6] HA
reservation status for cluster Default is OK
2015-09-15 12:34:49,432 INFO =
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] =
(DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM
foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> =
NotResponding
2015-09-15 12:34:49,578 WARN =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler_Worker-41)
[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, =
Message: VM foreman is not responding.
2015-09-15 12:38:48,273 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-10) [7a800766] HA
reservation status for cluster Default is OK
2015-09-15 12:43:48,320 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-42) [440f1c40] HA
reservation status for cluster Default is OK
2015-09-15 12:48:48,366 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-70) HA reservation
status for cluster Default is OK
2015-09-15 12:53:48,412 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-12) [50221cdc] HA
reservation status for cluster Default is OK
2015-09-15 12:58:48,459 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-3) HA reservation
status for cluster Default is OK
On 29.08.2015 22:48, Christian Hailer wrote:
Hello,
=20
last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a =
single
Centos=20
7 server, so I started by suspending the VMs in order to set the
oVirt =
engine=20
host to maintenance mode. During the process of suspending the VMs
the =
server=20
crashed, kernel panic=E2=80=A6
=20
After restarting the server I installed the updates via yum an =
restarted the=20
server again. Afterwards, all the VMs could be started again. Some =
hours later=20
my monitoring system registered some unresponsive hosts, I had a look
=
in the=20
oVirt interface, 3 of the VMs were in the state =E2=80=9Cnot =
responding=E2=80=9D, marked by a=20
question mark.
=20
I tried to shut down the VMs, but oVirt wasn=E2=80=99t able to do so. =
I tried to
reset=20
the status in the database with the sql statement
=20
update vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid =
from
vm_static=20
where vm_name =3D 'MYVMNAME');
=20
but that didn=E2=80=99t help, either. Only rebooting the whole =
hypervisor
helped=E2=80=A6=20
afterwards everything worked again. But only for a few hours, then
one =
of the=20
VMs entered the =E2=80=9Cnot responding=E2=80=9D state again=E2=80=A6
=
again only a reboot helped.=20
Yesterday it happened again:
=20
2015-08-28 17:44:22,664 INFO =20
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]=20
(DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC=20
0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
=20
2015-08-28 17:44:22,692 WARN =20
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, =
Call Stack:=20
null, Custom Event ID: -1, Message: VM DC is not responding.
=20
Does anybody know what I can do? Where should I have a look? Hints are =
greatly=20
appreciated!
=20
Thanks,
=20
Christian
=20
--=20
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de <http://www.m-box.de> www.monkeymen.tv =
<http://www.monkeymen.tv>=20
Gesch=C3=A4ftsf=C3=BChrer: Martin Retschitzegger / Michaela G=C3=B6llner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
------=_NextPart_000_0479_01D0EFFC.E11214C0
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta =
name=3DGenerator content=3D"Microsoft Word 15 (filtered =
medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-MailFormatvorlage19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DDE
link=3Dblue =
vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>Hi
Markus,<o:p></o:p></span></p><p =
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'><o:p> </o:p></span></p><p
=
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>gdb is available on CentOS 7, but what do =
you mean by qemu-debug? I Installed qemu-kvm-tools, maybe this is the =
pendant for CentOS?<o:p></o:p></span></p><p
class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'><o:p> </o:p></span></p><p
=
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>qemu-kvm-tools.x86_64 : KVM debugging and =
diagnostics tools<o:p></o:p></span></p><p
class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>qemu-kvm-tools-ev.x86_64 : KVM debugging and =
diagnostics tools<o:p></o:p></span></p><p
class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>qemu-kvm-tools-rhev.x86_64 : KVM debugging =
and diagnostics tools<o:p></o:p></span></p><p
class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'><o:p> </o:p></span></p><p
=
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'>Regards,
Christian<o:p></o:p></span></p><p =
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;=
mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div
=
style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm'><p class=3DMsoNormal><b><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>Von:</span></=
b><span lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> Markus
=
Stockhausen [mailto:stockhausen@collogia.de] =
<br><b>Gesend</b></span><b><span =
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>et:</span></b=
<span
style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> =
Dienstag, 15. September 2015 20:40<br><b>An:</b> Daniel
Helgenberger =
<daniel.helgenberger@m-box.de><br><b>Cc:</b> Christian
Hailer =
&lt;christian(a)hailer.eu&gt;; ydary(a)redhat.com; =
users@ovirt.org<br><b>Betreff:</b> Re: [ovirt-users] Some VMs in status
=
"not responding" in oVirt =
interface<o:p></o:p></span></p></div></div><p =
class=3DMsoNormal><o:p> </o:p></p><p>Do you have a
chance to =
install qemu-debug? If yes I would try a
backtrace.<o:p></o:p></p><p>gdb =
-p <qemu-pid><br># =
bt<o:p></o:p></p><p>Markus<o:p></o:p></p><div><p
class=3DMsoNormal>Am =
15.09.2015 4:15 nachm. schrieb Daniel Helgenberger <<a =
href=3D"mailto:daniel.helgenberger@m-box.de">daniel.helgenberger@m-box.de=
</a>>:<o:p></o:p></p><blockquote =
style=3D'border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm =
6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p =
class=3DMsoNormal>Hello,<br><br>I do not want to hijack the thread but =
maybe my issue is related?<br><br>It might have started with ovirt =
3.5.3; but I cannot tell for sure.<br><br>For me, one vm (foreman) is =
affected; the second time in 14 days. I can confirm this as I also loose =
any network connection to the VM and<br>the ability to connect a =
console.<br>Also, the only thing witch 'fixes' the issue is right now =
'kill -9 <pid of qemu-kvm process>'<br><br>As far as I
can tell =
the VM became unresponsive at around Sep 15 12:30:01; engine logged this =
at 12:34. Nothing obvious in VDSM logs (see<br>attached).<br><br>Below
=
the engine.log =
part.<br><br>Versions:<br>ovirt-engine-3.5.4.2-1.el7.centos.noarch<br><br=
vdsm-4.16.26-0.el7.centos<br>libvirt-1.2.8-16.el7_1.3<br><br>engine.log
=
(1200 - 1300:<br>2015-09-15 12:03:47,949 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-56) [264d502a] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:08:02,708 INFO =
[org.ovirt.engine.core.bll.OvfDataUpdater] =
(DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to =
update<br>VMs/Templates Ovf.<br>2015-09-15 12:08:02,709 INFO =
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] =
(DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Running command: =
ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected =
: ID:<br>00000002-0002-0002-0002-000000000088 Type: =
l<br>2015-09-15 12:08:02,780 INFO =
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] =
(DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Lock freed to object =
EngineLock [exclusiveLocks=3D key: 00000002-0002-0002-0002-000000000088 =
value: OVF_UPDATE<br>2015-09-15 12:08:47,997 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-21) [3fc854a2] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:13:06,998 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] =
(org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] START, =
GetFileStatsVDSCommand( storagePoolId =3D =
00000002-0002-0002-0002-000000000088, ignoreFailoverLimit =3D false), =
log id: 1503968<br>2015-09-15 12:13:07,137 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] =
(org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] FINISH, =
GetFileStatsVDSCommand, return: =
{pfSense-2.0-RELEASE-i386.iso=3D{status=3D0, ctime=3D1432286887.0, =
size=3D115709952},<br>Fedora-15-i686-Live8<br>2015-09-15 12:13:07,178 =
INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] =
(org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished<br>automatic =
refresh process for ISO file type with success, for storage domain id =
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.<br>2015-09-15 12:13:48,043 =
INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:18:48,088 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-44) [6345e698] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:23:48,137 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-13) HA reservation<br>status for cluster =
Default is OK<br>2015-09-15 12:28:48,183 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-76) [154c91d5] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:33:48,229 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-36) [27c73ac6] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:34:49,432 INFO =
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] =
(DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM<br>foreman =
8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> =
NotResponding<br>2015-09-15 12:34:49,578 WARN =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler_Worker-41)<br>[5f2a4b68] Correlation ID: null, =
Call Stack: null, Custom Event ID: -1, Message: VM foreman is not =
responding.<br>2015-09-15 12:38:48,273 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-10) [7a800766] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:43:48,320 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-42) [440f1c40] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:48:48,366 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-70) HA reservation<br>status for cluster =
Default is OK<br>2015-09-15 12:53:48,412 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-12) [50221cdc] HA<br>reservation status =
for cluster Default is OK<br>2015-09-15 12:58:48,459 INFO =
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling] =
(DefaultQuartzScheduler_Worker-3) HA reservation<br>status for cluster =
Default is OK<br><br><br><br>On 29.08.2015 22:48, Christian Hailer
=
wrote:<br>> Hello,<br>> <br>> last Wednesday I
wanted to update =
my oVirt 3.5 hypervisor. It is a single Centos <br>> 7 server, so I =
started by suspending the VMs in order to set the oVirt engine <br>> =
host to maintenance mode. During the process of suspending the VMs the =
server <br>> crashed, kernel panic=E2=80=A6<br>>
<br>> After =
restarting the server I installed the updates via yum an restarted the =
<br>> server again. Afterwards, all the VMs could be started again. =
Some hours later <br>> my monitoring system registered some =
unresponsive hosts, I had a look in the <br>> oVirt interface, 3 of =
the VMs were in the state =E2=80=9Cnot responding=E2=80=9D, marked by a =
<br>> question mark.<br>> <br>> I tried to shut
down the VMs, =
but oVirt wasn=E2=80=99t able to do so. I tried to reset <br>> the =
status in the database with the sql statement<br>> <br>> update
=
vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid from =
vm_static <br>> where vm_name =3D 'MYVMNAME');<br>>
<br>> but =
that didn=E2=80=99t help, either. Only rebooting the whole hypervisor =
helped=E2=80=A6 <br>> afterwards everything worked again. But only =
for a few hours, then one of the <br>> VMs entered the =E2=80=9Cnot =
responding=E2=80=9D state again=E2=80=A6 again only a reboot helped. =
<br>> Yesterday it happened again:<br>> <br>>
2015-08-28 =
17:44:22,664 INFO <br>> =
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] <br>> =
(DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC <br>> =
0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> =
NotResponding<br>> <br>> 2015-08-28 17:44:22,692 WARN
=
<br>> =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
<br>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: =
null, Call Stack: <br>> null, Custom Event ID: -1, Message: VM DC is =
not responding.<br>> <br>> Does anybody know what I can do?
Where =
should I have a look? Hints are greatly <br>> appreciated!<br>>
=
<br>> Thanks,<br>> <br>>
Christian<br>> <br><br>-- =
<br>Daniel Helgenberger<br>m box bewegtbild GmbH<br><br>P: =
+49/30/2408781-22<br>F: +49/30/2408781-10<br><br>ACKERSTR.
19<br>D-10115 =
BERLIN<br><br><br><a
href=3D"http://www.m-box.de">www.m-box.de</a> =
<a =
href=3D"http://www.monkeymen.tv">www.monkeymen.tv</a><br><br>Gesch=C3=A4f=
tsf=C3=BChrer: Martin Retschitzegger / Michaela =
G=C3=B6llner<br>Handeslregister: Amtsgericht Charlottenburg / HRB =
112767<o:p></o:p></p></div></div></blockquote></div></div></body></html>
------=_NextPart_000_0479_01D0EFFC.E11214C0--
10:30 p.m.
New subject: Some VMs in status "not responding" in oVirt interface
This is a multi-part message in MIME format.
------=_NextPartTM-000-c61d044c-4b16-4064-ab82-f5449098e861
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi Christian,=0A=
=0A=
I think of a package similar like this:=0A=
=0A=
qemu-debuginfo.x86_64 2:2.1.3-10.fc21=0A=
=0A=
That allows gdb to show information about backtrace symbols. See=0A=
comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=3D1262251=0A=
Makes error search much simpler - especially if qemu hangs.=0A=
=0A=
Markus=0A=
=0A=
**********************************=0A=
=0A=
Von: Christian Hailer [christian(a)hailer.eu]=0A=
=0A=
Gesendet: Dienstag, 15. September 2015 21:24=0A=
=0A=
An: Markus Stockhausen; 'Daniel Helgenberger'=0A=
=0A=
Cc: ydary(a)redhat.com; users(a)ovirt.org=0A=
=0A=
Betreff: AW: [ovirt-users] Some VMs in status "not responding" in oVirt int=
erface=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Hi Markus,=0A=
=0A=
gdb is available on CentOS 7, but what do you mean by qemu-debug? I Install=
ed qemu-kvm-tools, maybe this is the pendant for CentOS?=0A=
=0A=
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools=0A=
qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools=0A=
qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools=0A=
=0A=
Regards, Christian=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Von: Markus Stockhausen [mailto:stockhausen@collogia.de]=0A=
=0A=
=0A=
Gesendet: Dienstag, 15. September 2015 20:40=0A=
=0A=
An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>=0A=
=0A=
Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com; users(a)ovirt.o=
rg=0A=
=0A=
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in oVirt int=
erface=0A=
=0A=
=0A=
=0A=
Do you have a chance to install qemu-debug? If yes I would try a backtrace.=
=0A=
gdb -p <qemu-pid>=0A=
=0A=
# bt=0A=
Markus=0A=
=0A=
=0A=
Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger <daniel.helgenberger@=
m-box.de>:=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Hello,=0A=
=0A=
=0A=
=0A=
I do not want to hijack the thread but maybe my issue is related?=0A=
=0A=
=0A=
=0A=
It might have started with ovirt 3.5.3; but I cannot tell for sure.=0A=
=0A=
=0A=
=0A=
For me, one vm (foreman) is affected; the second time in 14 days. I can con=
firm this as I also loose any network connection to the VM and=0A=
=0A=
the ability to connect a console.=0A=
=0A=
Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid of
=
qemu-kvm process>'=0A=
=0A=
=0A=
=0A=
As far as I can tell the VM became unresponsive at around Sep 15 12:30:01; =
engine logged this at 12:34. Nothing obvious in VDSM logs (see=0A=
=0A=
attached).=0A=
=0A=
=0A=
=0A=
Below the engine.log part.=0A=
=0A=
=0A=
=0A=
Versions:=0A=
=0A=
ovirt-engine-3.5.4.2-1.el7.centos.noarch=0A=
=0A=
=0A=
=0A=
vdsm-4.16.26-0.el7.centos=0A=
=0A=
libvirt-1.2.8-16.el7_1.3=0A=
=0A=
=0A=
=0A=
engine.log (1200 - 1300:=0A=
=0A=
2015-09-15 12:03:47,949 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-56) [264d502a] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:08:02,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (D=
efaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update=0A=
=0A=
VMs/Templates Ovf.=0A=
=0A=
2015-09-15 12:08:02,709 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateFo=
rStoragePoolCommand] (DefaultQuartzScheduler_Worker-89)=0A=
=0A=
[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal:=
true. Entities affected : ID:=0A=
=0A=
00000002-0002-0002-0002-000000000088 Type: l=0A=
=0A=
2015-09-15 12:08:02,780 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateFo=
rStoragePoolCommand] (DefaultQuartzScheduler_Worker-89)=0A=
=0A=
[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks=3D key: 00000002=
-0002-0002-0002-000000000088 value: OVF_UPDATE=0A=
=0A=
2015-09-15 12:08:47,997 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:13:06,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge=
tFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48)=0A=
=0A=
[50221cdc] START, GetFileStatsVDSCommand( storagePoolId =3D 00000002-0002-0=
002-0002-000000000088, ignoreFailoverLimit =3D false), log id: 1503968=0A=
=0A=
2015-09-15 12:13:07,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge=
tFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48)=0A=
=0A=
[50221cdc] FINISH, GetFileStatsVDSCommand, return: {pfSense-2.0-RELEASE-i38=
6.iso=3D{status=3D0, ctime=3D1432286887.0, size=3D115709952},=0A=
=0A=
Fedora-15-i686-Live8=0A=
=0A=
2015-09-15 12:13:07,178 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncr=
onizer] (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished=0A=
=0A=
automatic refresh process for ISO file type with success, for storage domai=
n id 84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.=0A=
=0A=
2015-09-15 12:13:48,043 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:18:48,088 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-44) [6345e698] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:23:48,137 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-13) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
2015-09-15 12:28:48,183 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-76) [154c91d5] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:33:48,229 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:34:49,432 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRun=
TimeInfo] (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM=0A=
=0A=
foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> NotRespondin=
g=0A=
=0A=
2015-09-15 12:34:49,578 WARN [org.ovirt.engine.core.dal.dbbroker.auditlogh=
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41)=0A=
=0A=
[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Mes=
sage: VM foreman is not responding.=0A=
=0A=
2015-09-15 12:38:48,273 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-10) [7a800766] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:43:48,320 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-42) [440f1c40] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:48:48,366 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-70) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
2015-09-15 12:53:48,412 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-12) [50221cdc] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:58:48,459 INFO [org.ovirt.engine.core.bll.scheduling.HaReser=
vationHandling] (DefaultQuartzScheduler_Worker-3) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
On 29.08.2015 22:48, Christian Hailer wrote:=0A=
=0A=
Hello,=0A=
=0A=
=0A=
=0A=
last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a
single=
Centos=0A=
=0A=
=0A=
7 server, so I started by suspending the VMs in order to set the
oVirt en=
gine =0A=
=0A=
host to maintenance mode. During the process of suspending the VMs
the se=
rver =0A=
=0A=
crashed, kernel panic=85=0A=
=0A=
=0A=
=0A=
After restarting the server I installed the updates via yum an
restarted =
the =0A=
=0A=
server again. Afterwards, all the VMs could be started again. Some
hours =
later =0A=
=0A=
my monitoring system registered some unresponsive hosts, I had a look
in =
the =0A=
=0A=
oVirt interface, 3 of the VMs were in the state =93not responding=94,
mar=
ked by a =0A=
=0A=
question mark.=0A=
=0A=
=0A=
=0A=
I tried to shut down the VMs, but oVirt wasn=92t able to do so. I
tried t=
o reset =0A=
=0A=
the status in the database with the sql statement=0A=
=0A=
=0A=
=0A=
update vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid
from=
vm_static=0A=
=0A=
=0A=
where vm_name =3D 'MYVMNAME');=0A=
=0A=
=0A=
=0A=
but that didn=92t help, either. Only rebooting the whole hypervisor
helpe=
d=85 =0A=
=0A=
afterwards everything worked again. But only for a few hours, then
one of=
the =0A=
=0A=
VMs entered the =93not responding=94 state again=85 again only a
reboot h=
elped. =0A=
=0A=
Yesterday it happened again:=0A=
=0A=
=0A=
=0A=
2015-08-28 17:44:22,664 INFO =0A=
=0A=
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] =0A=
=0A=
(DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC =0A=
=0A=
0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up -->
NotResponding=0A=
=0A=
=0A=
=0A=
2015-08-28 17:44:22,692 WARN =0A=
=0A=
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
=0A=
=0A=
(DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null,
Call =
Stack:=0A=
=0A=
=0A=
null, Custom Event ID: -1, Message: VM DC is not responding.=0A=
=0A=
=0A=
=0A=
Does anybody know what I can do? Where should I have a look? Hints
are gr=
eatly =0A=
=0A=
appreciated!=0A=
=0A=
=0A=
=0A=
Thanks,=0A=
=0A=
=0A=
=0A=
Christian=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
-- =0A=
=0A=
Daniel Helgenberger=0A=
=0A=
m box bewegtbild GmbH=0A=
=0A=
=0A=
=0A=
P: +49/30/2408781-22=0A=
=0A=
F: +49/30/2408781-10=0A=
=0A=
=0A=
=0A=
ACKERSTR. 19=0A=
=0A=
D-10115 BERLIN=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
www.m-box.de =0A=
=0A=
www.monkeymen.tv=0A=
=0A=
=0A=
=0A=
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner=0A=
=0A=
Handeslregister: Amtsgericht Charlottenburg / HRB 112767=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
------=_NextPartTM-000-c61d044c-4b16-4064-ab82-f5449098e861
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-c61d044c-4b16-4064-ab82-f5449098e861--
11:23 p.m.
New subject: Some VMs in status "not responding" in oVirt interface
On 15.09.2015 21:31, Markus Stockhausen wrote:
Hi Christian,
I think of a package similar like this:
qemu-debuginfo.x86_64 2:2.1.3-10.fc21
That allows gdb to show information about backtrace symbols. See
comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=1262251
Makes error search much simpler - especially if qemu hangs.
Markus, thanks for the BZ. I think I do see the same issue. Actually my VM is currently
the only with a live snapshot and (puppetmaster)
does a lot of I/O.
Christian, maybe this BZ1262251 also applicable?
I'll go ahead and delete the live snapshot. If I see this issue again I will submit
the trace to your BZ.
Markus
**********************************
Von: Christian Hailer [christian(a)hailer.eu]
Gesendet: Dienstag, 15. September 2015 21:24
An: Markus Stockhausen; 'Daniel Helgenberger'
Cc: ydary(a)redhat.com; users(a)ovirt.org
Betreff: AW: [ovirt-users] Some VMs in status "not responding" in oVirt
interface
Hi Markus,
gdb is available on CentOS 7, but what do you mean by qemu-debug? I Installed
qemu-kvm-tools, maybe this is the pendant for CentOS?
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools
Regards, Christian
Von: Markus Stockhausen [mailto:stockhausen@collogia.de]
Gesendet: Dienstag, 15. September 2015 20:40
An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>
Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com; users(a)ovirt.org
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in oVirt
interface
Do you have a chance to install qemu-debug? If yes I would try a backtrace.
gdb -p <qemu-pid>
# bt
Markus
Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger
<daniel.helgenberger(a)m-box.de>:
Hello,
I do not want to hijack the thread but maybe my issue is related?
It might have started with ovirt 3.5.3; but I cannot tell for sure.
For me, one vm (foreman) is affected; the second time in 14 days. I can confirm this as I
also loose any network connection to the VM and
the ability to connect a console.
Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid of
qemu-kvm process>'
As far as I can tell the VM became unresponsive at around Sep 15 12:30:01; engine logged
this at 12:34. Nothing obvious in VDSM logs (see
attached).
Below the engine.log part.
Versions:
ovirt-engine-3.5.4.2-1.el7.centos.noarch
vdsm-4.16.26-0.el7.centos
libvirt-1.2.8-16.el7_1.3
engine.log (1200 - 1300:
2015-09-15 12:03:47,949 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-56) [264d502a] HA
reservation status for cluster Default is OK
2015-09-15 12:08:02,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater]
(DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update
VMs/Templates Ovf.
2015-09-15 12:08:02,709 INFO
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true.
Entities affected : ID:
00000002-0002-0002-0002-000000000088 Type: l
2015-09-15 12:08:02,780 INFO
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks= key:
00000002-0002-0002-0002-000000000088 value: OVF_UPDATE
2015-09-15 12:08:47,997 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-21) [3fc854a2] HA
reservation status for cluster Default is OK
2015-09-15 12:13:06,998 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] START, GetFileStatsVDSCommand( storagePoolId =
00000002-0002-0002-0002-000000000088, ignoreFailoverLimit = false), log id: 1503968
2015-09-15 12:13:07,137 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] FINISH, GetFileStatsVDSCommand, return:
{pfSense-2.0-RELEASE-i386.iso={status=0, ctime=1432286887.0, size=115709952},
Fedora-15-i686-Live8
2015-09-15 12:13:07,178 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer]
(org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished
automatic refresh process for ISO file type with success, for storage domain id
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.
2015-09-15 12:13:48,043 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA
reservation status for cluster Default is OK
2015-09-15 12:18:48,088 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-44) [6345e698] HA
reservation status for cluster Default is OK
2015-09-15 12:23:48,137 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-13) HA reservation
status for cluster Default is OK
2015-09-15 12:28:48,183 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-76) [154c91d5] HA
reservation status for cluster Default is OK
2015-09-15 12:33:48,229 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-36) [27c73ac6] HA
reservation status for cluster Default is OK
2015-09-15 12:34:49,432 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM
foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> NotResponding
2015-09-15 12:34:49,578 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-41)
[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM
foreman is not responding.
2015-09-15 12:38:48,273 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-10) [7a800766] HA
reservation status for cluster Default is OK
2015-09-15 12:43:48,320 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-42) [440f1c40] HA
reservation status for cluster Default is OK
2015-09-15 12:48:48,366 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-70) HA reservation
status for cluster Default is OK
2015-09-15 12:53:48,412 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-12) [50221cdc] HA
reservation status for cluster Default is OK
2015-09-15 12:58:48,459 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-3) HA reservation
status for cluster Default is OK
On 29.08.2015 22:48, Christian Hailer wrote:
> Hello,
>
> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a single Centos
> 7 server, so I started by suspending the VMs in order to set the oVirt engine
> host to maintenance mode. During the process of suspending the VMs the server
> crashed, kernel panic…
>
> After restarting the server I installed the updates via yum an restarted the
> server again. Afterwards, all the VMs could be started again. Some hours later
> my monitoring system registered some unresponsive hosts, I had a look in the
> oVirt interface, 3 of the VMs were in the state “not responding”, marked by a
> question mark.
>
> I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried to reset
> the status in the database with the sql statement
>
> update vm_dynamic set status = 0 where vm_guid = (select vm_guid from vm_static
> where vm_name = 'MYVMNAME');
>
> but that didn’t help, either. Only rebooting the whole hypervisor helped…
> afterwards everything worked again. But only for a few hours, then one of the
> VMs entered the “not responding” state again… again only a reboot helped.
> Yesterday it happened again:
>
> 2015-08-28 17:44:22,664 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC
> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
>
> 2015-08-28 17:44:22,692 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack:
> null, Custom Event ID: -1, Message: VM DC is not responding.
>
> Does anybody know what I can do? Where should I have a look? Hints are greatly
> appreciated!
>
> Thanks,
>
> Christian
>
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
8:39 a.m.
New subject: Some VMs in status "not responding" in oVirt interface
Hi,
just to get it straight: most of my VMs had one or more existing snapshots. Do you think
this is a problem currently? If I understand it correctly the BZ of Markus concerns only a
short period of time while removing a snapshot, but my VMs stopped responding in the
middle of the night without any interaction...
I deleted all the snapshots, just in case :) my system is running fine for nearly three
days now, I'm not quite sure but I think it helped that I changed the HDD and NIC of
the Windows 2012 VMs to VirtIO devices...
Best regards, Christian
-----Ursprüngliche Nachricht-----
Von: Daniel Helgenberger [mailto:daniel.helgenberger@m-box.de]
Gesendet: Dienstag, 15. September 2015 22:24
An: Markus Stockhausen <stockhausen(a)collogia.de>; Christian Hailer
<christian(a)hailer.eu>
Cc: ydary(a)redhat.com; users(a)ovirt.org
Betreff: Re: AW: [ovirt-users] Some VMs in status "not responding" in oVirt
interface
On 15.09.2015 21:31, Markus Stockhausen wrote:
Hi Christian,
I think of a package similar like this:
qemu-debuginfo.x86_64 2:2.1.3-10.fc21
That allows gdb to show information about backtrace symbols. See
comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=1262251
Makes error search much simpler - especially if qemu hangs.
Markus, thanks for the BZ. I think I do see the same issue. Actually my VM is currently
the only with a live snapshot and (puppetmaster) does a lot of I/O.
Christian, maybe this BZ1262251 also applicable?
I'll go ahead and delete the live snapshot. If I see this issue again I will submit
the trace to your BZ.
Markus
**********************************
Von: Christian Hailer [christian(a)hailer.eu]
Gesendet: Dienstag, 15. September 2015 21:24
An: Markus Stockhausen; 'Daniel Helgenberger'
Cc: ydary(a)redhat.com; users(a)ovirt.org
Betreff: AW: [ovirt-users] Some VMs in status "not responding" in
oVirt interface
Hi Markus,
gdb is available on CentOS 7, but what do you mean by qemu-debug? I Installed
qemu-kvm-tools, maybe this is the pendant for CentOS?
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools
qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools
Regards, Christian
Von: Markus Stockhausen [mailto:stockhausen@collogia.de]
Gesendet: Dienstag, 15. September 2015 20:40
An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>
Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com;
users(a)ovirt.org
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in
oVirt interface
Do you have a chance to install qemu-debug? If yes I would try a backtrace.
gdb -p <qemu-pid>
# bt
Markus
Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger
<daniel.helgenberger(a)m-box.de>:
Hello,
I do not want to hijack the thread but maybe my issue is related?
It might have started with ovirt 3.5.3; but I cannot tell for sure.
For me, one vm (foreman) is affected; the second time in 14 days. I
can confirm this as I also loose any network connection to the VM and
the ability to connect a console.
Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid of
qemu-kvm process>'
As far as I can tell the VM became unresponsive at around Sep 15
12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs
(see
attached).
Below the engine.log part.
Versions:
ovirt-engine-3.5.4.2-1.el7.centos.noarch
vdsm-4.16.26-0.el7.centos
libvirt-1.2.8-16.el7_1.3
engine.log (1200 - 1300:
2015-09-15 12:03:47,949 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-56) [264d502a] HA
reservation status for cluster Default is OK
2015-09-15 12:08:02,708 INFO
[org.ovirt.engine.core.bll.OvfDataUpdater]
(DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update
VMs/Templates Ovf.
2015-09-15 12:08:02,709 INFO
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true.
Entities affected : ID:
00000002-0002-0002-0002-000000000088 Type: l
2015-09-15 12:08:02,780 INFO
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
(DefaultQuartzScheduler_Worker-89)
[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks= key:
00000002-0002-0002-0002-000000000088 value: OVF_UPDATE
2015-09-15 12:08:47,997 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-21) [3fc854a2] HA
reservation status for cluster Default is OK
2015-09-15 12:13:06,998 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] START, GetFileStatsVDSCommand( storagePoolId =
00000002-0002-0002-0002-000000000088, ignoreFailoverLimit = false),
log id: 1503968
2015-09-15 12:13:07,137 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
(org.ovirt.thread.pool-8-thread-48)
[50221cdc] FINISH, GetFileStatsVDSCommand, return:
{pfSense-2.0-RELEASE-i386.iso={status=0, ctime=1432286887.0,
size=115709952},
Fedora-15-i686-Live8
2015-09-15 12:13:07,178 INFO
[org.ovirt.engine.core.bll.IsoDomainListSyncronizer]
(org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished
automatic refresh process for ISO file type with success, for storage domain id
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.
2015-09-15 12:13:48,043 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA
reservation status for cluster Default is OK
2015-09-15 12:18:48,088 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-44) [6345e698] HA
reservation status for cluster Default is OK
2015-09-15 12:23:48,137 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-13) HA reservation
status for cluster Default is OK
2015-09-15 12:28:48,183 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-76) [154c91d5] HA
reservation status for cluster Default is OK
2015-09-15 12:33:48,229 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-36) [27c73ac6] HA
reservation status for cluster Default is OK
2015-09-15 12:34:49,432 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM
foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up -->
NotResponding
2015-09-15 12:34:49,578 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-41)
[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM
foreman is not responding.
2015-09-15 12:38:48,273 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-10) [7a800766] HA
reservation status for cluster Default is OK
2015-09-15 12:43:48,320 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-42) [440f1c40] HA
reservation status for cluster Default is OK
2015-09-15 12:48:48,366 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-70) HA reservation
status for cluster Default is OK
2015-09-15 12:53:48,412 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-12) [50221cdc] HA
reservation status for cluster Default is OK
2015-09-15 12:58:48,459 INFO
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
(DefaultQuartzScheduler_Worker-3) HA reservation
status for cluster Default is OK
On 29.08.2015 22:48, Christian Hailer wrote:
> Hello,
>
> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a
> single Centos
> 7 server, so I started by suspending the VMs in order to set the
> oVirt engine
> host to maintenance mode. During the process of suspending the VMs
> the server
> crashed, kernel panic…
>
> After restarting the server I installed the updates via yum an
> restarted the
> server again. Afterwards, all the VMs could be started again. Some
> hours later
> my monitoring system registered some unresponsive hosts, I had a look
> in the
> oVirt interface, 3 of the VMs were in the state “not responding”,
> marked by a
> question mark.
>>
> I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried
> to reset
> the status in the database with the sql statement
>
> update vm_dynamic set status = 0 where vm_guid = (select vm_guid from
> vm_static
> where vm_name = 'MYVMNAME');
>
> but that didn’t help, either. Only rebooting the whole hypervisor
> helped…
> afterwards everything worked again. But only for a few hours, then
> one of the
> VMs entered the “not responding” state again… again only a reboot helped.
> Yesterday it happened again:
>
> 2015-08-28 17:44:22,664 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC
> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
>
> 2015-08-28 17:44:22,692 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ]
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack:
> null, Custom Event ID: -1, Message: VM DC is not responding.
>
> Does anybody know what I can do? Where should I have a look? Hints
> are greatly
> appreciated!
>
> Thanks,
>
> Christian
>
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
8:48 a.m.
New subject: Some VMs in status "not responding" in oVirt interface
This is a multi-part message in MIME format.
------=_NextPartTM-000-4a9428dc-2b1c-4caf-8ee3-a447a009db9b
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi Christian,=0A=
=0A=
you are right. These are different errors. In the past we had VM hangs or c=
rashes from time to time.=0A=
So we implemented 2 features for mitigation. That helped to analyze a lot o=
f problems.=0A=
=0A=
1) Install debug packages to get backtraces in case of stalled qemu process=
es.=0A=
2) Setup the hosts to generate coredumps for qemu:=0A=
=0A=
a) /etc/security/limits.conf=0A=
* soft core unlimited=0A=
b) /usr/lib/systemd/system/libvirtd.service section Service=0A=
LimitCORE=3Dinfinity=0A=
c) create new before vdsm start hook to disable full qemu memory dump=0A=
libvirtpid=3D`ps -ef | grep libvirt | grep listen | awk '{ print $2 }'`=
=0A=
echo 0 > /proc/$libvirtpid/coredump_filter=0A=
=0A=
Best regards.=0A=
=0A=
Markus=0A=
=0A=
________________________________________=0A=
Von: Christian Hailer [christian(a)hailer.eu]=0A=
Gesendet: Donnerstag, 17. September 2015 07:39=0A=
An: 'Daniel Helgenberger'; Markus Stockhausen=0A=
Cc: ydary(a)redhat.com; users(a)ovirt.org=0A=
Betreff: AW: AW: [ovirt-users] Some VMs in status "not responding" in oVirt=
interface=0A=
=0A=
Hi,=0A=
=0A=
just to get it straight: most of my VMs had one or more existing snapshots.=
Do you think this is a problem currently? If I understand it correctly the=
BZ of Markus concerns only a short period of time while removing a snapsho=
t, but my VMs stopped responding in the middle of the night without any int=
eraction...=0A=
I deleted all the snapshots, just in case :) my system is running fine for =
nearly three days now, I'm not quite sure but I think it helped that I chan=
ged the HDD and NIC of the Windows 2012 VMs to VirtIO devices...=0A=
=0A=
Best regards, Christian=0A=
=0A=
-----Urspr=FCngliche Nachricht-----=0A=
Von: Daniel Helgenberger [mailto:daniel.helgenberger@m-box.de]=0A=
Gesendet: Dienstag, 15. September 2015 22:24=0A=
An: Markus Stockhausen <stockhausen(a)collogia.de>; Christian Hailer <christi=
an(a)hailer.eu>=0A=
Cc: ydary(a)redhat.com; users(a)ovirt.org=0A=
Betreff: Re: AW: [ovirt-users] Some VMs in status "not responding" in oVirt=
interface=0A=
=0A=
=0A=
=0A=
On 15.09.2015 21:31, Markus Stockhausen wrote:=0A=
Hi Christian,=0A=
=0A=
I think of a package similar like this:=0A=
=0A=
qemu-debuginfo.x86_64 2:2.1.3-10.fc21=0A=
=0A=
That allows gdb to show information about backtrace symbols. See=0A=
comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=3D1262251=0A=
Makes error search much simpler - especially if qemu hangs.=0A=
=0A=
Markus, thanks for the BZ. I think I do see the same issue. Actually my VM =
is currently the only with a live snapshot and (puppetmaster) does a lot of=
I/O.=0A=
=0A=
Christian, maybe this BZ1262251 also applicable?=0A=
=0A=
I'll go ahead and delete the live snapshot. If I see this issue again I wil=
l submit the trace to your BZ.=0A=
=0A=
=0A=
=0A=
Markus=0A=
=0A=
**********************************=0A=
=0A=
Von: Christian Hailer [christian(a)hailer.eu]=0A=
=0A=
Gesendet: Dienstag, 15. September 2015 21:24=0A=
=0A=
An: Markus Stockhausen; 'Daniel Helgenberger'=0A=
=0A=
Cc: ydary(a)redhat.com; users(a)ovirt.org=0A=
=0A=
Betreff: AW: [ovirt-users] Some VMs in status "not responding" in=0A=
oVirt interface=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Hi Markus,=0A=
=0A=
gdb is available on CentOS 7, but what do you mean by qemu-debug? I Insta=
lled
qemu-kvm-tools, maybe this is the pendant for CentOS?=0A=
=0A=
qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools=0A=
qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools=0A=
qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools=0A=
=0A=
Regards, Christian=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Von: Markus Stockhausen [mailto:stockhausen@collogia.de]=0A=
=0A=
=0A=
Gesendet: Dienstag, 15. September 2015 20:40=0A=
=0A=
An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>=0A=
=0A=
Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com;=0A=
users(a)ovirt.org=0A=
=0A=
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in=0A=
oVirt interface=0A=
=0A=
=0A=
=0A=
Do you have a chance to install qemu-debug? If yes I would try a backtrac=
e.=0A=
gdb -p <qemu-pid>=0A=
=0A=
# bt=0A=
Markus=0A=
=0A=
=0A=
Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger <daniel.helgenberge=
r@m-box.de>:=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
Hello,=0A=
=0A=
=0A=
=0A=
I do not want to hijack the thread but maybe my issue is related?=0A=
=0A=
=0A=
=0A=
It might have started with ovirt 3.5.3; but I cannot tell for sure.=0A=
=0A=
=0A=
=0A=
For me, one vm (foreman) is affected; the second time in 14 days. I=0A=
can confirm this as I also loose any network connection to the VM and=0A=
=0A=
the ability to connect a console.=0A=
=0A=
Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid o=
f qemu-kvm process>'=0A=
=0A=
=0A=
=0A=
As far as I can tell the VM became unresponsive at around Sep 15=0A=
12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs=0A=
(see=0A=
=0A=
attached).=0A=
=0A=
=0A=
=0A=
Below the engine.log part.=0A=
=0A=
=0A=
=0A=
Versions:=0A=
=0A=
ovirt-engine-3.5.4.2-1.el7.centos.noarch=0A=
=0A=
=0A=
=0A=
vdsm-4.16.26-0.el7.centos=0A=
=0A=
libvirt-1.2.8-16.el7_1.3=0A=
=0A=
=0A=
=0A=
engine.log (1200 - 1300:=0A=
=0A=
2015-09-15 12:03:47,949 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-56) [264d502a] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:08:02,708 INFO=0A=
[org.ovirt.engine.core.bll.OvfDataUpdater]=0A=
(DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update=0A=
=0A=
VMs/Templates Ovf.=0A=
=0A=
2015-09-15 12:08:02,709 INFO=0A=
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]=0A=
(DefaultQuartzScheduler_Worker-89)=0A=
=0A=
[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand interna=
l: true.
Entities affected : ID:=0A=
=0A=
00000002-0002-0002-0002-000000000088 Type: l=0A=
=0A=
2015-09-15 12:08:02,780 INFO=0A=
[org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]=0A=
(DefaultQuartzScheduler_Worker-89)=0A=
=0A=
[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks=3D key:=0A=
00000002-0002-0002-0002-000000000088 value: OVF_UPDATE=0A=
=0A=
2015-09-15 12:08:47,997 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-21) [3fc854a2] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:13:06,998 INFO=0A=
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]=0A=
(org.ovirt.thread.pool-8-thread-48)=0A=
=0A=
[50221cdc] START, GetFileStatsVDSCommand( storagePoolId =3D=0A=
00000002-0002-0002-0002-000000000088, ignoreFailoverLimit =3D false),=0A=
log id: 1503968=0A=
=0A=
2015-09-15 12:13:07,137 INFO=0A=
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]=0A=
(org.ovirt.thread.pool-8-thread-48)=0A=
=0A=
[50221cdc] FINISH, GetFileStatsVDSCommand, return:=0A=
{pfSense-2.0-RELEASE-i386.iso=3D{status=3D0, ctime=3D1432286887.0,=0A=
size=3D115709952},=0A=
=0A=
Fedora-15-i686-Live8=0A=
=0A=
2015-09-15 12:13:07,178 INFO=0A=
[org.ovirt.engine.core.bll.IsoDomainListSyncronizer]=0A=
(org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished=0A=
=0A=
automatic refresh process for ISO file type with success, for storage dom=
ain id
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.=0A=
=0A=
2015-09-15 12:13:48,043 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:18:48,088 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-44) [6345e698] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:23:48,137 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-13) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
2015-09-15 12:28:48,183 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-76) [154c91d5] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:33:48,229 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-36) [27c73ac6] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:34:49,432 INFO=0A=
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]=0A=
(DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM=0A=
=0A=
foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up -->=0A=
NotResponding=0A=
=0A=
2015-09-15 12:34:49,578 WARN=0A=
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]=0A=
(DefaultQuartzScheduler_Worker-41)=0A=
=0A=
[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, M=
essage:
VM foreman is not responding.=0A=
=0A=
2015-09-15 12:38:48,273 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-10) [7a800766] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:43:48,320 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-42) [440f1c40] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:48:48,366 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-70) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
2015-09-15 12:53:48,412 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-12) [50221cdc] HA=0A=
=0A=
reservation status for cluster Default is OK=0A=
=0A=
2015-09-15 12:58:48,459 INFO=0A=
[org.ovirt.engine.core.bll.scheduling.HaReservationHandling]=0A=
(DefaultQuartzScheduler_Worker-3) HA reservation=0A=
=0A=
status for cluster Default is OK=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
On 29.08.2015 22:48, Christian Hailer wrote:=0A=
=0A=
> Hello,=0A=
=0A=
>=0A=
=0A=
> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a=0A=
> single Centos=0A=
=0A=
=0A=
> 7 server, so I started by suspending the VMs in order to set the=0A=
> oVirt engine=0A=
=0A=
> host to maintenance mode. During the process of suspending the VMs=0A=
> the server=0A=
=0A=
> crashed, kernel panic=85=0A=
=0A=
>=0A=
=0A=
> After restarting the server I installed the updates via yum an=0A=
> restarted the=0A=
=0A=
> server again. Afterwards, all the VMs could be started again. Some=0A=
> hours later=0A=
=0A=
> my monitoring system registered some unresponsive hosts, I had a look=0A=
> in the=0A=
=0A=
> oVirt interface, 3 of the VMs were in the state =93not responding=94,=0A=
> marked by a=0A=
=0A=
> question mark.=0A=
=0A=
>>=0A=
> I tried to shut down the VMs, but oVirt wasn=92t able to do so. I tried=
=0A=
> to reset=0A=
=0A=
> the status in the database with the sql statement=0A=
=0A=
>=0A=
=0A=
> update vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid fro=
m=0A=
> vm_static=0A=
=0A=
=0A=
> where vm_name =3D 'MYVMNAME');=0A=
=0A=
>=0A=
=0A=
> but that didn=92t help, either. Only rebooting the whole hypervisor=0A=
> helped=85=0A=
=0A=
> afterwards everything worked again. But only for a few hours, then=0A=
> one of the=0A=
=0A=
> VMs entered the =93not responding=94 state again=85 again only a reboot =
helped.=0A=
=0A=
> Yesterday it happened again:=0A=
=0A=
>=0A=
=0A=
> 2015-08-28 17:44:22,664 INFO=0A=
=0A=
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]=0A=
=0A=
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC=0A=
=0A=
> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding=0A=
=0A=
>=0A=
=0A=
> 2015-08-28 17:44:22,692 WARN=0A=
=0A=
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector=0A=
> ]=0A=
=0A=
> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call=
Stack:=0A=
=0A=
=0A=
> null, Custom Event ID: -1, Message: VM DC is not responding.=0A=
=0A=
>=0A=
=0A=
> Does anybody know what I can do? Where should I have a look? Hints=0A=
> are greatly=0A=
=0A=
> appreciated!=0A=
=0A=
>=0A=
=0A=
> Thanks,=0A=
=0A=
>=0A=
=0A=
> Christian=0A=
=0A=
>=0A=
=0A=
=0A=
=0A=
=0A=
--=0A=
Daniel Helgenberger=0A=
m box bewegtbild GmbH=0A=
=0A=
P: +49/30/2408781-22=0A=
F: +49/30/2408781-10=0A=
=0A=
ACKERSTR. 19=0A=
D-10115 BERLIN=0A=
=0A=
=0A=
www.m-box.de www.monkeymen.tv=0A=
=0A=
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner=0A=
Handeslregister: Amtsgericht Charlottenburg / HRB 112767=0A=
------=_NextPartTM-000-4a9428dc-2b1c-4caf-8ee3-a447a009db9b
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-4a9428dc-2b1c-4caf-8ee3-a447a009db9b--
12:09 p.m.
New subject: Some VMs in status "not responding" in oVirt interface
On 17.09.2015 07:39, Christian Hailer wrote:
Hi,
just to get it straight: most of my VMs had one or more existing snapshots. Do you think
this is a problem currently? If I understand it correctly the BZ of Markus concerns only a
short period of time while removing a snapshot, but my VMs stopped responding in the
middle of the night without any interaction...
I deleted all the snapshots, just in case :) my system is running fine for nearly three
days now, I'm not quite sure but I think it helped that I changed the HDD and NIC of
the Windows 2012 VMs to VirtIO devices...
In my case these these Linux guests. So far I only had one VM with a live snap; the one
showing the issue. This points into the direction
qemu-rhev; might be independent from guest os.
For testing I created a new guest just for this purpose; it has a live snap shot. Maybe
you do the same?
To dig deeper into the issue: my storage is NFS3 - backed.
Best regards, Christian
-----Ursprüngliche Nachricht-----
Von: Daniel Helgenberger [mailto:daniel.helgenberger@m-box.de]
Gesendet: Dienstag, 15. September 2015 22:24
An: Markus Stockhausen <stockhausen(a)collogia.de>; Christian Hailer
<christian(a)hailer.eu>
Cc: ydary(a)redhat.com; users(a)ovirt.org
Betreff: Re: AW: [ovirt-users] Some VMs in status "not responding" in oVirt
interface
On 15.09.2015 21:31, Markus Stockhausen wrote:
> Hi Christian,
>
> I think of a package similar like this:
>
> qemu-debuginfo.x86_64 2:2.1.3-10.fc21
>
> That allows gdb to show information about backtrace symbols. See
> comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=1262251
> Makes error search much simpler - especially if qemu hangs.
Markus, thanks for the BZ. I think I do see the same issue. Actually my VM is currently
the only with a live snapshot and (puppetmaster) does a lot of I/O.
Christian, maybe this BZ1262251 also applicable?
I'll go ahead and delete the live snapshot. If I see this issue again I will submit
the trace to your BZ.
>
> Markus
>
> **********************************
>
> Von: Christian Hailer [christian(a)hailer.eu]
>
> Gesendet: Dienstag, 15. September 2015 21:24
>
> An: Markus Stockhausen; 'Daniel Helgenberger'
>
> Cc: ydary(a)redhat.com; users(a)ovirt.org
>
> Betreff: AW: [ovirt-users] Some VMs in status "not responding" in
> oVirt interface
>
>
>
>
>
>
>
>
>
>
> Hi Markus,
>
> gdb is available on CentOS 7, but what do you mean by qemu-debug? I Installed
qemu-kvm-tools, maybe this is the pendant for CentOS?
>
> qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools
> qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools
> qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools
>
> Regards, Christian
>
>
>
>
>
> Von: Markus Stockhausen [mailto:stockhausen@collogia.de]
>
>
> Gesendet: Dienstag, 15. September 2015 20:40
>
> An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>
>
> Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com;
> users(a)ovirt.org
>
> Betreff: Re: [ovirt-users] Some VMs in status "not responding" in
> oVirt interface
>
>
>
> Do you have a chance to install qemu-debug? If yes I would try a backtrace.
> gdb -p <qemu-pid>
>
> # bt
> Markus
>
>
> Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger
<daniel.helgenberger(a)m-box.de>:
>
>
>
>
>
> Hello,
>
>
>
> I do not want to hijack the thread but maybe my issue is related?
>
>
>
> It might have started with ovirt 3.5.3; but I cannot tell for sure.
>
>
>
> For me, one vm (foreman) is affected; the second time in 14 days. I
> can confirm this as I also loose any network connection to the VM and
>
> the ability to connect a console.
>
> Also, the only thing witch 'fixes' the issue is right now 'kill -9
<pid of qemu-kvm process>'
>
>
>
> As far as I can tell the VM became unresponsive at around Sep 15
> 12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs
> (see
>
> attached).
>
>
>
> Below the engine.log part.
>
>
>
> Versions:
>
> ovirt-engine-3.5.4.2-1.el7.centos.noarch
>
>
>
> vdsm-4.16.26-0.el7.centos
>
> libvirt-1.2.8-16.el7_1.3
>
>
>
> engine.log (1200 - 1300:
>
> 2015-09-15 12:03:47,949 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-56) [264d502a] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:08:02,708 INFO
> [org.ovirt.engine.core.bll.OvfDataUpdater]
> (DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update
>
> VMs/Templates Ovf.
>
> 2015-09-15 12:08:02,709 INFO
> [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
> (DefaultQuartzScheduler_Worker-89)
>
> [5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true.
Entities affected : ID:
>
> 00000002-0002-0002-0002-000000000088 Type: l
>
> 2015-09-15 12:08:02,780 INFO
> [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
> (DefaultQuartzScheduler_Worker-89)
>
> [5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks= key:
> 00000002-0002-0002-0002-000000000088 value: OVF_UPDATE
>
> 2015-09-15 12:08:47,997 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:13:06,998 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
> (org.ovirt.thread.pool-8-thread-48)
>
> [50221cdc] START, GetFileStatsVDSCommand( storagePoolId =
> 00000002-0002-0002-0002-000000000088, ignoreFailoverLimit = false),
> log id: 1503968
>
> 2015-09-15 12:13:07,137 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
> (org.ovirt.thread.pool-8-thread-48)
>
> [50221cdc] FINISH, GetFileStatsVDSCommand, return:
> {pfSense-2.0-RELEASE-i386.iso={status=0, ctime=1432286887.0,
> size=115709952},
>
> Fedora-15-i686-Live8
>
> 2015-09-15 12:13:07,178 INFO
> [org.ovirt.engine.core.bll.IsoDomainListSyncronizer]
> (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished
>
> automatic refresh process for ISO file type with success, for storage domain id
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.
>
> 2015-09-15 12:13:48,043 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:18:48,088 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-44) [6345e698] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:23:48,137 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-13) HA reservation
>
> status for cluster Default is OK
>
> 2015-09-15 12:28:48,183 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-76) [154c91d5] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:33:48,229 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:34:49,432 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM
>
> foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up -->
> NotResponding
>
> 2015-09-15 12:34:49,578 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-41)
>
> [5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM
foreman is not responding.
>
> 2015-09-15 12:38:48,273 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-10) [7a800766] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:43:48,320 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-42) [440f1c40] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:48:48,366 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-70) HA reservation
>
> status for cluster Default is OK
>
> 2015-09-15 12:53:48,412 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-12) [50221cdc] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:58:48,459 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-3) HA reservation
>
> status for cluster Default is OK
>
>
>
>
>
>
>
> On 29.08.2015 22:48, Christian Hailer wrote:
>
>> Hello,
>
>>
>
>> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a
>> single Centos
>
>
>> 7 server, so I started by suspending the VMs in order to set the
>> oVirt engine
>
>> host to maintenance mode. During the process of suspending the VMs
>> the server
>
>> crashed, kernel panic…
>
>>
>
>> After restarting the server I installed the updates via yum an
>> restarted the
>
>> server again. Afterwards, all the VMs could be started again. Some
>> hours later
>
>> my monitoring system registered some unresponsive hosts, I had a look
>> in the
>
>> oVirt interface, 3 of the VMs were in the state “not responding”,
>> marked by a
>
>> question mark.
>
>>>
>> I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried
>> to reset
>
>> the status in the database with the sql statement
>
>>
>
>> update vm_dynamic set status = 0 where vm_guid = (select vm_guid from
>> vm_static
>
>
>> where vm_name = 'MYVMNAME');
>
>>
>
>> but that didn’t help, either. Only rebooting the whole hypervisor
>> helped…
>
>> afterwards everything worked again. But only for a few hours, then
>> one of the
>
>> VMs entered the “not responding” state again… again only a reboot helped.
>
>> Yesterday it happened again:
>
>>
>
>> 2015-08-28 17:44:22,664 INFO
>
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>
>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC
>
>> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
>
>>
>
>> 2015-08-28 17:44:22,692 WARN
>
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
>> ]
>
>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack:
>
>
>> null, Custom Event ID: -1, Message: VM DC is not responding.
>
>>
>
>> Does anybody know what I can do? Where should I have a look? Hints
>> are greatly
>
>> appreciated!
>
>>
>
>> Thanks,
>
>>
>
>> Christian
>
>>
>
>
>
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
8:07 p.m.
New subject: Some VMs in status "not responding" in oVirt interface
Hello Christian,
just a quick round up:
Did you still see the issue? It stopped for me after removing live snap shots.
On 17.09.2015 07:39, Christian Hailer wrote:
Hi,
just to get it straight: most of my VMs had one or more existing snapshots. Do you think
this is a problem currently? If I understand it correctly the BZ of Markus concerns only a
short period of time while removing a snapshot, but my VMs stopped responding in the
middle of the night without any interaction...
I deleted all the snapshots, just in case :) my system is running fine for nearly three
days now, I'm not quite sure but I think it helped that I changed the HDD and NIC of
the Windows 2012 VMs to VirtIO devices...
Best regards, Christian
-----Ursprüngliche Nachricht-----
Von: Daniel Helgenberger [mailto:daniel.helgenberger@m-box.de]
Gesendet: Dienstag, 15. September 2015 22:24
An: Markus Stockhausen <stockhausen(a)collogia.de>; Christian Hailer
<christian(a)hailer.eu>
Cc: ydary(a)redhat.com; users(a)ovirt.org
Betreff: Re: AW: [ovirt-users] Some VMs in status "not responding" in oVirt
interface
On 15.09.2015 21:31, Markus Stockhausen wrote:
> Hi Christian,
>
> I think of a package similar like this:
>
> qemu-debuginfo.x86_64 2:2.1.3-10.fc21
>
> That allows gdb to show information about backtrace symbols. See
> comment 12 of https://bugzilla.redhat.com/show_bug.cgi?id=1262251
> Makes error search much simpler - especially if qemu hangs.
Markus, thanks for the BZ. I think I do see the same issue. Actually my VM is currently
the only with a live snapshot and (puppetmaster) does a lot of I/O.
Christian, maybe this BZ1262251 also applicable?
I'll go ahead and delete the live snapshot. If I see this issue again I will submit
the trace to your BZ.
>
> Markus
>
> **********************************
>
> Von: Christian Hailer [christian(a)hailer.eu]
>
> Gesendet: Dienstag, 15. September 2015 21:24
>
> An: Markus Stockhausen; 'Daniel Helgenberger'
>
> Cc: ydary(a)redhat.com; users(a)ovirt.org
>
> Betreff: AW: [ovirt-users] Some VMs in status "not responding" in
> oVirt interface
>
>
>
>
>
>
>
>
>
>
> Hi Markus,
>
> gdb is available on CentOS 7, but what do you mean by qemu-debug? I Installed
qemu-kvm-tools, maybe this is the pendant for CentOS?
>
> qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools
> qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools
> qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools
>
> Regards, Christian
>
>
>
>
>
> Von: Markus Stockhausen [mailto:stockhausen@collogia.de]
>
>
> Gesendet: Dienstag, 15. September 2015 20:40
>
> An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de>
>
> Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com;
> users(a)ovirt.org
>
> Betreff: Re: [ovirt-users] Some VMs in status "not responding" in
> oVirt interface
>
>
>
> Do you have a chance to install qemu-debug? If yes I would try a backtrace.
> gdb -p <qemu-pid>
>
> # bt
> Markus
>
>
> Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger
<daniel.helgenberger(a)m-box.de>:
>
>
>
>
>
> Hello,
>
>
>
> I do not want to hijack the thread but maybe my issue is related?
>
>
>
> It might have started with ovirt 3.5.3; but I cannot tell for sure.
>
>
>
> For me, one vm (foreman) is affected; the second time in 14 days. I
> can confirm this as I also loose any network connection to the VM and
>
> the ability to connect a console.
>
> Also, the only thing witch 'fixes' the issue is right now 'kill -9
<pid of qemu-kvm process>'
>
>
>
> As far as I can tell the VM became unresponsive at around Sep 15
> 12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs
> (see
>
> attached).
>
>
>
> Below the engine.log part.
>
>
>
> Versions:
>
> ovirt-engine-3.5.4.2-1.el7.centos.noarch
>
>
>
> vdsm-4.16.26-0.el7.centos
>
> libvirt-1.2.8-16.el7_1.3
>
>
>
> engine.log (1200 - 1300:
>
> 2015-09-15 12:03:47,949 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-56) [264d502a] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:08:02,708 INFO
> [org.ovirt.engine.core.bll.OvfDataUpdater]
> (DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update
>
> VMs/Templates Ovf.
>
> 2015-09-15 12:08:02,709 INFO
> [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
> (DefaultQuartzScheduler_Worker-89)
>
> [5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true.
Entities affected : ID:
>
> 00000002-0002-0002-0002-000000000088 Type: l
>
> 2015-09-15 12:08:02,780 INFO
> [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand]
> (DefaultQuartzScheduler_Worker-89)
>
> [5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks= key:
> 00000002-0002-0002-0002-000000000088 value: OVF_UPDATE
>
> 2015-09-15 12:08:47,997 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:13:06,998 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
> (org.ovirt.thread.pool-8-thread-48)
>
> [50221cdc] START, GetFileStatsVDSCommand( storagePoolId =
> 00000002-0002-0002-0002-000000000088, ignoreFailoverLimit = false),
> log id: 1503968
>
> 2015-09-15 12:13:07,137 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
> (org.ovirt.thread.pool-8-thread-48)
>
> [50221cdc] FINISH, GetFileStatsVDSCommand, return:
> {pfSense-2.0-RELEASE-i386.iso={status=0, ctime=1432286887.0,
> size=115709952},
>
> Fedora-15-i686-Live8
>
> 2015-09-15 12:13:07,178 INFO
> [org.ovirt.engine.core.bll.IsoDomainListSyncronizer]
> (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished
>
> automatic refresh process for ISO file type with success, for storage domain id
84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.
>
> 2015-09-15 12:13:48,043 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:18:48,088 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-44) [6345e698] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:23:48,137 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-13) HA reservation
>
> status for cluster Default is OK
>
> 2015-09-15 12:28:48,183 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-76) [154c91d5] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:33:48,229 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:34:49,432 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM
>
> foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up -->
> NotResponding
>
> 2015-09-15 12:34:49,578 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-41)
>
> [5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM
foreman is not responding.
>
> 2015-09-15 12:38:48,273 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-10) [7a800766] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:43:48,320 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-42) [440f1c40] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:48:48,366 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-70) HA reservation
>
> status for cluster Default is OK
>
> 2015-09-15 12:53:48,412 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-12) [50221cdc] HA
>
> reservation status for cluster Default is OK
>
> 2015-09-15 12:58:48,459 INFO
> [org.ovirt.engine.core.bll.scheduling.HaReservationHandling]
> (DefaultQuartzScheduler_Worker-3) HA reservation
>
> status for cluster Default is OK
>
>
>
>
>
>
>
> On 29.08.2015 22:48, Christian Hailer wrote:
>
>> Hello,
>
>>
>
>> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a
>> single Centos
>
>
>> 7 server, so I started by suspending the VMs in order to set the
>> oVirt engine
>
>> host to maintenance mode. During the process of suspending the VMs
>> the server
>
>> crashed, kernel panic…
>
>>
>
>> After restarting the server I installed the updates via yum an
>> restarted the
>
>> server again. Afterwards, all the VMs could be started again. Some
>> hours later
>
>> my monitoring system registered some unresponsive hosts, I had a look
>> in the
>
>> oVirt interface, 3 of the VMs were in the state “not responding”,
>> marked by a
>
>> question mark.
>
>>>
>> I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried
>> to reset
>
>> the status in the database with the sql statement
>
>>
>
>> update vm_dynamic set status = 0 where vm_guid = (select vm_guid from
>> vm_static
>
>
>> where vm_name = 'MYVMNAME');
>
>>
>
>> but that didn’t help, either. Only rebooting the whole hypervisor
>> helped…
>
>> afterwards everything worked again. But only for a few hours, then
>> one of the
>
>> VMs entered the “not responding” state again… again only a reboot helped.
>
>> Yesterday it happened again:
>
>>
>
>> 2015-08-28 17:44:22,664 INFO
>
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>
>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC
>
>> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
>
>>
>
>> 2015-08-28 17:44:22,692 WARN
>
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
>> ]
>
>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack:
>
>
>> null, Custom Event ID: -1, Message: VM DC is not responding.
>
>>
>
>> Does anybody know what I can do? Where should I have a look? Hints
>> are greatly
>
>> appreciated!
>
>>
>
>> Thanks,
>
>>
>
>> Christian
>
>>
>
>
>
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
3341
days inactive
3355
days old
7 comments
3 participants
participants (3)
-
Christian Hailer -
Daniel Helgenberger -
Markus Stockhausen