From alin.ilie at matrixrom.com Sun Jul 5 01:40:42 2015 Content-Type: multipart/mixed; boundary="===============6873589381102546381==" MIME-Version: 1.0 From: Alin Ilie To: users at ovirt.org Subject: [ovirt-users] Host failed to attach one of the Storage Domains attached to it Date: Sun, 05 Jul 2015 08:40:39 +0300 Message-ID: <5598C357.3030704@matrixrom.com> --===============6873589381102546381== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------060503080301050806000704 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hello, We have a small oVirt deplopyment, with 2 INTEL CPUs and 1 AMD, therefor = we defined 2 clusters. One of the Intel machines is running the management as well (We called = this SOL). This deployment is connected to a NAS through NFS shares. Suddenly (no more than 2 days) SOL is reported as "Non operational" and = in the Action Items is saying " Host failed to attach one of the Storage Domains attached to it." All other machines seems to be fine. In the past we encountered similar errors after (accidental) reboot of = the hosts, but now it is not the case: [root(a)sol ~]# uptime 08:10:48 up 9 days, 15:48, 1 user, load average: 0.08, 0.05, 0.12 The "funny" is that the virtual machines already started on SOL are = still running without any problems. I am not sure what logs to check. In /var/log/vdsm/vdsm.log we have this error, but I do not think it is = relevant: Thread-38964::ERROR::2015-07-05 = 08:37:24,882::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) = Internal server error Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line = 501, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1330, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats =3D self._mom.getStatistics()['host'] File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in = getStatistics host_stats =3D self.threads['host_monitor'].interrogate().statistics[-= 1] AttributeError: 'NoneType' object has no attribute 'statistics' Thank you ! --------------060503080301050806000704 Content-Type: multipart/related; boundary=3D"------------000103070605000308060409" --------------000103070605000308060409 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hello,

We have a small oVirt deplopyment, with 2 INTEL CPUs and 1 AMD, therefor we defined 2 clusters.
One of the Intel machines is running the management as well (We called this SOL).
This deployment is connected to a NAS through NFS shares.

Suddenly (no more than 2 days) SOL is reported as "Non operational" and in the Action Items is saying "
Host failed to attach one of the Storage Domains attached to it."
All other machines seems to be fine.

In the past we encountered similar errors after (accidental) reboot of the hosts, but now it is not the case:
[root(a)sol ~]# uptime
=C2=A008:10:48 up 9 days, 15:48,=C2=A0 1 user,=C2=A0 load average: = 0.08, 0.05, 0.12

The "funny" is that the virtual machines already started on SOL are still running without any problems.

I am not sure what logs to check.

In /var/log/vdsm/vdsm.log we have this error, but I do not think it is relevant:
Thread-38964::ERROR::2015-07-05 08:37:24,882::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
=C2=A0 File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py= ", line 501, in _serveRequest
=C2=A0=C2=A0=C2=A0 res =3D method(**params)
=C2=A0 File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod
=C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs)
=C2=A0 File "/usr/share/vdsm/API.py", line 1330, in getStats
=C2=A0=C2=A0=C2=A0 stats.update(self._cif.mom.getKsmStats())
=C2=A0 File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
=C2=A0=C2=A0=C2=A0 stats =3D self._mom.getStatistics()['host']
=C2=A0 File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in getStatistics
=C2=A0=C2=A0=C2=A0 host_stats =3D self.threads['host_monitor'].interrogate().statistics[-1]
AttributeError: 'NoneType' object has no attribute 'statistics'

Thank you !
--------------000103070605000308060409-- --------------060503080301050806000704-- --===============6873589381102546381== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNjA1MDMwODAzMDEwNTA4MDYwMDA3MDQKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhlbGxvLAoKV2UgaGF2ZSBhIHNtYWxsIG9WaXJ0IGRlcGxvcHltZW50LCB3aXRoIDIgSU5URUwg Q1BVcyBhbmQgMSBBTUQsIHRoZXJlZm9yIAp3ZSBkZWZpbmVkIDIgY2x1c3RlcnMuCk9uZSBvZiB0 aGUgSW50ZWwgbWFjaGluZXMgaXMgcnVubmluZyB0aGUgbWFuYWdlbWVudCBhcyB3ZWxsIChXZSBj YWxsZWQgCnRoaXMgU09MKS4KVGhpcyBkZXBsb3ltZW50IGlzIGNvbm5lY3RlZCB0byBhIE5BUyB0 aHJvdWdoIE5GUyBzaGFyZXMuCgpTdWRkZW5seSAobm8gbW9yZSB0aGFuIDIgZGF5cykgU09MIGlz IHJlcG9ydGVkIGFzICJOb24gb3BlcmF0aW9uYWwiIGFuZCAKaW4gdGhlIEFjdGlvbiBJdGVtcyBp cyBzYXlpbmcgIgpIb3N0IGZhaWxlZCB0byBhdHRhY2ggb25lIG9mIHRoZSBTdG9yYWdlIERvbWFp bnMgYXR0YWNoZWQgdG8gaXQuIgpBbGwgb3RoZXIgbWFjaGluZXMgc2VlbXMgdG8gYmUgZmluZS4K CkluIHRoZSBwYXN0IHdlIGVuY291bnRlcmVkIHNpbWlsYXIgZXJyb3JzIGFmdGVyIChhY2NpZGVu dGFsKSByZWJvb3Qgb2YgCnRoZSBob3N0cywgYnV0IG5vdyBpdCBpcyBub3QgdGhlIGNhc2U6Clty b290QHNvbCB+XSMgdXB0aW1lCiAgMDg6MTA6NDggdXAgOSBkYXlzLCAxNTo0OCwgIDEgdXNlciwg IGxvYWQgYXZlcmFnZTogMC4wOCwgMC4wNSwgMC4xMgoKVGhlICJmdW5ueSIgaXMgdGhhdCB0aGUg dmlydHVhbCBtYWNoaW5lcyBhbHJlYWR5IHN0YXJ0ZWQgb24gU09MIGFyZSAKc3RpbGwgcnVubmlu ZyB3aXRob3V0IGFueSBwcm9ibGVtcy4KCkkgYW0gbm90IHN1cmUgd2hhdCBsb2dzIHRvIGNoZWNr LgoKSW4gL3Zhci9sb2cvdmRzbS92ZHNtLmxvZyB3ZSBoYXZlIHRoaXMgZXJyb3IsIGJ1dCBJIGRv IG5vdCB0aGluayBpdCBpcyAKcmVsZXZhbnQ6ClRocmVhZC0zODk2NDo6RVJST1I6OjIwMTUtMDct MDUgCjA4OjM3OjI0LDg4Mjo6X19pbml0X186OjUwNjo6anNvbnJwYy5Kc29uUnBjU2VydmVyOjoo X3NlcnZlUmVxdWVzdCkgCkludGVybmFsIHNlcnZlciBlcnJvcgpUcmFjZWJhY2sgKG1vc3QgcmVj ZW50IGNhbGwgbGFzdCk6CiAgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNi9zaXRlLXBhY2thZ2Vz L3lhanNvbnJwYy9fX2luaXRfXy5weSIsIGxpbmUgCjUwMSwgaW4gX3NlcnZlUmVxdWVzdAogICAg IHJlcyA9IG1ldGhvZCgqKnBhcmFtcykKICAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL3JwYy9Ccmlk Z2UucHkiLCBsaW5lIDI3MSwgaW4gX2R5bmFtaWNNZXRob2QKICAgICByZXN1bHQgPSBmbigqbWV0 aG9kQXJncykKICAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL0FQSS5weSIsIGxpbmUgMTMzMCwgaW4g Z2V0U3RhdHMKICAgICBzdGF0cy51cGRhdGUoc2VsZi5fY2lmLm1vbS5nZXRLc21TdGF0cygpKQog ICBGaWxlICIvdXNyL3NoYXJlL3Zkc20vbW9tSUYucHkiLCBsaW5lIDYwLCBpbiBnZXRLc21TdGF0 cwogICAgIHN0YXRzID0gc2VsZi5fbW9tLmdldFN0YXRpc3RpY3MoKVsnaG9zdCddCiAgIEZpbGUg Ii91c3IvbGliL3B5dGhvbjIuNi9zaXRlLXBhY2thZ2VzL21vbS9NT01GdW5jcy5weSIsIGxpbmUg NzUsIGluIApnZXRTdGF0aXN0aWNzCiAgICAgaG9zdF9zdGF0cyA9IHNlbGYudGhyZWFkc1snaG9z dF9tb25pdG9yJ10uaW50ZXJyb2dhdGUoKS5zdGF0aXN0aWNzWy0xXQpBdHRyaWJ1dGVFcnJvcjog J05vbmVUeXBlJyBvYmplY3QgaGFzIG5vIGF0dHJpYnV0ZSAnc3RhdGlzdGljcycKClRoYW5rIHlv dSAhCgotLS0tLS0tLS0tLS0tLTA2MDUwMzA4MDMwMTA1MDgwNjAwMDcwNApDb250ZW50LVR5cGU6 IG11bHRpcGFydC9yZWxhdGVkOwogYm91bmRhcnk9Ii0tLS0tLS0tLS0tLTAwMDEwMzA3MDYwNTAw MDMwODA2MDQwOSIKCgotLS0tLS0tLS0tLS0tLTAwMDEwMzA3MDYwNTAwMDMwODA2MDQwOQpDb250 ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29k aW5nOiA4Yml0Cgo8aHRtbD4KICA8aGVhZD4KCiAgICA8bWV0YSBodHRwLWVxdWl2PSJjb250ZW50 LXR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxi b2R5IGJnY29sb3I9IiNGRkZGRkYiIHRleHQ9IiMwMDAwMDAiPgogICAgSGVsbG8sPGJyPgogICAg PGJyPgogICAgV2UgaGF2ZSBhIHNtYWxsIG9WaXJ0IGRlcGxvcHltZW50LCB3aXRoIDIgSU5URUwg Q1BVcyBhbmQgMSBBTUQsCiAgICB0aGVyZWZvciB3ZSBkZWZpbmVkIDIgY2x1c3RlcnMuPGJyPgog ICAgT25lIG9mIHRoZSBJbnRlbCBtYWNoaW5lcyBpcyBydW5uaW5nIHRoZSBtYW5hZ2VtZW50IGFz IHdlbGwgKFdlCiAgICBjYWxsZWQgdGhpcyBTT0wpLjxicj4KICAgIFRoaXMgZGVwbG95bWVudCBp cyBjb25uZWN0ZWQgdG8gYSBOQVMgdGhyb3VnaCBORlMgc2hhcmVzLjxicj4KICAgIDxicj4KICAg IFN1ZGRlbmx5IChubyBtb3JlIHRoYW4gMiBkYXlzKSBTT0wgaXMgcmVwb3J0ZWQgYXMgIk5vbiBv cGVyYXRpb25hbCIKICAgIGFuZCBpbiB0aGUgQWN0aW9uIEl0ZW1zIGlzIHNheWluZyAiPGltZyBj bGFzcz0iZ3d0LUltYWdlIgogICAgICBvbmxvYWQ9J3RoaXMuX19nd3RMYXN0VW5oYW5kbGVkRXZl bnQ9ImxvYWQiOycKICAgICAgc3JjPSJjaWQ6cGFydDEuMDgwODA2MDUuMDcwNTAxMDZAbWF0cml4 cm9tLmNvbSIgc3R5bGU9IndpZHRoOgogICAgICAxNHB4OyBoZWlnaHQ6IDEycHg7IGJhY2tncm91 bmQ6IHRyYW5zcGFyZW50CiAgICAgIHVybCgmcXVvdDtkYXRhOmltYWdlL3BuZztiYXNlNjQsaVZC T1J3MEtHZ29BQUFBTlNVaEVVZ0FBQUE0QUFBQU1DQVlBQUFCU2dJemFBQUFBS2tsRVFWUjQybU5n d0FMK24wbjcvMyttTVJpZlNUUCt6MEFzR05WSUM0MGd4Y2lZZ2VaZ2NQb1JBSnJtZDRuOEdaWlBB QUFBQUVsRlRrU3VRbUNDJnF1b3Q7KQogICAgICBuby1yZXBlYXQgc2Nyb2xsIDBweCAwcHg7IGRp c3BsYXk6IGlubGluZTsiIGJvcmRlcj0iMCI+CiAgICA8ZGl2PgogICAgICA8ZGl2IHN0eWxlPSJk aXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iZ3d0LUxhYmVsIj5Ib3N0IGZhaWxlZCB0bwogICAgICAg IGF0dGFjaCBvbmUgb2YgdGhlIFN0b3JhZ2UgRG9tYWlucyBhdHRhY2hlZCB0byBpdC4iPGJyPgog ICAgICAgIEFsbCBvdGhlciBtYWNoaW5lcyBzZWVtcyB0byBiZSBmaW5lLjxicj4KICAgICAgICA8 YnI+CiAgICAgICAgSW4gdGhlIHBhc3Qgd2UgZW5jb3VudGVyZWQgc2ltaWxhciBlcnJvcnMgYWZ0 ZXIgKGFjY2lkZW50YWwpCiAgICAgICAgcmVib290IG9mIHRoZSBob3N0cywgYnV0IG5vdyBpdCBp cyBub3QgdGhlIGNhc2U6PGJyPgogICAgICAgIFtyb290QHNvbCB+XSMgdXB0aW1lPGJyPgogICAg ICAgIMKgMDg6MTA6NDggdXAgOSBkYXlzLCAxNTo0OCzCoCAxIHVzZXIswqAgbG9hZCBhdmVyYWdl OiAwLjA4LCAwLjA1LAogICAgICAgIDAuMTI8YnI+CiAgICAgICAgPGJyPgogICAgICAgIFRoZSAi ZnVubnkiIGlzIHRoYXQgdGhlIHZpcnR1YWwgbWFjaGluZXMgYWxyZWFkeSBzdGFydGVkIG9uIFNP TAogICAgICAgIGFyZSBzdGlsbCBydW5uaW5nIHdpdGhvdXQgYW55IHByb2JsZW1zLjxicj4KICAg ICAgICA8YnI+CiAgICAgICAgSSBhbSBub3Qgc3VyZSB3aGF0IGxvZ3MgdG8gY2hlY2suPGJyPgog ICAgICAgIDxicj4KICAgICAgICBJbiAvdmFyL2xvZy92ZHNtL3Zkc20ubG9nIHdlIGhhdmUgdGhp cyBlcnJvciwgYnV0IEkgZG8gbm90IHRoaW5rCiAgICAgICAgaXQgaXMgcmVsZXZhbnQ6PGJyPgog ICAgICAgIFRocmVhZC0zODk2NDo6RVJST1I6OjIwMTUtMDctMDUKICAgICAgICAwODozNzoyNCw4 ODI6Ol9faW5pdF9fOjo1MDY6Ompzb25ycGMuSnNvblJwY1NlcnZlcjo6KF9zZXJ2ZVJlcXVlc3Qp CiAgICAgICAgSW50ZXJuYWwgc2VydmVyIGVycm9yPGJyPgogICAgICAgIFRyYWNlYmFjayAobW9z dCByZWNlbnQgY2FsbCBsYXN0KTo8YnI+CiAgICAgICAgwqAgRmlsZSAiL3Vzci9saWIvcHl0aG9u Mi42L3NpdGUtcGFja2FnZXMveWFqc29ucnBjL19faW5pdF9fLnB5IiwKICAgICAgICBsaW5lIDUw MSwgaW4gX3NlcnZlUmVxdWVzdDxicj4KICAgICAgICDCoMKgwqAgcmVzID0gbWV0aG9kKCoqcGFy YW1zKTxicj4KICAgICAgICDCoCBGaWxlICIvdXNyL3NoYXJlL3Zkc20vcnBjL0JyaWRnZS5weSIs IGxpbmUgMjcxLCBpbgogICAgICAgIF9keW5hbWljTWV0aG9kPGJyPgogICAgICAgIMKgwqDCoCBy ZXN1bHQgPSBmbigqbWV0aG9kQXJncyk8YnI+CiAgICAgICAgwqAgRmlsZSAiL3Vzci9zaGFyZS92 ZHNtL0FQSS5weSIsIGxpbmUgMTMzMCwgaW4gZ2V0U3RhdHM8YnI+CiAgICAgICAgwqDCoMKgIHN0 YXRzLnVwZGF0ZShzZWxmLl9jaWYubW9tLmdldEtzbVN0YXRzKCkpPGJyPgogICAgICAgIMKgIEZp bGUgIi91c3Ivc2hhcmUvdmRzbS9tb21JRi5weSIsIGxpbmUgNjAsIGluIGdldEtzbVN0YXRzPGJy PgogICAgICAgIMKgwqDCoCBzdGF0cyA9IHNlbGYuX21vbS5nZXRTdGF0aXN0aWNzKClbJ2hvc3Qn XTxicj4KICAgICAgICDCoCBGaWxlICIvdXNyL2xpYi9weXRob24yLjYvc2l0ZS1wYWNrYWdlcy9t b20vTU9NRnVuY3MucHkiLCBsaW5lCiAgICAgICAgNzUsIGluIGdldFN0YXRpc3RpY3M8YnI+CiAg ICAgICAgwqDCoMKgIGhvc3Rfc3RhdHMgPQogICAgICAgIHNlbGYudGhyZWFkc1snaG9zdF9tb25p dG9yJ10uaW50ZXJyb2dhdGUoKS5zdGF0aXN0aWNzWy0xXTxicj4KICAgICAgICBBdHRyaWJ1dGVF cnJvcjogJ05vbmVUeXBlJyBvYmplY3QgaGFzIG5vIGF0dHJpYnV0ZSAnc3RhdGlzdGljcyc8YnI+ CiAgICAgICAgPGJyPgogICAgICAgIFRoYW5rIHlvdSAhPGJyPgogICAgICA8L2Rpdj4KICAgIDwv ZGl2PgogIDwvYm9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS0wMDAxMDMwNzA2MDUwMDAzMDgw NjA0MDktLQoKLS0tLS0tLS0tLS0tLS0wNjA1MDMwODAzMDEwNTA4MDYwMDA3MDQtLQo= --===============6873589381102546381==--