From alin.ilie at matrixrom.com Sun Jul 5 02:26:47 2015 Content-Type: multipart/mixed; boundary="===============6215325029312408833==" MIME-Version: 1.0 From: Alin Ilie To: users at ovirt.org Subject: [ovirt-users] Host failed to attach one of the Storage Domains attached to it Date: Sun, 05 Jul 2015 09:26:38 +0300 Message-ID: <5598CE1E.5070809@matrixrom.com> --===============6215325029312408833== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050503030109030902030106 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hello, We have a small oVirt deplopyment, with 2 INTEL CPUs and 1 AMD, therefor = we defined 2 clusters. One of the Intel machines is running the management as well (We called = this SOL). This deployment is connected to a NAS through NFS shares. Suddenly (no more than 2 days) SOL is reported as "Non operational" and = in the Action Items is saying " Host failed to attach one of the Storage Domains attached to it." All other machines seems to be fine. In the past we encountered similar errors after (accidental) reboot of = the hosts, but now it is not the case: [root(a)sol ~]# uptime 08:10:48 up 9 days, 15:48, 1 user, load average: 0.08, 0.05, 0.12 The "funny" is that the virtual machines already started on SOL are = still running without any problems. I am not sure what logs to check. In /var/log/vdsm/vdsm.log we have this error, but I do not think it is = relevant: Thread-38964::ERROR::2015-07-05 = 08:37:24,882::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) = Internal server error Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line = 501, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1330, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats =3D self._mom.getStatistics()['host'] File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in = getStatistics host_stats =3D self.threads['host_monitor'].interrogate().statistics[-= 1] AttributeError: 'NoneType' object has no attribute 'statistics' Thank you ! --------------050503030109030902030106 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hello,

We have a small oVirt deplopyment, with 2 INTEL CPUs and 1 AMD, therefor we defined 2 clusters.
One of the Intel machines is running the management as well (We called this SOL).
This deployment is connected to a NAS through NFS shares.

Suddenly (no more than 2 days) SOL is reported as "Non operational" and in the Action Items is saying "
Host failed to attach one of the Storage Domains attached to it."
All other machines seems to be fine.

In the past we encountered similar errors after (accidental) reboot of the hosts, but now it is not the case:
[root(a)sol ~]# uptime
=C2=A008:10:48 up 9 days, 15:48,=C2=A0 1 user,=C2=A0 load average: = 0.08, 0.05, 0.12

The "funny" is that the virtual machines already started on SOL are still running without any problems.

I am not sure what logs to check.

In /var/log/vdsm/vdsm.log we have this error, but I do not think it is relevant:
Thread-38964::ERROR::2015-07-05 08:37:24,882::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
=C2=A0 File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py= ", line 501, in _serveRequest
=C2=A0=C2=A0=C2=A0 res =3D method(**params)
=C2=A0 File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod
=C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs)
=C2=A0 File "/usr/share/vdsm/API.py", line 1330, in getStats
=C2=A0=C2=A0=C2=A0 stats.update(self._cif.mom.getKsmStats())
=C2=A0 File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
=C2=A0=C2=A0=C2=A0 stats =3D self._mom.getStatistics()['host']
=C2=A0 File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in getStatistics
=C2=A0=C2=A0=C2=A0 host_stats =3D self.threads['host_monitor'].interrogate().statistics[-1]
AttributeError: 'NoneType' object has no attribute 'statistics'

Thank you !
--------------050503030109030902030106-- --===============6215325029312408833== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTA1MDMwMzAxMDkwMzA5MDIwMzAxMDYKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhlbGxvLAoKV2UgaGF2ZSBhIHNtYWxsIG9WaXJ0IGRlcGxvcHltZW50LCB3aXRoIDIgSU5URUwg Q1BVcyBhbmQgMSBBTUQsIHRoZXJlZm9yIAp3ZSBkZWZpbmVkIDIgY2x1c3RlcnMuCk9uZSBvZiB0 aGUgSW50ZWwgbWFjaGluZXMgaXMgcnVubmluZyB0aGUgbWFuYWdlbWVudCBhcyB3ZWxsIChXZSBj YWxsZWQgCnRoaXMgU09MKS4KVGhpcyBkZXBsb3ltZW50IGlzIGNvbm5lY3RlZCB0byBhIE5BUyB0 aHJvdWdoIE5GUyBzaGFyZXMuCgpTdWRkZW5seSAobm8gbW9yZSB0aGFuIDIgZGF5cykgU09MIGlz IHJlcG9ydGVkIGFzICJOb24gb3BlcmF0aW9uYWwiIGFuZCAKaW4gdGhlIEFjdGlvbiBJdGVtcyBp cyBzYXlpbmcgIgpIb3N0IGZhaWxlZCB0byBhdHRhY2ggb25lIG9mIHRoZSBTdG9yYWdlIERvbWFp bnMgYXR0YWNoZWQgdG8gaXQuIgpBbGwgb3RoZXIgbWFjaGluZXMgc2VlbXMgdG8gYmUgZmluZS4K CkluIHRoZSBwYXN0IHdlIGVuY291bnRlcmVkIHNpbWlsYXIgZXJyb3JzIGFmdGVyIChhY2NpZGVu dGFsKSByZWJvb3Qgb2YgCnRoZSBob3N0cywgYnV0IG5vdyBpdCBpcyBub3QgdGhlIGNhc2U6Clty b290QHNvbCB+XSMgdXB0aW1lCiAgMDg6MTA6NDggdXAgOSBkYXlzLCAxNTo0OCwgIDEgdXNlciwg IGxvYWQgYXZlcmFnZTogMC4wOCwgMC4wNSwgMC4xMgoKVGhlICJmdW5ueSIgaXMgdGhhdCB0aGUg dmlydHVhbCBtYWNoaW5lcyBhbHJlYWR5IHN0YXJ0ZWQgb24gU09MIGFyZSAKc3RpbGwgcnVubmlu ZyB3aXRob3V0IGFueSBwcm9ibGVtcy4KCkkgYW0gbm90IHN1cmUgd2hhdCBsb2dzIHRvIGNoZWNr LgoKSW4gL3Zhci9sb2cvdmRzbS92ZHNtLmxvZyB3ZSBoYXZlIHRoaXMgZXJyb3IsIGJ1dCBJIGRv IG5vdCB0aGluayBpdCBpcyAKcmVsZXZhbnQ6ClRocmVhZC0zODk2NDo6RVJST1I6OjIwMTUtMDct MDUgCjA4OjM3OjI0LDg4Mjo6X19pbml0X186OjUwNjo6anNvbnJwYy5Kc29uUnBjU2VydmVyOjoo X3NlcnZlUmVxdWVzdCkgCkludGVybmFsIHNlcnZlciBlcnJvcgpUcmFjZWJhY2sgKG1vc3QgcmVj ZW50IGNhbGwgbGFzdCk6CiAgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNi9zaXRlLXBhY2thZ2Vz L3lhanNvbnJwYy9fX2luaXRfXy5weSIsIGxpbmUgCjUwMSwgaW4gX3NlcnZlUmVxdWVzdAogICAg IHJlcyA9IG1ldGhvZCgqKnBhcmFtcykKICAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL3JwYy9Ccmlk Z2UucHkiLCBsaW5lIDI3MSwgaW4gX2R5bmFtaWNNZXRob2QKICAgICByZXN1bHQgPSBmbigqbWV0 aG9kQXJncykKICAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL0FQSS5weSIsIGxpbmUgMTMzMCwgaW4g Z2V0U3RhdHMKICAgICBzdGF0cy51cGRhdGUoc2VsZi5fY2lmLm1vbS5nZXRLc21TdGF0cygpKQog ICBGaWxlICIvdXNyL3NoYXJlL3Zkc20vbW9tSUYucHkiLCBsaW5lIDYwLCBpbiBnZXRLc21TdGF0 cwogICAgIHN0YXRzID0gc2VsZi5fbW9tLmdldFN0YXRpc3RpY3MoKVsnaG9zdCddCiAgIEZpbGUg Ii91c3IvbGliL3B5dGhvbjIuNi9zaXRlLXBhY2thZ2VzL21vbS9NT01GdW5jcy5weSIsIGxpbmUg NzUsIGluIApnZXRTdGF0aXN0aWNzCiAgICAgaG9zdF9zdGF0cyA9IHNlbGYudGhyZWFkc1snaG9z dF9tb25pdG9yJ10uaW50ZXJyb2dhdGUoKS5zdGF0aXN0aWNzWy0xXQpBdHRyaWJ1dGVFcnJvcjog J05vbmVUeXBlJyBvYmplY3QgaGFzIG5vIGF0dHJpYnV0ZSAnc3RhdGlzdGljcycKClRoYW5rIHlv dSAhCgotLS0tLS0tLS0tLS0tLTA1MDUwMzAzMDEwOTAzMDkwMjAzMDEwNgpDb250ZW50LVR5cGU6 IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0 Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIgY29u dGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+CiAgPGJvZHkgYmdjb2xv cj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICBIZWxsbyw8YnI+CiAgICA8YnI+CiAgICBX ZSBoYXZlIGEgc21hbGwgb1ZpcnQgZGVwbG9weW1lbnQsIHdpdGggMiBJTlRFTCBDUFVzIGFuZCAx IEFNRCwKICAgIHRoZXJlZm9yIHdlIGRlZmluZWQgMiBjbHVzdGVycy48YnI+CiAgICBPbmUgb2Yg dGhlIEludGVsIG1hY2hpbmVzIGlzIHJ1bm5pbmcgdGhlIG1hbmFnZW1lbnQgYXMgd2VsbCAoV2UK ICAgIGNhbGxlZCB0aGlzIFNPTCkuPGJyPgogICAgVGhpcyBkZXBsb3ltZW50IGlzIGNvbm5lY3Rl ZCB0byBhIE5BUyB0aHJvdWdoIE5GUyBzaGFyZXMuPGJyPgogICAgPGJyPgogICAgU3VkZGVubHkg KG5vIG1vcmUgdGhhbiAyIGRheXMpIFNPTCBpcyByZXBvcnRlZCBhcyAiTm9uIG9wZXJhdGlvbmFs IgogICAgYW5kIGluIHRoZSBBY3Rpb24gSXRlbXMgaXMgc2F5aW5nICI8aW1nIGNsYXNzPSJnd3Qt SW1hZ2UiCiAgICAgIG9ubG9hZD0ndGhpcy5fX2d3dExhc3RVbmhhbmRsZWRFdmVudD0ibG9hZCI7 JwogICAgICBzcmM9ImNpZDpwYXJ0MS4wODA4MDYwNS4wNzA1MDEwNkBtYXRyaXhyb20uY29tIiBz dHlsZT0id2lkdGg6CiAgICAgIDE0cHg7IGhlaWdodDogMTJweDsgYmFja2dyb3VuZDogdHJhbnNw YXJlbnQKICAgICAgdXJsKCZxdW90O2RhdGE6aW1hZ2UvcG5nO2Jhc2U2NCxpVkJPUncwS0dnb0FB QUFOU1VoRVVnQUFBQTRBQUFBTUNBWUFBQUJTZ0l6YUFBQUFLa2xFUVZSNDJtTmd3QUwrbjBuNy8z K21NUmlmU1RQK3owQXNHTlZJQzQwZ3hjaVlnZVpnY1BvUkFKcm1kNG44R1paUEFBQUFBRWxGVGtT dVFtQ0MmcXVvdDspCgogICAgICBuby1yZXBlYXQgc2Nyb2xsIDBweCAwcHg7IGRpc3BsYXk6IGlu bGluZTsiIGJvcmRlcj0iMCI+CiAgICA8ZGl2PgogICAgICA8ZGl2IHN0eWxlPSJkaXNwbGF5OiBp bmxpbmU7IiBjbGFzcz0iZ3d0LUxhYmVsIj5Ib3N0IGZhaWxlZCB0bwogICAgICAgIGF0dGFjaCBv bmUgb2YgdGhlIFN0b3JhZ2UgRG9tYWlucyBhdHRhY2hlZCB0byBpdC4iPGJyPgogICAgICAgIEFs bCBvdGhlciBtYWNoaW5lcyBzZWVtcyB0byBiZSBmaW5lLjxicj4KICAgICAgICA8YnI+CiAgICAg ICAgSW4gdGhlIHBhc3Qgd2UgZW5jb3VudGVyZWQgc2ltaWxhciBlcnJvcnMgYWZ0ZXIgKGFjY2lk ZW50YWwpCiAgICAgICAgcmVib290IG9mIHRoZSBob3N0cywgYnV0IG5vdyBpdCBpcyBub3QgdGhl IGNhc2U6PGJyPgogICAgICAgIFtyb290QHNvbCB+XSMgdXB0aW1lPGJyPgogICAgICAgIMKgMDg6 MTA6NDggdXAgOSBkYXlzLCAxNTo0OCzCoCAxIHVzZXIswqAgbG9hZCBhdmVyYWdlOiAwLjA4LCAw LjA1LAogICAgICAgIDAuMTI8YnI+CiAgICAgICAgPGJyPgogICAgICAgIFRoZSAiZnVubnkiIGlz IHRoYXQgdGhlIHZpcnR1YWwgbWFjaGluZXMgYWxyZWFkeSBzdGFydGVkIG9uIFNPTAogICAgICAg IGFyZSBzdGlsbCBydW5uaW5nIHdpdGhvdXQgYW55IHByb2JsZW1zLjxicj4KICAgICAgICA8YnI+ CiAgICAgICAgSSBhbSBub3Qgc3VyZSB3aGF0IGxvZ3MgdG8gY2hlY2suPGJyPgogICAgICAgIDxi cj4KICAgICAgICBJbiAvdmFyL2xvZy92ZHNtL3Zkc20ubG9nIHdlIGhhdmUgdGhpcyBlcnJvciwg YnV0IEkgZG8gbm90IHRoaW5rCiAgICAgICAgaXQgaXMgcmVsZXZhbnQ6PGJyPgogICAgICAgIFRo cmVhZC0zODk2NDo6RVJST1I6OjIwMTUtMDctMDUKICAgICAgICAwODozNzoyNCw4ODI6Ol9faW5p dF9fOjo1MDY6Ompzb25ycGMuSnNvblJwY1NlcnZlcjo6KF9zZXJ2ZVJlcXVlc3QpCiAgICAgICAg SW50ZXJuYWwgc2VydmVyIGVycm9yPGJyPgogICAgICAgIFRyYWNlYmFjayAobW9zdCByZWNlbnQg Y2FsbCBsYXN0KTo8YnI+CiAgICAgICAgwqAgRmlsZSAiL3Vzci9saWIvcHl0aG9uMi42L3NpdGUt cGFja2FnZXMveWFqc29ucnBjL19faW5pdF9fLnB5IiwKICAgICAgICBsaW5lIDUwMSwgaW4gX3Nl cnZlUmVxdWVzdDxicj4KICAgICAgICDCoMKgwqAgcmVzID0gbWV0aG9kKCoqcGFyYW1zKTxicj4K ICAgICAgICDCoCBGaWxlICIvdXNyL3NoYXJlL3Zkc20vcnBjL0JyaWRnZS5weSIsIGxpbmUgMjcx LCBpbgogICAgICAgIF9keW5hbWljTWV0aG9kPGJyPgogICAgICAgIMKgwqDCoCByZXN1bHQgPSBm bigqbWV0aG9kQXJncyk8YnI+CiAgICAgICAgwqAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL0FQSS5w eSIsIGxpbmUgMTMzMCwgaW4gZ2V0U3RhdHM8YnI+CiAgICAgICAgwqDCoMKgIHN0YXRzLnVwZGF0 ZShzZWxmLl9jaWYubW9tLmdldEtzbVN0YXRzKCkpPGJyPgogICAgICAgIMKgIEZpbGUgIi91c3Iv c2hhcmUvdmRzbS9tb21JRi5weSIsIGxpbmUgNjAsIGluIGdldEtzbVN0YXRzPGJyPgogICAgICAg IMKgwqDCoCBzdGF0cyA9IHNlbGYuX21vbS5nZXRTdGF0aXN0aWNzKClbJ2hvc3QnXTxicj4KICAg ICAgICDCoCBGaWxlICIvdXNyL2xpYi9weXRob24yLjYvc2l0ZS1wYWNrYWdlcy9tb20vTU9NRnVu Y3MucHkiLCBsaW5lCiAgICAgICAgNzUsIGluIGdldFN0YXRpc3RpY3M8YnI+CiAgICAgICAgwqDC oMKgIGhvc3Rfc3RhdHMgPQogICAgICAgIHNlbGYudGhyZWFkc1snaG9zdF9tb25pdG9yJ10uaW50 ZXJyb2dhdGUoKS5zdGF0aXN0aWNzWy0xXTxicj4KICAgICAgICBBdHRyaWJ1dGVFcnJvcjogJ05v bmVUeXBlJyBvYmplY3QgaGFzIG5vIGF0dHJpYnV0ZSAnc3RhdGlzdGljcyc8YnI+CiAgICAgICAg PGJyPgogICAgICAgIFRoYW5rIHlvdSAhPGJyPgogICAgICA8L2Rpdj4KICAgIDwvZGl2PgogIDwv Ym9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS0wNTA1MDMwMzAxMDkwMzA5MDIwMzAxMDYtLQo= --===============6215325029312408833==--