8:45 a.m.
--Sig_/+j1R2P+c=Zv.QGARGGJyPq/
Content-Type: multipart/mixed; boundary="MP_/_7Zm8Xi//waNSwzp1NQwbMu"
--MP_/_7Zm8Xi//waNSwzp1NQwbMu
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Hi,
I have oVirt 3.5.4 on CentOS 7.1 hosts, and everyone once in a while one of
my hosts starts sending me the 4 engine status messages above about every
10-15 minutes.
It looks like the ha broker on the host currently running is having issues
(already tried restarting it once.
I've attached a tarball with log snippets for the engine, host with active
engine vm running, and the complaining host...
(I'll be hanging out in #ovirt all day too, will respond to questions or
suggestions here or there.)
Robert
--=20
Senior Software Engineer @ Parsons
--MP_/_7Zm8Xi//waNSwzp1NQwbMu
Content-Type: application/gzip
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=ovirt-logs.tar.gz
H4sICDt9L1YAA292aXJ0LWxvZ3MudGFyAO2dW3MjN5Lv9exPUW/qjihSuAPFiH7w2O51x3F79ri9
nocZh6JIliSGSZYOL/L2bsx3PwCKVPMGoCSRmZJbFQ63eP9V4pL/xCVR341mi864vp5fnJ3qIvbS
Urp/meZi89/1dUaZolxpxQU5I/aB0meZPBnRxrWcL8pZlp3N5ot69jn8vtTrL/Sqv5T/oJ7cjsvR
dDS97tzU88XRKsSDyl9TW/6KCvVa/hBXrPz7s/qPata1Lz7tN1wBKyEOlz+1L2l+X/7U1gXCiP37
LCPHucX49ZWX/683s6ocdlSv9+Hn93/v9azpZYeSDtMZKXqU9yTPRVH0etX0ejStLm+qcry46fUU
33mq+93t8qe6HP5c/+Cf7vXelIPFqJ6+zX77mE3rRVZPs8XNaJ65uvXN6ncZMVpwHf51mRtpf308
mi+qaTXr2SdFr+dr7aX7omp4ucYou+sKu3pz97t6Oq08xI/ldDh2H38zrxbL27fZl5eyytaAvv3M
TTV8CJamW1hGPQnrxv+1xTUY1/NdJBNHYkiWSmBJBEsVcSSFZKkEloG3lCBxJKTWl8AyCK1P0DgS
UutLYQkES7E4kkSyVByrgK1TIgZTMOtgJ9Xk8mpWWV8qNx51P1aT9/7Zew87WT2T8cKY9ffLwPeL
HhG5Zta7DW6Xl2Prsy+n9erm7G27n9p/odt49R9X7v/N8nZYLqpLa+jF5dXI3WfEyz+ShNE2JINy
PFiOHYx739vs02dbKJPMPcgW9aIcvyNdqinLs+bz9qHTnbmlnXbun3Lv2K4rPAxti0capCocx8IQ
JSJUkVdIWN1ipNY5LIGElbAWglZKWUojIIUCkhUSVutLYBUIlgpFKQ2SIUiWSmBh9FOhKGWFhNVP
JbAw5FsoSrFIRc4UVkgQx8IInmQoSmmQsFpfCoshWCoUpRxEgrNUAguh9clQlLJCQgqeUlgKwVIR
MeyQNJKlElgIKkFGVKdFKrD6qQQWsEoIwlCZG2ZtdDuaXttgnDV/df/TP7yPwj8tB4NqPr9ajsef
M/eGaphR0tW0a//PJN++9bDopionEqtE4lhu2B+87oZFt0dCCuRSWBj9YVh0eySs/jCBhdEfhkX3
ASQ4S8WxMEZ3ZFh0H0CCs1QCC6GfUmHR7ZGQ+qkUFsKAkwqLbiZzQpD6qThWIQiCpcKaxCNh1akE
FoLvU2HR7ZGw6lQcSyFE4iosuj0SR7JUAgthdEeFxbBHQlIJCSyNMGahwmKY7c9qwFkqjoUxCq3C
YpipnBMsS8WxKIJGV2Ex7JAYUoSswsrTYXEslxy3lsCo6nFLSQSNrsNi2CEppGgmjiUIQo+uI2LY
ISH1UyksBOWpI2LYISEpzwQWha1TEXFnYdzCrEctdtLbtxwW21zmEmvVQAKrQIgBdLg8LJIiSLNx
KSyM1h0W2weQ4CwVx6IIYxU6LLYPIMFZKoGF4VvDYtsjYfnWBBZCDKDDqtYhYcUACSwN209FO01N
Hutb1fYth8W8oLZyII0aJbAYgscwYTHvkZD0YAoLoXWbsJh3SBypdaewEEJpExbzHglJr6WwECJ8
ExbbHglpeCaFhTALZ8J+wyMhKdsUFsI8gAmLbWE9rEDyfQksjPE1ExbbHgmr9SWwEHY4mLDY9khI
YxUJLIUQV5qw2D6ABGepBBbCqI6JiOF9JDhLJbAQIvAiIob3kcAslcJ6LnGlgynUI+NKuf7+iCRS
OXHDDZPryeKyPxsNr91PbD/R/Wj//tvqtfsfel8vp8OseUfmjeM+kv05Wtxkt/VsMV//eNgjSJ4r
J2xPmPGiCEcYbiMylpxPYGEs6ynCEYZHQnKTCSyJ4CaLcHM6gARnqQQWgpsswh3bASQ4SyWwMNxk
OMLwSEgD1SksBDlfRPzJPhKcpRJYsIN+8QpePDbjxfZAdRGOYKTOmcLyGHEsjE32RTiC8UhIyxsT
WBiL9opwBHMACc5SCSz4ASBJwhGMR8IJrJJY8MpWkojY1mibV5NY8MOvkkTEtt7bTwtnqThWARsD
hCSRdDmWjP0LPdtTQ+KWgZw625OUIpHtyb5juyxDUYqFVjkhWFUsjsXgQwJJQiJuhYQza5TEwui2
Ig1hHwnOUnEsDj90IUkoSlkh4cirJBaGvApFESskLHmVwIKf3ZYkFEWskLD6qQQWfBgoSSiKkNYp
54Zhtb44Fkbro6EoYoWE1PpoSLKvsJAiiZS1EFxyylLwa+AkDUURDZLAGbZPYsEP20saFsMHkOAs
lcBC0Og0LIY9Es6wfRILeBAmDOO2BqtjZnuSNCy6qcopxfIccSyM+JKGRbdHQtJtKSwMZxYW3QeQ
4CwVx8KIL2lYdHskJIWbwsLwsRHR7ZBwVromsRD6KRZRtw4JyccmsARsJB5x+BbG7aJ62hEHzS2H
RT1TOcPyrQksDN/KwprHI2HV2QQWwpgIC4t6j4TkW1NYCPErC7dxh4S0hyuJheBbWVhseyQk35rC
wuinwmLbI2H1Uwks2H4qXsFdPqfH+NaCbt9yWMwzkwukhcqShZWzwxJIQWIKC6N7jhcgClJYzDsk
pIXKSSyE+QYeFvMeCUnZprAQ9BqPiG2HhLR+I4GFkE1d8ojYdkhYdSqBhTC+xsNim6ucG6QYII5V
SISljDysRTwS0gRkCgvB9/Gw2HZISPkHk1gI0RIPi22PhBQtpbAQVroEz6k+iARnqQQWhkoI6/MD
SHCWimMBZ+iPd5puy8Gj4kq2fcthsS0E2gkAKSyG0LqD52CvkJBC3RQWgm8NnoN9EAnOUnEshGxS
MngO9goJaVQnhYUQLQWPW14hIUVLKSyEGCB4rvEKCalHT2EhxJXB45alz3uBlB8whYWxMjl43HKD
hLU2MoWF4fvCYlvgnSaYxEKIK4PHLR9EgrNUAgthFi543PJBJDhLJbAQfF/wuOUVElKPnsKC9X0R
R2xhzGPjyvtsUuFOWfLcuMzy6xs6UkKns7/C5Uu8M66v5xeDenI7LkfT0fS6427woryupouufe2J
v+E24Coh3L9M861/7UWFouqMMkW50srG2GeEahuKn2XkKHeYuJa28cyy7Gw2X9Szz+H3pV5/oddH
W+BNAwo0H+6Ww/uJfrf7u5rbduNWtgV6iqbKbD3d/dE/um9Sg3o6X05sT9E8kw3rP6d5Nq4H5di3
K/tENfet7aa8q7K+7d2y+aCeVd+0YVXNQQNbBL0eF+HOrQWyqyKLy0k9Hdk6YJuH7eWWs5n9WOZt
srqT/5pW/31ru75qOP78vb2p7I3H7mXkbRt0pnM/s/oC0blFd4c9+a+8nJSDm2ZLv0vN+Xj0WWW7
//nN2+w/xnXfVo5JtSiH5aLsZf97PrFQ1heV00F13svel+N59e8nkcqjkLrns9L+3Z1Wi3k1yN6M
hhl964ir/17MSst6vr6Ny9tyNq8u76rZ3LqYd/Rf0/tXrqpysZxtvbYYTZybn9y+Y6JQvKDZm1+X
Vfb3wSJzN6d7gvYIydwNv/3X1NF2RkP3QV+Q7xjnwv7AF6u98zazLztDvFvVgtt/Tc/z7Nx9elpO
nGXPN27GvVSOR3f3Fl+91f6QfYbaR411Ou47l/Nzd9eNY3VfdF3XQ/cNdxP3aHnr/h7aGx6NV4//
bZ/wsPaxw7UPD5Ty+jcX7vtXpng2JX9TzcrNkmdHLnnCjY3DszcfrQ7xJa8yYnpW5lG+W/JsXfJU
mSJZ8q7N75f9xu3Ey54dLHtbHPN66r7obuL789ly6sTFloryv3lfSfrlRh1xjmGnlkz/mLonN6uK
u782VaWxXeuq4k5kO2VVqQbj0e282qwt/Li1hVPFKWXZm/ezUfapus2YzCjtCXt/are28Pt+QhDy
qH5i+37i1YUftasQLrtLsvxX1ng25V/e1uNxvVn88tidBTOUqu3OQjTJdXeLX94XP2lR/Ic7i60b
ipe/ROwuGGlVXVbWe1p1MUepLj95ZexqiGhqSGO5eyuc2nJNLmX77K+zpTOSDb47Li63z9imx7ph
cw5ulx2XmMk+RbqcMOcmrq2N/iw/r76unX1Nc3TfCxTGosj9yN6LQxdWTFq9xDeiPv6k3s//9tvs
k/sns1WSZEMrXxd1trzHzGztnd8sF646ZuUi29S3Rc/+xxt9+xD6F2h4SnOfJeYFojOL7oaFn/9I
gWVluZAv08yc59wdMfcC0YXIuZsUeoHoUuacv0R0n1lKuG35L7EvX9G/xJYqe1S/0E5GuuFInz38
+fflzQ4B/TKbJi+a3SkvEF2S3J9Z8OLQVY/Q3J/f9NzQ4ed/Nub/1oGlm/o76m+4WT4tZWD+z1/3
83+CqTNbWpSKs0welSJwfeXzf4HyX8/4P3nu113R+V9GuFT35S+1cOWvKZWv878Q13q9iFbhvO9c
58ytOF31bFYLJJeKrN7a/bj+yKpznC/7q5esfnTPuGGqL0/6PGrftGYyG0zutIojMVn5uoOUZ24e
z+1iEMZWWSUIVa0xJQUxnTvnrNMM1rVG0wzCghtku4bk1Ij2tDB1cD3A2Z5LgVhxhbVtQkGJEKY9
qgYx4XrktzOtV0n32xJyBlIl9wF3zEoKZUh7aAFi1q2h/9ZwXEBYdIttw5i0UFJqzlrXUc38LH09
s4r7sl8O/qimQxt/UpeRNkA+HvW7+594czm4qQZ/XM4/T8aj6R9zK8fHVeknRpa3TpKPq+yn3zL3
WnZ+Mbup7i7ctFpnYBV7NbuYTBcXjTqaXq2XQPQuNwlWNeeCDct+Scp+h/X5sCNst9bpD64GHToY
Ek0HnBBuLm7KSx9CXGx91iqswR/z23JQnW/YJ3xUlLMPRkqlRJkVWnxFZbaegG1dZEUBvMlLq/AZ
Wp4HYZV7kgl4ibtW4dOzuMkJQdiLn2SCtpEOn5vleChCDowUE4NuazrSNZq9TcNANkowAW/m0jq0
tXPFg7A3N8UkgBMV+0PHIzzOxT58hwTlm5GqDh/G61ZZYORN0zp62xIjKUOSCXiHn9bhA2/96hgM
RZhiAt7bp3X4qNsDPEA2SjCBu/OITHU8KK4qwQTuqiIydbUSDd5GcSbo40sTfkQJ4EwNOniuj3AT
rNydF37iMyc1Z8XmmZOykDtnTirN5YYFI0K/yCVGMuAEkwZviSYi9B0PRkuMMxnoc0C0idZ8gyKs
E0zQeVy0iQj9ojl3Ht5GUaYCOuu2NpEe3fFgKIMIkyA5gc7Iq004KvA8CAkRUkwKOogNHvrneag7
RulRQezGXIwJBx6C5gTjvI8EE9XQxWDCgYflYQTFdYaFvmOiGEFsigk6iDVhoS/c4nqEwRAVTHna
MLnksEc6KS9RbRV0VsFEcSjos8xUJO6yPD4McostLpvFFrZEiq0nuh/t339bvXZfPO/r5XSYNe/I
PL/7SPbnaHGT3dazxfyLPYpI705zg1E9tQlHLd4mKO4gymQMdD1OlFsB7Z6KcITgeTDGoYuwIrdM
BYrLjNmJWakDPe5TRPoftpfLFL/c2F7qOwAbRRS55TEodTsiT1lOcZiidQn8LLOEjRj02eCJesTA
pyeLiDZjOScYI00JJnD5XkTCCceDMdKUYgLOaa4ic6uOx6//fMQIAdlYR1xEZLv7CZRiSDABF4Mu
IhLV8aC4hDgTB7aRIRHZ7HgQbJRkAp6AMCQqUQXGiTZJJuCBY0OiElUShBBVBQ8daZjEYwdqdevb
FsDziSp4ZkXD48Zw17+5mvqlbDdtbDxH7Gjulilvpt1phlGaz37OdvZCGBLRwDxXGIccJZg0tKMy
JKIXeLNbAN5GUSYDna3bkEjswvPCIIzLpZigx5wMicQK7sghhBnOJBNwD2lIJFawPBSlHsWZGHCM
Z0jEce6fvQJkoziTgI7xEjxuWPNR4oJt3HIkVpA5xxjSMrHJb5kLguLOo3baZTp980nZCFpeRJYO
eh6UepRggo7xaETI7/MA2SgiUy3TztAfEFPcTtDDkUkbQctUGpHNjgdDgqWYgKfYZGxBl+WRjxwe
3dgFZGhEmaucKJQuL80EnX4kxeSWS2OkHxGU6o3t9ClMg5d+JIFGnRZATD8iKCkUaU8LZMjd9CMp
LgVSD0PpRzjlrbsWvw4SN/1IkhDEmNH0I9L9TxvTuolTA2PWw+lHUnBufQZe+hGXZYgKpVlbXn9i
6FeTyuJA+hHDItGHyg3KIFCszHRO3czgV1Nm++lHokVmzQO9U8KwSCDkslBhCP0UE3RQzSLBmeXB
SNGSZIIenGGRYMjxYEQqKSbwehTtGlmBMT8WZ+IEOEWLYZF5FseD0tbiTBTaRjQyZq+bhSWPGnjY
zO8YrRbSZUo/9TZtobe2aWtK9rZpq2KjkCKTT5bYYIzox5nUTmUGaFzRiqNQZj1STOCOLDIL43hQ
HFmCCdqR8cjMkOPBcGQJJvBOmkeFvkKZ9UgxQc8y8KjQVyizDCkm8LYWEfpFzhSKjaJMogCeiTE8
IvSLXKFMkySYoBNKGR5RdJbHoNSjiLC2TCiBdYRJkpww8LodLjfHI6EXnvGwpnU8O7lhgcosykSh
c/qayIpty8MExkr7OBPn4DYKa1rLIxlCjgkTWXbumDhG3RZhzeaZMPptEdZIngnF38brE/jAeqLc
wPvtRN2G5wlrSM+DsVJNhDWbY9Io7T/q/6XGaGsiqpF2E4ECMcXrkwGv39E+En7MJl6PFIfWtfG2
hsAT1ZCKo9TpqGZTGImSjYj6Wc0wkq9E8rE7JsWfnMou0eUV0DknjIy4c2rDMIwh2Fh1pTlj0EMe
sSZtecBDnljToc2hac+oDiHwRCSz48GQXzLixmnOJUY7i9uJQ6ewTNlIQ0/hyYgUtDwoQ1QpJmi5
IyMukyJtppIRiWqZMNKOppgEtA9JlBv0+QJGRnTRPg9QmUWZJAX3a1EdIhlGaJFggs6XKlVUi0i/
NP0xuaY2lubLqDxVO9vTgYohzgR90p6MS1Q/k3ridV9KK7a57ktpvbvuixi5YcEosWYYzj5emTVG
xq6onVhOoYOzWPp5uUqa+KgGX7QrBra33vb0fiByMqvlkRhHNKaYGLSvVJFYwmV+woi3VES7s1yh
xIBxOxkDPdYSt5EBn4KPDQlynKMnkkzQUxSRRd6OR6DU6ziTZNA2isQSlkdhiMYUE3jbj8QSlkdj
LCtPMUGPSUWO1HY8KHnfEkzQ+6NiiS4tDxWPnMYhfOOWIypY5Axl2CvOJKCrqoqkA3Q80K48ycM2
dq27XIUP37Ve3+5sWq9vbw/kG9nYrM45UYawjeGFh2E+anP9IcxoCpKnUp7KmME8H1xSItXzsulp
YE9m2oPJPyxqIdkzs+vxSU9l1GgSECs1mNEPguYA5j0us3/59IYO5gbxvHLj5IAUrwvwTm7j9rg6
Ns4i4LdeteEBd6ouA4wSZCMBWhJTwjvVp1KCO1UrrKl5QFWEsOlpYGGdqqBECPPM7Hp8UhynaqFJ
YcVre2iF7lSfyAztVGmhpNScPaBiQNi4Pa7SaVzQSLUFD6BT5YRopjYjlBSfhvSmT8QDdqNe0XH9
zIx5GlhAN2obtdLFZrr452HX45OiuFEPLYsNQZWENthu9KnM4LGpjwc0I+15C8zYdB83tsV8hQs7
D5DmAY9N3VCq1rxgrTEZwRnwfQolrFMtuLCSToj2VRHEpqeBBXSqDerW2PSzsOvxSVGcqocWXD6z
ruDYzBTZqQpqcR/Q2iiuU93B1ZHDsNa4oAO+LXjAnaoNVARleqNWPgwTaMD34ZS4s6iCUrPRpT6M
FnrA90mw4AO+fHNs+lnY9fikiJGqLh7byPAi1cczIwz4Kl2Qh7Q2iJnq9rhGR1b0O1yMVXQJJvBd
5onBB46xyyDBJKB3LLvMGTEejARtKSbonRg6sqLf8WCsoE8xQe9+1ZEV/Y4Hpa3FmRRwW4udMOt4
Cv30Y/J0PDQsUIohzXTvWaGOyYszeTFwzwR2TJ6PrtjGuFoKc+ssOuBj8hJoW/oO+Jg8P/ZbiAcY
ckvdQR6Tl+JSIFbcj5waE2r1ABNqEBPGjslLEHKQKhmNk7xZjVGiPTRMzdyKPNrDgRw8GJu/o2br
NEwTVdf+9Ihn5v4KNyL+1RwDd+jovph9ZM6cI/lq7HPgmLxYlZYIR3eZSHAmcykw9o2ZSDAk9zKe
AjFF7bR7MgxAuUVtpMBTAUXODHc8KJmzE0wCequsiXaNSmJsJ04xQW/dN5Et4NJGcig2ijPBD85E
Bh5k7nfRP2rgQW/ccmT4fPUT8MUQZzLgXV68GAzGOGKKCbqqmsgWcMeD4soTTNCuvIjMs1ieAiM7
ZIqJQdsoKptNgeEWUkzQ8qKIStRdHiAbRZkKAp0GMC4JC5cU9VGucyNvXxFR5mrvbFSgs+sjXZ6y
CoafPPuh4YxvZT8sip3sh1ox0daKWoF30pGKY3k0SiedYALvpCOxhONB6aQTTOCddCSW2OcBslGC
CToGLCJC3/IYlLaWYAJvaxGhv88DZKMEE3SfHTlO3vIY+tg4edNNRRyryQ3KwXoJJujhioJEYgnH
gxADppgk8FqSgkRiCceDsG4ryQTc5RUkEks4HgR5kWQClhcFicjmfR4gG8WZFPA6Uh90RngwpHyK
CXq8JXpqgbFx+yPj5I19RDKWkqNozhI+dUb+QoqtmNTe2k5GflaojUIKxRKyR9xxFwjOviAh7e6Z
FMZ8SZIJvFMMafcVD8JkRpIJeB1Aom6Dz28XwcW1DQ/GmSYFCQnrhgljUXSKSUI71+DpFysejHJL
MYH3R/G6raCdffB4y4aneOSJJXTjxJLgqjD/E7oA3wOS5kHaNNxuwfghTMhNww+iBEm5+fS144do
QTcNPxUWcNPwQxaUQ9r1+KQom4YfsrAc0LwnZcZIb7Wz3vw52Lg9rhQJXGBd04YH0KlyIqx00eyR
fKfPEvkkPCg36jHZRqNOYkKkUj0yJYjj9IybmXSehyWPiAjsKj2t3kgG+iwMehpYDOdYSP0Qb4OZ
QvkAbvAc4BUu8KKGNjzgEaeXbEpsjHE/DBMw9+MTKBEiTi03khQlaSEy054GFjzipOwBTRzErscn
xcv9+LAeFD2h8lOZoZ2q52Vk48CKZ2HjB+Dq6Myehj54ug0PuFNtktSbjcXMD8MEG8Z9EiXCKQWK
cPa8bHokWEOwEirLoqCEavpYVDin+lRSvBP0TPHczHtSZpQT9GxP1rojg7Fxe1xDo4G1IdB5Flvw
wDtVn09Xb5yR9DBMsBP0nkSJkFCZbE2ApGgZbkLlJ8BCJ1TmlKtnZtfjk6I4Ven+p415QCODMO9J
mVHmRoXamJ16FjZuj1vQ0Opii8stLsamkwQTBVrzefbMriYP17i+nl+sC9je8IVPqNW1Tx/jN4i9
lBDuX6b51r/uT3c69xllynaFWirBzwhVmqqzjBzjx1PX0uXNy7KzmUtz9jn8vtTrL/T6WI6mTRsJ
bgjgOi/cNtStltDriUiSwqb6bD3d/dE/+mH16TeXI9utjMrx6H+qy6ZJvc3+5v/NvrwyzLNyPN7o
hebZvElq+E0bcJNTF2/sgEu3xvM44PNy6tLwvc1+mM6XM9e/j6tyXmVXtr+8z8+XbeWly/1D331m
o3lWDv7fcjSzPe2bq9G46mUXd+XsYracXtwN55OLVeq9dnnwiOgTMzBVR0l91RH9ctAphsOrzqAY
SM1tgyPFwH6TvpK2zXUEG9COEGzYMYzrzlW/LIZKG00Mebtl3B9++eXvv4SsS/et6xIkHdm6g3I6
rRfZdbXwVs1sd3pvxF7zZzl2tJ/tg/Fwfv+uMhuOrq6qmf3Z9ScedG8e2LoDV/2j99T8/9vm7W8u
bQE2OQltzZjNrIvNzt8wlmfnn5pb2qgc5XCYXZWj8XJWnds3fJje2ZsfZuXsejmxX3D+9jzrZIvZ
Z1e5FnU2q3wDyPy3b93LP7795ecPP//HgbsR3N6N/nI3JqxoUnfzS/PzDsY/Y9vnYlFNbhfZuT5v
0yYdi1uHvlNrGEtZOF5rbNXwH5yWE+t239fL6TAbVBb0ajQoF1U2qCe2B8ncyz1r22qddrIVssqt
+9pD5ubo/d+H9VPOvjdlp3newq9VRGtcqxebD7scnG7fVpjVpfj88t5u0wf/5D+2yim7oQpXZb8d
gte3jm2e/e+5rcyzc1vZKelq2rX/Z5Kf/7sttauXW9QiXFEfQr0cDKr5/OBQQVsyCmPPrfD7i1mX
c9sZzsfOsovZ0vcTzZsuXYV2T3sW92n3misFe8PuedLa+pTu3uNJre9HEx5Ldirr3wfpUdM/zrwu
aQ6cef24QksyvdvsTmTeQ0F61NB3k8vlcjR0T2krXio2MB1OldUuot/vWE1qZYvpV8IY1S/L4ePL
RkOVzZehiMeSnahsdoL751EsbocNVJNZDWG0IdM5dccdbZFFRG+UzMqWdXrtYT2xP34Izx9ZpQoh
HoL34qI1C870vsx6jdaOEq1566q/ZrTm7838VaI1yXLmMiWeNlozraI1y6LI/tDPc47W3DFILtnd
C4nWPK54adGaP9dp1weezDvfb3ptSwYkmrCitfWZUCDW/7Ivti3Zrmp7adHa+rAoQPPqdsHw+myo
rzdaW589BVc2flNtWzKgqv/sojV38wJw9G61GbcNmcyZ2m0yp4/WCkna1ZoV3ouL1nwe6Ndo7UTR
mrfuXzRa8/em/xrRWrPc3eX6Pm20Vpy3MO0BmFRjDMP8WtfZpJx+zipn43lWDwbL2cx1Gteju+ag
t272n02LnFV3o+rPbHFTWdtbeBfNWX80Gtr+xrY8f0NZf3ndTXcsq3vgX+7BZRx8wD3YW3ib+QfZ
/Ga58NYc1n9On93Cm2dy7a3/uTj+b7hVPlrKwPoff63X/ygh7fNUUMHOMnl8lP3rK1//s1/+K99x
pLVf7oqv/yLWf96v/1KcufJX9HX9F8y13f8S0hMk59xkzuNl2T/r2XXXV5HuqloM6lnVHZbj7rC/
WghZLoejha0tfs2j7W6737onfqqvv7faa2CN9nv25vvqqlyOF/93af3b/3wa3FTD5biaXf6jntlv
6BTibfZPyftScfV79l1t/cy49IsmP3zfy6bL8TjPvnNa1QZCgz/un7ElV0+yH+5cX+/e2KF59tFq
cesQetm3d9bvl/1x5cYJbMFl9dVKzNzW43Gd/VMwkn382+9OJy6nzlMNqyt7g0PrxqwOcFIn+ycl
TLg3db/ZMZPscZYbWWTOiQXN1B+Pu/PmZp1dfiyth69md+X9WlH7bNI43Fjj8OFgyKQpfs9+/NbJ
lPW3OMW+WM69Jh6MrUHsfay+zt3Y3//PLrh1sRZcqzi41cerwv1tOP+v22G5qH5ZTn8dTaoP06s6
iUwJscxXVF4N+v3fra65chb9bfLTyNr/buKkZZtQdH1z77J/lKPF+3r2U7mcDm6c9rSA2W09r0ZD
+2B0Pa19fDCyNz31MmTW/KQt2cVonE1G17PGXNYm9hPVAauonLBEcX6xype/vreabVZ//u37T9/V
E6uXhknjaGcbI3jR79vK/unXb3/5Nc/2vuaNk+4/l5PK3v36RvPMPflhaJ8aVgNB6LDfIWpYdYQy
/U4ptOhU1ZCxkhamfyVza+oPw3dtLJ27CjSo3l2V47kNouaVlW/D+a+1s/s7YiXfzKrpK9vqPq/f
YiXcvJ6+e5t7xTca9jIqClqRK3nYtqmmcgrbvv/w84dPPx4wbltow0/aTLZpn95Kvrcy9/iNg5Ie
5zkRGqCvk76vq64KpQblU/s6G0tYcMYMALhxRUkL3Rf0ydhFTxh39utJ6x5TFliIK6VKpl9AF22t
Im1gKBKFeZxuRHg9UlEqZDX4S/XRlWDiitPDxtUUwbhtOuk4tTxpQ9nBfa69NPO9NC8SHus4vXTh
FemVFn2lntrdMd9Ly5QiPQq40q7TGwyIlrx6Kjj3FhfqtP30sTXCibvpxiiGJtrkUUqTUmsaRclA
EsoeXprYgffr9Xq9Xq/X6/V6vV6v1+v1er1er9fr9Xq9Xq/X6/V6vV6v14mv/w8xCHBnAAgCAA==
--MP_/_7Zm8Xi//waNSwzp1NQwbMu--
--Sig_/+j1R2P+c=Zv.QGARGGJyPq/
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWL3/8AAoJEMHFVuy5l8Y4o4AP+gP1clGgqzj9nPnMNyq/IY7r
EaqE6VG+sO/WpFhax1NkCEDFxz2LEQ5V53ptcIkVrRX/Q0eV+x3IXsdfK0oNxDsM
bCW396uQmhOW2Y73ad5KOqA/uW2Lu2R1fnzQn4nTjXzvIDPG1/Y7BJDzN0NiJc3N
uxBl3dF3AZJDHi3mpMspAQFIqqVG1bOBa823VvY8PSwNENR1ayk3DkMF57ex5MyJ
hlynpKi98MukjdAY3/0NISawQGhKjJ1BCAlI6NErPAzhUa+SG6PTkGqdc9X16+Wu
EycdieVDyDSveARHVxu08OhcxWKsc/TxPIkuQPr6De8/GcF/4LRfPAlbN2fljGyx
D13Zl+A+e39A0YKJe0xZ/dYDAHim9buGqPkokHdLcXmflXrqWMtHw9V4oMdESl7H
BTmMhzr29SO2mLuCq4rvYg7S9XXMJX4+cwvkm12EyIGFWCi868uOmgup1CSgQSdR
NoblaGoQWwBxA56MvE0eCBOVLvRur3S3niMulBGQ7rTy1GIkMFi/v7+v/NHm2KYC
BnqsvEbF5UfVxCPE4oqKe/nGifccrdfLL+nrBQqdKvXwjgeDoO90OYEZf70cUTR5
ujX3aVDfUGdpXIC64xJDKT+QUF3IaVnn3QrlWZoxO6C5TCUqPcTw0KKxMkvGAJ7H
yVqQ23/zK4E0XWq/8u5e
=i2ip
-----END PGP SIGNATURE-----
--Sig_/+j1R2P+c=Zv.QGARGGJyPq/--
7:57 a.m.
--Sig_/sqDHQzA_mxf_gS/AkMx=kG+
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Tue, 27 Oct 2015 09:45:28 -0400 Robert wrote:
RS> I have oVirt 3.5.4 on CentOS 7.1 hosts, and everyone once in a while
RS> one of my hosts starts sending me the 4 engine status messages above
RS> about every 10-15 minutes.
I upgraded the engine and all hosts to 3.5.5, and then 2 hosts started
sending me 4 emails every 10-15 minutes. Currently I'm running with the
engine in global maintenance to keep my inbox from overflowing with these
messages.
Any suggestions on how to get this under control appreciated...
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/sqDHQzA_mxf_gS/AkMx=kG+
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWMhe3AAoJEMHFVuy5l8Y4dQQP/1PXm4lqMOwOTYpzDaHD5vHw
LPGlAvnsxIZKUC3SlWhzyJkc3BGygYUwM846R/8/IZrhAfNZ7wMZ5py6ezi08MzN
2vvKbEFY0lW08KE/iI628HOezM2RIhtKyJCuq9vMgtQp3oSy5cjS0L1fQwz8whtf
j9DYjj0d0/PaopTQyeybWtxMij/NXRCDl7AtidPsuyOTux5ywy/S1KNn6F9JkqUL
rdOFogz+43Jr/3dUNZZ7qENPkGLUQqU2MwXmEJHq+t09BqB0sxK75IH+hQmytJPb
tDyON6dfptTMGDC67pBeY65eVIOnP09yjyqjOz3lfFXbUDZ/D4Yjuq48XGdxKUOi
TSItyLl5xt9OmAP2DASrIJyt4BUavMnsxMTvm7SEDNDwnrJMp2kfwskpzC+Vx8rW
0reimy1pO+zl3hGAXfKzP8Ld+jfmodRzo6CzoIZ35C+khVsq+O6+Su33DMuzJ56t
VusITyPfN/BAZV9xgN/jFl4xZkb1a98AzfE8WCc8IUEC1viLcLYGFZYtAQZCqn7r
OUNnAqjR2mlNz8LTOvSMIYZYm0IchCLQ8La/iYCeUFakeq9JQJkujCSJTyFs/ktG
u+zd34fr59BPLr6mhekyW1zvakPztVoF8F87iDLdux6vItVQivUGUfOMEG3M6B9R
5yGy7vTaT2ApZ+AYWLHN
=R99M
-----END PGP SIGNATURE-----
--Sig_/sqDHQzA_mxf_gS/AkMx=kG+--
8:08 a.m.
Hi Robert,
it seams that two hosts are fighting fir the same host ID:
MainThread::INFO::2015-10-27
09:14:56,764::hosted_engine::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file:
/var/run/vdsm/storage/2daba0ab-2b3d-4026-bcfc-1cd071c30038/04b08c8e-657f-4bac-9ddf-c9c57373409c/2d7f5020-42c1-442d-8237-fba9d6787080)
MainThread::ERROR::2015-10-27
09:14:56,766::hosted_engine::578::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
cannot get lock on host id 1: host already holds lock on a different host id
MainThread::ERROR::2015-10-27
09:14:56,767::agent::177::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Error: '(22, 'Sanlock lockspace add failure', 'Invalid argument')'
- trying
to restart agent
MainThread::WARNING::2015-10-27
09:15:01,772::agent::180::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Restarting agent, attempt '9'
MainThread::ERROR::2015-10-27
09:15:01,772::agent::182::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Too many errors occurred, giving up. Please review the log and consider
filing a bug.
MainThread::INFO::2015-10-27
09:15:01,773::agent::121::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
Agent shutting down
can you please share the output of: hosted-engine --vm-status
On Thu, Oct 29, 2015 at 1:57 PM, Robert Story <rstory(a)tislabs.com> wrote:
On Tue, 27 Oct 2015 09:45:28 -0400 Robert wrote:
RS> I have oVirt 3.5.4 on CentOS 7.1 hosts, and everyone once in a while
RS> one of my hosts starts sending me the 4 engine status messages above
RS> about every 10-15 minutes.
I upgraded the engine and all hosts to 3.5.5, and then 2 hosts started
sending me 4 emails every 10-15 minutes. Currently I'm running with the
engine in global maintenance to keep my inbox from overflowing with these
messages.
Any suggestions on how to get this under control appreciated...
Robert
--
Senior Software Engineer @ Parsons
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:52 a.m.
--Sig_/wmMSgNEZcHF++Et.y/v2+kZ
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Thu, 29 Oct 2015 14:08:22 +0100 Simone wrote:
ST> it seams that two hosts are fighting fir the same host ID:
ST>=20
ST> MainThread::INFO::2015-10-27
ST> 09:14:56,764::hosted_engine::562::ovirt_hosted_engine_ha.agent.hosted_e=
ngine.HostedEngine::(_initialize_sanlock)
ST> Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file:
ST> /var/run/vdsm/storage/2daba0ab-2b3d-4026-bcfc-1cd071c30038/04b08c8e-657=
f-4bac-9ddf-c9c57373409c/2d7f5020-42c1-442d-8237-fba9d6787080)
ST> MainThread::ERROR::2015-10-27
ST> 09:14:56,766::hosted_engine::578::ovirt_hosted_engine_ha.agent.hosted_e=
ngine.HostedEngine::(_initialize_sanlock)
ST> cannot get lock on host id 1: host already holds lock on a different
ST> host id MainThread::ERROR::2015-10-27
ST> 09:14:56,767::agent::177::ovirt_hosted_engine_ha.agent.agent.Agent::(_r=
un_agent)
ST> Error: '(22, 'Sanlock lockspace add failure', 'Invalid
argument')' -
ST> trying to restart agent
ST>=20
ST> can you please share the output of: hosted-engine --vm-status
Hi Simone, thanks for taking the time to look at this. Here is the outpu:
# hosted-engine --vm-status
!! Cluster is in GLOBAL MAINTENANCE mode !!
--=3D=3D Host 1 status =3D=3D--
Status up-to-date : False
Hostname : ares.netsec
Host ID : 1
Engine status : unknown stale-data
Score : 2334
Local maintenance : False
Host timestamp : 2496391
Extra metadata (valid at timestamp):
metadata_parse_version=3D1
metadata_feature_version=3D1
timestamp=3D2496391 (Tue Oct 27 07:41:00 2015)
host-id=3D1
score=3D2334
maintenance=3DFalse
state=3DEngineUp
--=3D=3D Host 2 status =3D=3D--
Status up-to-date : False
Hostname : hera.netsec
Host ID : 2
Engine status : unknown stale-data
Score : 1689
Local maintenance : False
Host timestamp : 2038037
Extra metadata (valid at timestamp):
metadata_parse_version=3D1
metadata_feature_version=3D1
timestamp=3D2038037 (Mon Oct 26 08:50:13 2015)
host-id=3D2
score=3D1689
maintenance=3DFalse
state=3DEngineDown
--=3D=3D Host 3 status =3D=3D--
Status up-to-date : False
Hostname : eclipse.netsec
Host ID : 3
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 2298393
Extra metadata (valid at timestamp):
metadata_parse_version=3D1
metadata_feature_version=3D1
timestamp=3D2298393 (Thu Oct 29 09:46:21 2015)
host-id=3D3
score=3D2000
maintenance=3DFalse
state=3DGlobalMaintenance
--=3D=3D Host 4 status =3D=3D--
Status up-to-date : False
Hostname : poseidon.netsec
Host ID : 4
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 123241
Extra metadata (valid at timestamp):
metadata_parse_version=3D1
metadata_feature_version=3D1
timestamp=3D123241 (Thu Oct 29 09:46:30 2015)
host-id=3D4
score=3D2000
maintenance=3DFalse
state=3DGlobalMaintenance
--=3D=3D Host 5 status =3D=3D--
Status up-to-date : False
Hostname : apollo.netsec
Host ID : 5
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 2028116
Extra metadata (valid at timestamp):
metadata_parse_version=3D1
metadata_feature_version=3D1
timestamp=3D2028116 (Mon Oct 26 04:14:46 2015)
host-id=3D5
score=3D2000
maintenance=3DFalse
state=3DEngineDown
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/wmMSgNEZcHF++Et.y/v2+kZ
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWMiSRAAoJEMHFVuy5l8Y4uRUP/jwLzddfisVkNgGfN47gMcZ/
zC2CbWiEE+3l7J5/fPXFwO9MaqwW7QlPO+WhTLjF1gZ36ZyScvtBpqyEuCNd8DZe
LsYK5JdBD9J7+SxhGeIIiZJansowLo0Hno1ZKlDnFnubLLI0tpJIOlryQybD/euA
tjkBYvZnUtNw5mOymCdIQ16LGx02zdceUkJLctIFO74QA2Vz7P6QzBArE8TL4f7d
c2Z+IbYrL83I79r8JrMnMlUjlt36jVT/E/1uMDtzVxGAtG+GHxV6WcLJUcsqWkAi
ieC86oPhYdsPfEi6m9Lrq4xhAx3LjulmvbY2bPMemW3QWQTb4iMkhfadc4I8oNx2
n4WI/1hGhwCxf6QcJIX0gkDFyLoP/LEajmbKBIuRowZuuHFdQrOa7Bxkn2NiDfRh
tczXnbbTOAcqtc4seXp/JLuAZ2zyyqh/6IBEYMeaxiWL9EnMYh5xTeY1+EGrqvnB
FCLvr/ukfMm42XAojli1Du0jsDGq/Mk6hbrNb9pLYj/RIrtm7MmxQMCNYRntaDWa
mPeEHFs1dyCMEneNe4NrBtnr66xZFWDT5DG1XLRe5GWJA+cIkcXZ3owka596GxPV
JnPYBEhr1BixfoeYgPLuIvUOpstqmJB9Viq1mb0OccLmFeRDnwHT2t2kM5djQWKq
Cn9yj4XFVzG9zVnNqwW6
=JkhW
-----END PGP SIGNATURE-----
--Sig_/wmMSgNEZcHF++Et.y/v2+kZ--
9:40 a.m.
On Thu, Oct 29, 2015 at 2:52 PM, Robert Story <rstory(a)tislabs.com> wrote:
On Thu, 29 Oct 2015 14:08:22 +0100 Simone wrote:
ST> it seams that two hosts are fighting fir the same host ID:
ST>
ST> MainThread::INFO::2015-10-27
ST>
09:14:56,764::hosted_engine::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
ST> Ensuring lease for lockspace hosted-engine, host id 1 is acquired
(file:
ST>
/var/run/vdsm/storage/2daba0ab-2b3d-4026-bcfc-1cd071c30038/04b08c8e-657f-4bac-9ddf-c9c57373409c/2d7f5020-42c1-442d-8237-fba9d6787080)
ST> MainThread::ERROR::2015-10-27
ST>
09:14:56,766::hosted_engine::578::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
ST> cannot get lock on host id 1: host already holds lock on a different
ST> host id MainThread::ERROR::2015-10-27
ST>
09:14:56,767::agent::177::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
ST> Error: '(22, 'Sanlock lockspace add failure', 'Invalid
argument')' -
ST> trying to restart agent
ST>
ST> can you please share the output of: hosted-engine --vm-status
Hi Simone, thanks for taking the time to look at this. Here is the outpu:
# hosted-engine --vm-status
!! Cluster is in GLOBAL MAINTENANCE mode !!
--== Host 1 status ==--
Status up-to-date : False
Hostname : ares.netsec
Host ID : 1
Engine status : unknown stale-data
Score : 2334
Local maintenance : False
Host timestamp : 2496391
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2496391 (Tue Oct 27 07:41:00 2015)
host-id=1
score=2334
maintenance=False
state=EngineUp
--== Host 2 status ==--
Status up-to-date : False
Hostname : hera.netsec
Host ID : 2
Engine status : unknown stale-data
Score : 1689
Local maintenance : False
Host timestamp : 2038037
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2038037 (Mon Oct 26 08:50:13 2015)
host-id=2
score=1689
maintenance=False
state=EngineDown
--== Host 3 status ==--
Status up-to-date : False
Hostname : eclipse.netsec
Host ID : 3
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 2298393
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2298393 (Thu Oct 29 09:46:21 2015)
host-id=3
score=2000
maintenance=False
state=GlobalMaintenance
--== Host 4 status ==--
Status up-to-date : False
Hostname : poseidon.netsec
Host ID : 4
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 123241
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=123241 (Thu Oct 29 09:46:30 2015)
host-id=4
score=2000
maintenance=False
state=GlobalMaintenance
--== Host 5 status ==--
Status up-to-date : False
Hostname : apollo.netsec
Host ID : 5
Engine status : unknown stale-data
Score : 2000
Local maintenance : False
Host timestamp : 2028116
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=2028116 (Mon Oct 26 04:14:46 2015)
host-id=5
score=2000
maintenance=False
state=EngineDown
Here the host IDs seam coherent.
Can you please specify the name of the hosts where you took the logs in
your first log archive (complaining host and engine host) ?
Robert
--
Senior Software Engineer @ Parsons
9:47 a.m.
--Sig_/RT_yaXT/gJJK_NLrp_7i4Pt
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Thu, 29 Oct 2015 15:40:23 +0100 Simone wrote:
ST> Here the host IDs seam coherent.
ST> Can you please specify the name of the hosts where you took the logs in
ST> your first log archive (complaining host and engine host) ?
Hmm.. I know the complaining host was posedion, and I'm pretty sure the
engine was running on ares.
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/RT_yaXT/gJJK_NLrp_7i4Pt
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWMjGdAAoJEMHFVuy5l8Y4/bIQAJC2YrGBaxEhVXjL9JxfAylp
yb9plv1+aYTPAtOannhxpNcWJxZtL4x2Yj1KFjdIoP3OcV+QFVx5RHx3Zi9i25Wl
5Nj7DRs2q/RyHBwmIL+UUl4tS5raDsyZ7ARN44DwudJtgLjv5ooONuEdaJirzgif
Ha2jFElzdRVQ8R8RksvVuTRRroSiCcJGl6A+DtwiezLXcQD4obQpJmlZrtONp6pM
2L8aIJHYPsONf+S1vAYCELaRBlXCcVAMvibxUoyIGIRybunmumyr4ISPTfFig3gV
MZnHBggctLffjXVkoNsSz35EQnWielBxiaL9db5djTPh/7rRGJxn7DNuqgYbPCpC
RWPTvFcTAlNraCP+sMO9CWS0oCVO2Tx7u1EdAOryZIqziWSeU6PKgmmUjyqAWElk
2A6tKY9SC5co1pmqfk/SGMgvuwOAMIS8TTrsD5QTZKzbckhzbRsno9qbwHCpaVK0
+NFfD+pC0NHkX/+35E7XzPBC7ylR8dxC5VX+S16oFluZGNYXbkPQscKHttU/59ul
2yH399tWawkaWAMEwuT5xgn9C3K9EDJlFfY1Pa+KSEm7iZ8v0EPLkhew9zkjmOgb
5wnlj1fclkwA6ebto3KLaVkxH5XcO6/EbFwSJ/NYfcg0itK6ARs7UmPDb+6Vxb+9
yYfwi0ehn2PzTDNtZNe7
=5fzY
-----END PGP SIGNATURE-----
--Sig_/RT_yaXT/gJJK_NLrp_7i4Pt--
10 a.m.
On Thu, Oct 29, 2015 at 3:47 PM, Robert Story <rstory(a)tislabs.com> wrote:
On Thu, 29 Oct 2015 15:40:23 +0100 Simone wrote:
ST> Here the host IDs seam coherent.
ST> Can you please specify the name of the hosts where you took the logs in
ST> your first log archive (complaining host and engine host) ?
Hmm.. I know the complaining host was posedion, and I'm pretty sure the
engine was running on ares.
And indeed ares was host 1 so when it failed it was correctly trying to get
lock for host 1 but it seams that previously it acquired a lock as
different host.
Could you please check
grep host_id /etc/ovirt-hosted-engine/hosted-engine.conf
on ares and share vdsm and sanlock logs from that host?
MainThread::INFO::2015-10-27
09:14:56,764::hosted_engine::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file:
/var/run/vdsm/storage/2daba0ab-2b3d-4026-bcfc-1cd071c30038/04b08c8e-657f-4bac-9ddf-c9c57373409c/2d7f5020-42c1-442d-8237-fba9d6787080)
MainThread::ERROR::2015-10-27
09:14:56,766::hosted_engine::578::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
cannot get lock on host id 1: host already holds lock on a different host id
MainThread::ERROR::2015-10-27
09:14:56,767::agent::177::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Error: '(22, 'Sanlock lockspace add failure', 'Invalid argument')'
- trying
to restart agent
MainThread::WARNING::2015-10-27
09:15:01,772::agent::180::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Restarting agent, attempt '9'
MainThread::ERROR::2015-10-27
09:15:01,772::agent::182::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Too many errors occurred, giving up. Please review the log and consider
filing a bug.
MainThread::INFO::2015-10-27
09:15:01,773::agent::121::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
Agent shutting down
Robert
--
Senior Software Engineer @ Parsons
11:37 a.m.
--Sig_/0zE=B++yeXzvy1Vlh+eojbF
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Thu, 29 Oct 2015 16:00:27 +0100 Simone wrote:
ST> And indeed ares was host 1 so when it failed it was correctly trying to
ST> get lock for host 1 but it seams that previously it acquired a lock as
ST> different host.
ST> Could you please check
ST> grep host_id /etc/ovirt-hosted-engine/hosted-engine.conf
ST> on ares and share vdsm and sanlock logs from that host?
$ for x in ares hera eclipse poseidon apollo; do echo "* $x"; ssh root@$x g=
rep host_id /etc/ovirt-hosted-engine/hosted-engine.conf 2>/dev/null; done
* ares
host_id=3D1
* hera
host_id=3D2
* eclipse
host_id=3D3
* poseidon
host_id=3D4
* apollo
host_id=3D5
Since I've upgraded, I figured I'd reproduce and send new logs. In that
process, I noticed that the ha-agent was down on 3 hosts, and the 2 other
hosts were the ones generating the messages. So I restarted ha-agent on all
5, disabled global maintenance for 2 minutes, re-enabeled it, then ran a
grep on all the logs on all 5 hosts for those 2 minutes. I'll sent that to
you directly, as it's rather large to be sending to the list..
All 3 hosts that had down ha-agents were down again, so I'm guessing
that's the issue..=20
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/0zE=B++yeXzvy1Vlh+eojbF
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWMktbAAoJEMHFVuy5l8Y4El8P/0LiuqLaTiv/WCrlamVUKti5
2PoSajUlH1ihSyHzhbL7ahNXO0ZrduIiPTgz1386FyVKUodiCr6w24k5at/Dqn70
Gh8BGHApc6FQvFBJx/Rxvpm/WqCDtuPE9DdCLzb7LSytElEHS9SsDgVGG4G+ts8N
ApLuPB5OgwyjCcaKpLF1OQDR6nZn2jchm8HAlZmisUc58R6By4MW4DdATBJDv6Vy
9UyBeovNrO73kFwIdLB/aXpqko9AcqnNI+nX+Zjipo5OsnzolhZKXVhoBrZ2m/PY
H6k29fQSsM7RXJKr8EBWBfaeLWIc7IZHN/hxHXFtWkBRukQ5qgX5VY72Q1xvoHRr
kZG3o1WCNZ65ook0XjDqEnJDYED2Z69t0LOBjjDO6dUSo2EE29FWMXbwh9XelUSm
xZQlpR485443GtO7voC3aAvDn2hL/rnF1TF1GCw+vQZbzjeuY2689p8ThUQTEX2S
kt6RnZfM2Y8zJ/1zcvMnnoMcb0Tz3qVOHFuZAkA67MMGoZzUYCJQaI9H9INRSAPq
4FENHiFhFg28sAbQ9+CjItShyWdrj0ruFyeMU05oXmRwmX1ZaSPpNTEyvLvinRbk
aCBw7Kq6QLAXZ2EQqh4kP9qjOUqg+nXh54ep08fkZFXJTn51dhwAFAdQumO+e266
2jkIENSyLi1Nw/n/q3Ap
=VDjl
-----END PGP SIGNATURE-----
--Sig_/0zE=B++yeXzvy1Vlh+eojbF--
9:51 p.m.
--Sig_/1ZxAK_x5H3SBqIdapmI_5i0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Thu, 29 Oct 2015 12:37:44 -0400 Robert wrote:
RS> All 3 hosts that had down ha-agents were down again, so I'm guessing
RS> that's the issue..=20
As an experiment, I migrated the engine VM to one of the hosts with a
working ha-agent process, and I'm no longer getting getting these emails.
So the next question is: how can I fix the ha-agent on my other three
nodes. Seems to be an issue with host_id and sanlock...
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/1ZxAK_x5H3SBqIdapmI_5i0
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWMtsoAAoJEMHFVuy5l8Y4CEYP/2Jd0o85PZSK3XvUTfSJ7wd2
s9mLVbrW7ua69IxcX038sq5IcUOfejfctTVcL4rZhbdnq3zaSjHlF+XvImzYzOHU
O3lznU/n9pMDqD1Jsrqe54Qz8RQc0DtkZPvPx/AMGMU0NnS3jzaOFg3j8VPbCDdq
oac57l5fY4kv0rMumUmVK5mr1sunniPK0O0GHXryKERQJtobUBVCjJl2KcPFAzpY
AkuUy/pUyTsNKcpXIAZFKtUl4sj4Tg92MVS0P9z66L6aDVqUWiG1wLLzcsp19dfx
p1qy2icP3XJvsb6ia17PyGoWwzLsHPgf0zYsT2OPY/y8IuBR+KMIzCzJaNTpcmZc
7QYaoyIVJG8yTx+46RG6fE5g7AaVl2bOY0QxdVgQRAl1DsfdX0UZkM8DQa8a8C/2
lf2CaOqaVLkmUCO1YBhFMDxDHqpArLsPDGjoNAI4GQH2smbodVqPV8JWKufem7Le
MEDP99hJug4ohBizlaTOoX1NonIZ8dcu4xm0KJAdNPx4puBYX8WkubEy6QynM4rR
VWxlHeMQhceqRQcwGNX8rLolfnUgLoTqqM65VNpuwUAFy8nqhQF497fpOJ9NLA/N
fVWnqpvmzb5kv+/WUrL4aXtlr8ImY+sKJ+jZgD8dSbpQlZ8ScqfbpafITvYINor5
Uuy+b0+0IaExuV9xaoBf
=n2/Z
-----END PGP SIGNATURE-----
--Sig_/1ZxAK_x5H3SBqIdapmI_5i0--
3314
days inactive
3317
days old
8 comments
2 participants
participants (2)
-
Robert Story -
Simone Tiraboschi