Hosted engine reboots every hour
--_000_CFEE1CAF46A1194DABFD2C2628D324067A591Dmcexch01mcongroup_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dear Experts, I am experiencing a problem with ovirt, every hour the hosted engine will s= hutdown and reboot. The engine-status value will move to "unknown stale-dat= a" every second minute, and then the hosted engine will be again operative = 14 minutes after that. As far as I can see the scores remain in 2400 at all= times, and seems I have a liveliness check failing, but I am not able to f= ind why. Why I have this problem every hour exactly? Why the liveliness check fails? I would appreciate if someone can bring some light, I am new to ovirt but I= really like it so far. During the period the machine is down I can see this messages on the /var/l= og/ovirt-hosted-engine-ha/broker.log : Thread-170803::INFO::2015-07-01 11:06:47,335::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170803::INFO::2015-07-01 11:06:47,342::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170804::INFO::2015-07-01 11:06:47,343::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170804::INFO::2015-07-01 11:06:47,344::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170805::INFO::2015-07-01 11:06:47,344::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170805::INFO::2015-07-01 11:06:47,346::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170806::INFO::2015-07-01 11:06:47,346::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170806::INFO::2015-07-01 11:06:47,348::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170807::INFO::2015-07-01 11:06:47,348::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170807::INFO::2015-07-01 11:06:47,350::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-7::INFO::2015-07-01 11:06:50,394::cpu_load_no_engine::121::cpu_load_= no_engine.EngineHealth::(calculate_load) System load total=3D0.0095, engine= =3D0.0046, non-engine=3D0.0049 Thread-8::WARNING::2015-07-01 11:06:50,464::engine_health::116::engine_heal= th.CpuLoadNoEngine::(action) bad health status: Hosted Engine is not up! and here the /var/log/ovirt-hosted-engine-ha/agent.log : MainThread::INFO::2015-07-01 11:01:04,216::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:04,217::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:24,724::states::393::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localh= ost MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:35,234::brokerlink::111::ovirt_hosted_en= gine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=3D143571969= 5.23 type=3Dstate_transition detail=3DEngineUp-EngineUpBadHealth ho stname=3D'mc-place-compute-01-live.mc.mcon.net' MainThread::INFO::2015-07-01 11:03:42,536::brokerlink::120::ovirt_hosted_en= gine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of st= ate_transition (EngineUp-EngineUpBadHealth) sent? ignored MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:03:53,060::state_machine::160::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'m= aintenance': False} MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-02-live.mc.mcon.net (id 2): {'extra': 'metadata_parse_versi on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226246 (Wed Jul 1 11:02:= 11 2015)\nhost-id=3D2\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineDow= n\n', 'hostname': 'mc-place-compute-02-live.mc.mcon.net', 'alive': True, 'h ost-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226246} MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-03-live.mc.mcon.net (id 3): {'extra': 'metadata_parse_versi on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226256 (Wed Jul 1 11:02:= 15 2015)\nhost-id=3D3\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineDow= n\n', 'hostname': 'mc-place-compute-03-live.mc.mcon.net', 'alive': True, 'h ost-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226256} MainThread::INFO::2015-07-01 11:03:53,061::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-04-live.mc.mcon.net (id 4): {'extra': 'metadata_parse_versi on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226300 (Wed Jul 1 11:02:= 11 2015)\nhost-id=3D4\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineDow= n\n', 'hostname': 'mc-place-compute-04-live.mc.mcon.net', 'alive': True, 'h ost-id': 4, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226300} MainThread::INFO::2015-07-01 11:03:53,061::state_machine::168::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engi= ne-health': {'reason': 'failed liveliness check', 'health': ' bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 136637.0, 'm= aintenance': False, 'cpu-load': 0.0035, 'gateway': True} MainThread::ERROR::2015-07-01 11:03:53,061::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 300 seconds MainThread::INFO::2015-07-01 11:03:53,081::state_decorators::95::ovirt_host= ed_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Wed J= ul 1 11:08:53 2015 while transitioning <class 'ovirt_hosted_ engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_h= a.agent.states.EngineUpBadHealth'> MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:03,559::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 289 seconds MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:14,007::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 279 seconds MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:24,505::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 268 seconds MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 Best Regards Carles Cortes Costa --_000_CFEE1CAF46A1194DABFD2C2628D324067A591Dmcexch01mcongroup_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html dir=3D"ltr"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style id=3D"owaParaStyle" type=3D"text/css">P {margin-top:0;margin-bottom:= 0;}</style> </head> <body ocsi=3D"0" fpstyle=3D"1"> <div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: = 10pt;"> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> Dear Experts,</div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> I am experiencing a problem with ovirt, every hour the hosted engine will s= hutdown and reboot. The engine-status value will move to "unknown stal= e-data" every second minute, and then the hosted engine will be again = operative 14 minutes after that. As far as I can see the scores remain in 2400 at all times, and seems I have a livel= iness check failing, but I am not able to find why.</div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> Why I have this problem every hour exactly?</div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> Why the liveliness check fails?</div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> I would appreciate if someone can bring some light, I am new to ovirt but I= really like it so far. </div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> During the period the machine is down I can see this messages on the /var/l= og/ovirt-hosted-engine-ha/broker.log :</div> <div class=3D"-x-evo-paragraph" id=3D"-x-evo-input-start" style=3D"color: r= gb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal= ; font-variant: normal; font-weight: normal; letter-spacing: normal; line-h= eight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit= -text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170803::INFO::2015-07-01 11:06:47,335::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170803::INFO::2015-07-01 11:06:47,342::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div=
<div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170804::INFO::2015-07-01 11:06:47,343::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170804::INFO::2015-07-01 11:06:47,344::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div=
<div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170805::INFO::2015-07-01 11:06:47,344::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170805::INFO::2015-07-01 11:06:47,346::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div=
<div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170806::INFO::2015-07-01 11:06:47,346::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170806::INFO::2015-07-01 11:06:47,348::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div=
<div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170807::INFO::2015-07-01 11:06:47,348::listener::134::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(setup) Connection established<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170807::INFO::2015-07-01 11:06:47,350::listener::186::ovirt_hosted_e= ngine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div=
<div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-7::INFO::2015-07-01 11:06:50,394::cpu_load_no_engine::121::cpu_load_= no_engine.EngineHealth::(calculate_load) System load total=3D0.0095, engine= =3D0.0046, non-engine=3D0.0049</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-8::WARNING::2015-07-01 11:06:50,464::engine_health::116::engine_heal= th.CpuLoadNoEngine::(action) bad health status: Hosted Engine is not up!</d= iv> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> and here the /var/log/ovirt-hosted-engine-ha/agent.log :</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:04,216::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:04,217::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:24,724::states::393::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localh= ost</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUp (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:01:35,234::brokerlink::111::ovirt_hosted_en= gine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=3D143571969= 5.23 type=3Dstate_transition detail=3DEngineUp-EngineUpBadHealth ho</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> stname=3D'mc-place-compute-01-live.mc.mcon.net'</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:42,536::brokerlink::120::ovirt_hosted_en= gine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of st= ate_transition (EngineUp-EngineUpBadHealth) sent? ignored</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,060::state_machine::160::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'m= aintenance': False}</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-02-live.mc.mcon.net (id 2): {'extra': 'metadata_parse_versi</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226246 (Wed Jul &nbs= p;1 11:02:11 2015)\nhost-id=3D2\nscore=3D2400\nmaintenance=3DFalse\nstate= =3DEngineDown\n', 'hostname': 'mc-place-compute-02-live.mc.mcon.net', 'aliv= e': True, 'h</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226246}</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-03-live.mc.mcon.net (id 3): {'extra': 'metadata_parse_versi</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226256 (Wed Jul &nbs= p;1 11:02:15 2015)\nhost-id=3D3\nscore=3D2400\nmaintenance=3DFalse\nstate= =3DEngineDown\n', 'hostname': 'mc-place-compute-03-live.mc.mcon.net', 'aliv= e': True, 'h</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226256}</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,061::state_machine::165::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-comput= e-04-live.mc.mcon.net (id 4): {'extra': 'metadata_parse_versi</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=3D1\nmetadata_feature_version=3D1\ntimestamp=3D226300 (Wed Jul &nbs= p;1 11:02:11 2015)\nhost-id=3D4\nscore=3D2400\nmaintenance=3DFalse\nstate= =3DEngineDown\n', 'hostname': 'mc-place-compute-04-live.mc.mcon.net', 'aliv= e': True, 'h</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 4, 'engine-status': {'reason': 'vm not running on this host', 'hea= lth': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenanc= e': False, 'host-ts': 226300}</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,061::state_machine::168::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engi= ne-health': {'reason': 'failed liveliness check', 'health': '</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 136637.0, 'm= aintenance': False, 'cpu-load': 0.0035, 'gateway': True}</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:03:53,061::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 300 seconds</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,081::state_decorators::95::ovirt_host= ed_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Wed J= ul 1 11:08:53 2015 while transitioning <class 'ovirt_hosted_<= /div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted= _engine_ha.agent.states.EngineUpBadHealth'></div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:03,559::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 289 seconds</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:14,007::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 279 seconds</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:24,505::states::562::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health s= tatus, timeout in 268 seconds</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::327::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta= te EngineUpBadHealth (score: 2400)</div> <div class=3D"-x-evo-paragraph" style=3D"color: rgb(46, 52, 54); font-famil= y: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; fo= nt-weight: normal; letter-spacing: normal; line-height: normal; orphans: au= to; text-align: start; text-indent: 0px; text-transform: none; white-space:= normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::332::ovirt_hosted= _engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote= host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <br class=3D"Apple-interchange-newline"> <div><br> <div style=3D"font-family:Tahoma; font-size:13px"> <div style=3D"font-family:Tahoma; font-size:13px">Best Regards<br> <br> Carles Cortes Costa<br> <br> <p><br> </p> </div> </div> </div> </div> </body> </html> --_000_CFEE1CAF46A1194DABFD2C2628D324067A591Dmcexch01mcongroup_--
This is a multi-part message in MIME format. --------------080404070003050909080805 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 07/01/2015 06:52 AM, Carles Costa wrote:
Dear Experts,
I am experiencing a problem with ovirt, every hour the hosted engine will shutdown and reboot. The engine-status value will move to "unknown stale-data" every second minute, and then the hosted engine will be again operative 14 minutes after that. As far as I can see the scores remain in 2400 at all times, and seems I have a liveliness check failing, but I am not able to find why.
Why I have this problem every hour exactly? Why the liveliness check fails?
I would appreciate if someone can bring some light, I am new to ovirt but I really like it so far.
Hi Carles, and welcome. The agent will try to a servlet running on the engine VM in http://{ENGINE_IP}/OvirtEngineWeb/HealthStatus Also debug log will help if we can't resolve this - see the conf the change it /etc/ovirt-hosted-engine-ha/broker-log.conf /etc/ovirt-hosted-engine-ha/agent-log.conf
During the period the machine is down I can see this messages on the /var/log/ovirt-hosted-engine-ha/broker.log :
Thread-170803::INFO::2015-07-01 11:06:47,335::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170803::INFO::2015-07-01 11:06:47,342::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170804::INFO::2015-07-01 11:06:47,343::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170804::INFO::2015-07-01 11:06:47,344::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170805::INFO::2015-07-01 11:06:47,344::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170805::INFO::2015-07-01 11:06:47,346::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170806::INFO::2015-07-01 11:06:47,346::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170806::INFO::2015-07-01 11:06:47,348::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-170807::INFO::2015-07-01 11:06:47,348::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established Thread-170807::INFO::2015-07-01 11:06:47,350::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed Thread-7::INFO::2015-07-01 11:06:50,394::cpu_load_no_engine::121::cpu_load_no_engine.EngineHealth::(calculate_load) System load total=0.0095, engine=0.0046, non-engine=0.0049 Thread-8::WARNING::2015-07-01 11:06:50,464::engine_health::116::engine_health.CpuLoadNoEngine::(action) bad health status: Hosted Engine is not up!
and here the /var/log/ovirt-hosted-engine-ha/agent.log :
MainThread::INFO::2015-07-01 11:01:04,216::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:04,217::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:14,682::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:24,724::states::393::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2015-07-01 11:01:25,174::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:01:35,234::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1435719695.23 type=state_transition detail=EngineUp-EngineUpBadHealth ho stname='mc-place-compute-01-live.mc.mcon.net' MainThread::INFO::2015-07-01 11:03:42,536::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUp-EngineUpBadHealth) sent? ignored MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:03:43,018::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::INFO::2015-07-01 11:03:53,060::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-02-live.mc.mcon.net (id 2): {'extra': 'metadata_parse_versi on=1\nmetadata_feature_version=1\ntimestamp=226246 (Wed Jul 1 11:02:11 2015)\nhost-id=2\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-02-live.mc.mcon.net', 'alive': True, 'h ost-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226246} MainThread::INFO::2015-07-01 11:03:53,060::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-03-live.mc.mcon.net (id 3): {'extra': 'metadata_parse_versi on=1\nmetadata_feature_version=1\ntimestamp=226256 (Wed Jul 1 11:02:15 2015)\nhost-id=3\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-03-live.mc.mcon.net', 'alive': True, 'h ost-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226256} MainThread::INFO::2015-07-01 11:03:53,061::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-04-live.mc.mcon.net (id 4): {'extra': 'metadata_parse_versi on=1\nmetadata_feature_version=1\ntimestamp=226300 (Wed Jul 1 11:02:11 2015)\nhost-id=4\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-04-live.mc.mcon.net', 'alive': True, 'h ost-id': 4, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226300} MainThread::INFO::2015-07-01 11:03:53,061::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': ' bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 136637.0, 'maintenance': False, 'cpu-load': 0.0035, 'gateway': True} MainThread::ERROR::2015-07-01 11:03:53,061::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 300 seconds MainThread::INFO::2015-07-01 11:03:53,081::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Wed Jul 1 11:08:53 2015 while transitioning <class 'ovirt_hosted_ engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:03:53,530::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:03,559::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 289 seconds MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:03,980::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:14,007::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 279 seconds MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:14,478::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240 0) MainThread::ERROR::2015-07-01 11:04:24,505::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 268 seconds MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400) MainThread::INFO::2015-07-01 11:04:24,994::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240
Best Regards
Carles Cortes Costa
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------080404070003050909080805 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body> <div class="moz-cite-prefix">On 07/01/2015 06:52 AM, Carles Costa wrote:<br> </div> <blockquote cite="mid:CFEE1CAF46A1194DABFD2C2628D324067A591D@mc-exch-01.mcon-group.net" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> <style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style> <div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Dear Experts,</div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> I am experiencing a problem with ovirt, every hour the hosted engine will shutdown and reboot. The engine-status value will move to "unknown stale-data" every second minute, and then the hosted engine will be again operative 14 minutes after that. As far as I can see the scores remain in 2400 at all times, and seems I have a liveliness check failing, but I am not able to find why.</div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Why I have this problem every hour exactly?</div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Why the liveliness check fails?</div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> I would appreciate if someone can bring some light, I am new to ovirt but I really like it so far. </div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> </div> </blockquote> <br> Hi Carles, and welcome.<br> <br> The agent will try to a servlet running on the engine VM in <a class="moz-txt-link-freetext" href="http://">http://</a>{ENGINE_IP}/OvirtEngineWeb/HealthStatus<br> <br> Also debug log will help if we can't resolve this - see the conf the change it<br> /etc/ovirt-hosted-engine-ha/broker-log.conf<br> /etc/ovirt-hosted-engine-ha/agent-log.conf<br> <br> <br> <blockquote cite="mid:CFEE1CAF46A1194DABFD2C2628D324067A591D@mc-exch-01.mcon-group.net" type="cite"> <div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> </div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> During the period the machine is down I can see this messages on the /var/log/ovirt-hosted-engine-ha/broker.log :</div> <div class="-x-evo-paragraph" id="-x-evo-input-start" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170803::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,335::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170803::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,342::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170804::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,343::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170804::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,344::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170805::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,344::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170805::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,346::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170806::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,346::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170806::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,348::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170807::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,348::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-170807::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:47,350::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-7::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:06:50,394::cpu_load_no_engine::121::cpu_load_no_engine.EngineHealth::(calculate_load) System load total=0.0095, engine=0.0046, non-engine=0.0049</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Thread-8::WARNING::2015-07-01 11:06:50,464::engine_health::116::engine_health.CpuLoadNoEngine::(action) bad health status: Hosted Engine is not up!</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> and here the /var/log/ovirt-hosted-engine-ha/agent.log :</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br> </div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:04,216::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:04,217::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:14,682::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:14,682::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:24,724::states::393::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:25,174::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:25,174::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:01:35,234::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1435719695.23 type=state_transition detail=EngineUp-EngineUpBadHealth ho</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> stname='mc-place-compute-01-live.mc.mcon.net'</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:42,536::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUp-EngineUpBadHealth) sent? ignored</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:43,018::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:43,018::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,060::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False}</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,060::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-02-live.mc.mcon.net (id 2): {'extra': 'metadata_parse_versi</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=1\nmetadata_feature_version=1\ntimestamp=226246 (Wed Jul 1 11:02:11 2015)\nhost-id=2\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-02-live.mc.mcon.net', 'alive': True, 'h</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226246}</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,060::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-03-live.mc.mcon.net (id 3): {'extra': 'metadata_parse_versi</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=1\nmetadata_feature_version=1\ntimestamp=226256 (Wed Jul 1 11:02:15 2015)\nhost-id=3\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-03-live.mc.mcon.net', 'alive': True, 'h</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226256}</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,061::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host mc-place-compute-04-live.mc.mcon.net (id 4): {'extra': 'metadata_parse_versi</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> on=1\nmetadata_feature_version=1\ntimestamp=226300 (Wed Jul 1 11:02:11 2015)\nhost-id=4\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': 'mc-place-compute-04-live.mc.mcon.net', 'alive': True, 'h</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> ost-id': 4, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 226300}</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,061::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': '</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 136637.0, 'maintenance': False, 'cpu-load': 0.0035, 'gateway': True}</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:03:53,061::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 300 seconds</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,081::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Wed Jul 1 11:08:53 2015 while transitioning <class 'ovirt_hosted_</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'></div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,530::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:03:53,530::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:03,559::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 289 seconds</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:03,980::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:03,980::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:14,007::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 279 seconds</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:14,478::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:14,478::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> 0)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::ERROR::2015-07-01 11:04:24,505::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 268 seconds</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:24,994::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 2400)</div> <div class="-x-evo-paragraph" style="color: rgb(46, 52, 54); font-family: Cantarell; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> MainThread::<a class="moz-txt-link-freetext" href="INFO::2015-07-01">INFO::2015-07-01</a> 11:04:24,994::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host mc-place-compute-02-live.mc.mcon.net (id: 2, score: 240</div> <br class="Apple-interchange-newline"> <div><br> <div style="font-family:Tahoma; font-size:13px"> <div style="font-family:Tahoma; font-size:13px">Best Regards<br> <br> Carles Cortes Costa<br> <br> <p><br> </p> </div> </div> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------080404070003050909080805--
participants (2)
-
Carles Costa -
Roy Golan