Re: [ovirt-users] [Users] Hosted Engine recovery failure of all HA - nodes

--=-jhmMaWvE9+4Rp/kN8BPH Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mi, 2014-04-09 at 09:18 +0200, Jiri Moskovcak wrote: =20 Hello Jirka, I accidentally already send the message without pointing out the interesting part; this is: <<< start logging ha-agent after reboot: /var/log/ovirt-hosted-engine-ha/agent.log:MainTMainThread::INFO::2014-04-08= 15:53:33,862::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) o= virt-hosted-engine-ha agent 1.1.2-1 started /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:33,936::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(_get_hostname) Found certificate common name: 192.168.50.201 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:33,937::hosted_engine::363::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(_initialize_broker) Initializing ha-broker connection /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:33,937::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Starting monitor ping, options {'addr': '192.168.50.1'} /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:33,939::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Success, id 139700911299600 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:33,939::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true',= 'bridge_name': 'ovirtmgmt', 'address': '0'} /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,013::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Success, id 139700911300304 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,013::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'a= ddress': '0'} /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,015::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Success, id 139700911300112 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,015::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': = 'true', 'vm_uuid': 'e68a11c8-1251-4c13-9e3b-3847bbb4fa3d', 'address': '0'} /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,018::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Success, id 139700911300240 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,018::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true= ', 'vm_uuid': 'e68a11c8-1251-4c13-9e3b-3847bbb4fa3d', 'address': '0'} /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,024::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink= ::(start_monitor) Success, id 139700723857104 /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,024::hosted_engine::386::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(_initialize_broker) Broker initialized, all submonitors starte= d /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 3:34,312::hosted_engine::430::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(_cond_start_service) Starting vdsmd /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::CRITICAL::2014-04-08 = 15:53:34,442::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) C= ould not start ha-agent =20 (10 min nothing) <<< here I did a 'service ovirt-hosted-ha start' /var/log/ovirt-hosted-engine-ha/agent.log:MainThread::INFO::2014-04-08 15:5= 9:16,698::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-= hosted-engine-ha agent 1.1.2-1 started .... after this things went quite smoothly. --=20 Daniel Helgenberger=20 m box bewegtbild GmbH=20 P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19=20 D-10115 BERLIN=20 www.m-box.de www.monkeymen.tv=20 Gesch=C3=A4ftsf=C3=BChrer: Martin Retschitzegger / Michaela G=C3=B6llner Handeslregister: Amtsgericht Charlottenburg / HRB 112767=20 --=-jhmMaWvE9+4Rp/kN8BPH Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIN9zCCBFcw ggM/oAMCAQICCwQAAAAAAS9O4TFGMA0GCSqGSIb3DQEBBQUAMFcxCzAJBgNVBAYTAkJFMRkwFwYD VQQKExBHbG9iYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT aWduIFJvb3QgQ0EwHhcNMTEwNDEzMTAwMDAwWhcNMTkwNDEzMTAwMDAwWjBUMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25h bFNpZ24gMiBDQSAtIEcyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwWtB+TXs+BJ9 3SJRaV+3uRNGJ3cUO+MTgW8+5HQXfgy19CzkDI1T1NwwICi/bo4R/mYR5FEWx91//eE0ElC/89iY 7GkL0tDasmVx4TOXnrqrsziUcxEPPqHRE8x4NhtBK7+8o0nsMIJMA1gyZ2FA5To2Ew1BBuvovvDJ +Nua3qOCNBNu+8A+eNpJlVnlu/qB7+XWaPXtUMlsIikxD+gREFVUgYE4VzBuLa2kkg0VLd09XkE2 ceRDm6YgRATuDk6ogUyX4OLxCGIJF8yi6Z37M0wemDA6Uff0EuqdwDQd5HwG/rernUjt1grLdAxq 8BwywRRg0eFHmE+ShhpyO3Fi+wIDAQABo4IBJTCCASEwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB /wQIMAYBAf8CAQAwHQYDVR0OBBYEFD8V0m18L+cxnkMKBqiUbCw7xe5lMEcGA1UdIARAMD4wPAYE VR0gADA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5 LzAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3JsLmdsb2JhbHNpZ24ubmV0L3Jvb3QuY3JsMD0G CCsGAQUFBwEBBDEwLzAtBggrBgEFBQcwAYYhaHR0cDovL29jc3AuZ2xvYmFsc2lnbi5jb20vcm9v dHIxMB8GA1UdIwQYMBaAFGB7ZhpFDZfKiVAvfQTNNKj//P1LMA0GCSqGSIb3DQEBBQUAA4IBAQDI WOF8oQHpI41wO21cUvjE819juuGa05F5yK/ESqW+9th9vfhG92eaBSLViTIJV7gfCFbt11WexfK/ 44NeiJMfi5wX6sK7Xnt8QIK5lH7ZX1Wg/zK1cXjrgRaYUOX/MA+PmuRm4gWV0zFwYOK2uv4OFgaM mVr+8en7K1aQY2ecI9YhEaDWOcSGj6SN8DvzPdE4G4tBk4/aIsUged9sGDqRYweKla3LTNjXPps1 Y+zsVbgHLtjdOIB0YZ1hrlAQcY2L/b+V+Yyoi7CMdOtmm1Rm6Jh5ILbwQTjlUCkgu5yVdfs9LDKc M0SPeCldkjfaGVSd+nURMOUy3hfxsMVux9+FMIIEyjCCA7KgAwIBAgIRAJZpZsDepakv5CafojXo PKcwDQYJKoZIhvcNAQEFBQAwVDELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYt c2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gUGVyc29uYWxTaWduIDIgQ0EgLSBHMjAeFw0xMzA4Mjcx NjU3NThaFw0xNjA4MjcxNjU3NThaMFgxCzAJBgNVBAYTAkRFMRwwGgYDVQQDExNEYW5pZWwgSGVs Z2VuYmVyZ2VyMSswKQYJKoZIhvcNAQkBFhxkYW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94LmRlMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzgFDm8+SeTU4Yt3WopJQgqZAuuNxyMlxiPuq 0C0D581goXz2nVVjhTCIVwX2MqWYD1Dyjy1hLHXothgWgZaiQ1EB4oVdmIFmIfIjR6SkR/Gjw3lx MwJzEpxJhZXyyrOYE8Kgw2maJWgLx5zw2/lKpcffhVW0OY0t+JWWxPKiYFcAmQnb+fleonM8sUZZ ZES08uRVVL67jbq+3+E2xCLlqQ2iJ1h5ej3wlyuZ4CkUnfMHYrG8zOIfHwsPirWACX026a1flgts Kl1Yv0CRZ1c5qujcP3OPpDovIbBr9RBStl2DcFdzTuGMdmfp32963VLOlvKpClPMzrfJeJfWZ4Qy UwIDAQABo4IBkTCCAY0wDgYDVR0PAQH/BAQDAgWgMEwGA1UdIARFMEMwQQYJKwYBBAGgMgEoMDQw MgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMCcGA1Ud EQQgMB6BHGRhbmllbC5oZWxnZW5iZXJnZXJAbS1ib3guZGUwCQYDVR0TBAIwADAdBgNVHSUEFjAU BggrBgEFBQcDAgYIKwYBBQUHAwQwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC5nbG9iYWxz aWduLmNvbS9ncy9nc3BlcnNvbmFsc2lnbjJnMi5jcmwwVQYIKwYBBQUHAQEESTBHMEUGCCsGAQUF BzAChjlodHRwOi8vc2VjdXJlLmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3BlcnNvbmFsc2lnbjJn Mi5jcnQwHQYDVR0OBBYEFLw0UD+6l35aKnDaePxEP8K35HYZMB8GA1UdIwQYMBaAFD8V0m18L+cx nkMKBqiUbCw7xe5lMA0GCSqGSIb3DQEBBQUAA4IBAQBdVOm7h+E4sRMBbTN1tCIjAEgxmB5U0mdZ XcawzEHLJxTrc/5YFBMGX2qPju8cuZV14XszMfRBJdlJz1Od+voJggianIhnFEAakCxaa1l/cmJ5 EDT6PgZAkXbMB5rU1dhegb35lJJkcFLEpR2tF1V0TfbSe5UZNPYeMQjYsRhs69pfKLoeGm4dSLK7 gsPT5EhPd+JPyNSIootOwClMP4CTxIsXQgRI5IDqG2Ku/r2YMMLsqWD11PtAE87t2mgohQ6V1XdW FqGd1V+wN98oPumRRS8bld+1gRA7GVYMnO5MF6p//iHFcy3MVT05ojqgomMt+voH5cFzrHA61z80 xaZ6MIIEyjCCA7KgAwIBAgIRAJZpZsDepakv5CafojXoPKcwDQYJKoZIhvcNAQEFBQAwVDELMAkG A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24g UGVyc29uYWxTaWduIDIgQ0EgLSBHMjAeFw0xMzA4MjcxNjU3NThaFw0xNjA4MjcxNjU3NThaMFgx CzAJBgNVBAYTAkRFMRwwGgYDVQQDExNEYW5pZWwgSGVsZ2VuYmVyZ2VyMSswKQYJKoZIhvcNAQkB FhxkYW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94LmRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAzgFDm8+SeTU4Yt3WopJQgqZAuuNxyMlxiPuq0C0D581goXz2nVVjhTCIVwX2MqWYD1Dy jy1hLHXothgWgZaiQ1EB4oVdmIFmIfIjR6SkR/Gjw3lxMwJzEpxJhZXyyrOYE8Kgw2maJWgLx5zw 2/lKpcffhVW0OY0t+JWWxPKiYFcAmQnb+fleonM8sUZZZES08uRVVL67jbq+3+E2xCLlqQ2iJ1h5 ej3wlyuZ4CkUnfMHYrG8zOIfHwsPirWACX026a1flgtsKl1Yv0CRZ1c5qujcP3OPpDovIbBr9RBS tl2DcFdzTuGMdmfp32963VLOlvKpClPMzrfJeJfWZ4QyUwIDAQABo4IBkTCCAY0wDgYDVR0PAQH/ BAQDAgWgMEwGA1UdIARFMEMwQQYJKwYBBAGgMgEoMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3 Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMCcGA1UdEQQgMB6BHGRhbmllbC5oZWxnZW5iZXJn ZXJAbS1ib3guZGUwCQYDVR0TBAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwQwYD VR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9ncy9nc3BlcnNvbmFsc2ln bjJnMi5jcmwwVQYIKwYBBQUHAQEESTBHMEUGCCsGAQUFBzAChjlodHRwOi8vc2VjdXJlLmdsb2Jh bHNpZ24uY29tL2NhY2VydC9nc3BlcnNvbmFsc2lnbjJnMi5jcnQwHQYDVR0OBBYEFLw0UD+6l35a KnDaePxEP8K35HYZMB8GA1UdIwQYMBaAFD8V0m18L+cxnkMKBqiUbCw7xe5lMA0GCSqGSIb3DQEB BQUAA4IBAQBdVOm7h+E4sRMBbTN1tCIjAEgxmB5U0mdZXcawzEHLJxTrc/5YFBMGX2qPju8cuZV1 4XszMfRBJdlJz1Od+voJggianIhnFEAakCxaa1l/cmJ5EDT6PgZAkXbMB5rU1dhegb35lJJkcFLE pR2tF1V0TfbSe5UZNPYeMQjYsRhs69pfKLoeGm4dSLK7gsPT5EhPd+JPyNSIootOwClMP4CTxIsX QgRI5IDqG2Ku/r2YMMLsqWD11PtAE87t2mgohQ6V1XdWFqGd1V+wN98oPumRRS8bld+1gRA7GVYM nO5MF6p//iHFcy3MVT05ojqgomMt+voH5cFzrHA61z80xaZ6MYIC5zCCAuMCAQEwaTBUMQswCQYD VQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQ ZXJzb25hbFNpZ24gMiBDQSAtIEcyAhEAlmlmwN6lqS/kJp+iNeg8pzAJBgUrDgMCGgUAoIIBUzAY BgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDA0MDkxMjMyNTlaMCMG CSqGSIb3DQEJBDEWBBRWTt0Kcygyy0JbkJmtXJgTR5DtoTB4BgkrBgEEAYI3EAQxazBpMFQxCzAJ BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYDVQQDEyFHbG9iYWxTaWdu IFBlcnNvbmFsU2lnbiAyIENBIC0gRzICEQCWaWbA3qWpL+Qmn6I16DynMHoGCyqGSIb3DQEJEAIL MWugaTBUMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMh R2xvYmFsU2lnbiBQZXJzb25hbFNpZ24gMiBDQSAtIEcyAhEAlmlmwN6lqS/kJp+iNeg8pzANBgkq hkiG9w0BAQEFAASCAQBv5QSdtI20ma9fS1Rl54Xc1+VUmS/llZSSo/nzxRJNYjVlWGXFA6PftdnI RIY5Zaia7cciNa3UqFlvJ4CWoQaHrkrMR4KH0HigQQtgW+HL5GpmgT2PhRmtCA65bgVZLo0K35+H t5xCQVaMUFFWDFphtGhWGpLkuIEmi/hchxX2KFEjevlaJQO5EyKL5i6Z1MXkq7P+jWkrXS/u+2WA nXjeRGApKTLP4afZ3dTI5A/5dkbZqhb7UEOJd+J6lBYeDnf3MePGTZByOAI2Ogsw/jm4CF325lAc YWVdS5XMOfkyxJWOOJorVpKEQBFsoVv0krl2SrZuRAU3dDTmiJHewTPkAAAAAAAA --=-jhmMaWvE9+4Rp/kN8BPH--