ovirt-ha-broker issue after upgrade CentOS 7.1 => 7.2 and oVirt 3.6.0 => 3.6.1
Process: 15245 ExecStart=3D/usr/lib/systemd/systemd-ovirt-ha-broker= start (code=3Dexited, status=3D0/SUCCESS)<br /> CGroup: /syste= m.slice/ovirt-ha-broker.service<br /> &n= bsp; =E2=94=94=E2=94=8015259 /usr/bin/python /usr/share/o= virt-hosted-engine-ha/ovirt-ha-broker<br /><br />déc. 17 17:41:46 ov= irt01 systemd[1]: Starting oVirt Hosted Engine High Availability Communicat= ions Broker...<br />déc. 17 17:41:46 ovirt01 systemd[1]: PID 15252 r= ead from file /run/ovirt-hosted-engine-ha/broker.pid does not exist or is a= zombie.</p> <p> </p> <p>[root@ovirt01 ~]# systemctl start ovirt-ha-broker<br />Job for ovi= rt-ha-broker.service failed because a timeout was exceeded. See "systemctl = status ovirt-ha-broker.service" and "journalctl -xe" for details.</p> <p> </p> <p>[root@ovirt01 ~]# tail -f /var/log/ovirt-hosted-engine-ha/broker.log<br = />MainThread::INFO::2015-12-17 17:44:28,562::broker::57::ovirt_hosted_engin= e_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 1.3.3.4 star= ted<br />MainThread::INFO::2015-12-17 17:44:28,588::monitor::40::ovirt_host= ed_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for = submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/brok= er/submonitors<br />MainThread::INFO::2015-12-17 17:44:28,588::monitor::49:= :ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loa= ded submonitor cpu-load<br />MainThread::INFO::2015-12-17 17:44:28,590::mon= itor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submoni= tors) Loaded submonitor cpu-load-no-engine<br />MainThread::INFO::2015-12-1= 7 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor:= :(_discover_submonitors) Loaded submonitor engine-health<br />MainThread::I= NFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.mo= nitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free<br />Main= Thread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha= =2Ebroker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-lo= ad<br />MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hoste= d_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonit= or mgmt-bridge<br />MainThread::INFO::2015-12-17 17:44:28,592::monitor::49:= :ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loa= ded submonitor ping<br />MainThread::INFO::2015-12-17 17:44:28,592::monitor= ::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors= ) Loaded submonitor cpu-load<br />MainThread::INFO::2015-12-17 17:44:28,593= ::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_su= bmonitors) Loaded submonitor cpu-load-no-engine<br />MainThread::INFO::2015= -12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Mon= itor::(_discover_submonitors) Loaded submonitor engine-health<br />MainThre= ad::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.brok= er.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free<br /= MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engin= e_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-= load<br />MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hos= ted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submon= itor mgmt-bridge<br />MainThread::INFO::2015-12-17 17:44:28,594::monitor::4= 9::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) L= oaded submonitor ping<br />MainThread::INFO::2015-12-17 17:44:28,594::monit= or::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonito= rs) Finished loading submonitors<br />MainThread::INFO::2015-12-17 17:44:28= ,595::listener::41::ovirt_hosted_engine_ha.broker.listener.Listener::(__ini= t__) Initializing SocketServer<br />MainThread::INFO::2015-12-17 17:44:28,5= 95::listener::56::ovirt_hosted_engine_ha.broker.listener.Listener::(__init_= _) SocketServer ready<br />MainThread::INFO::2015-12-17 17:45:59,215::broke= r::114::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Server shutting = down</p> <p> </p> <p>[root@ovirt01 ~]# tail -f /var/log/vdsm/vdsm.log<br />Reactor thread::IN= FO::2015-12-17 17:48:19,800::protocoldetector::72::ProtocolDetector.Accepto= rImpl::(handle_accept) Accepting connection from 127.0.0.1:44498<br />React= or thread::DEBUG::2015-12-17 17:48:19,808::protocoldetector::82::ProtocolDe= tector.Detector::(__init__) Using required_size=3D11<br />Reactor thread::I= NFO::2015-12-17 17:48:19,808::protocoldetector::118::ProtocolDetector.Detec= tor::(handle_read) Detected protocol xml from 127.0.0.1:44498<br />Reactor =
--=_f943a51baf446e98d98ebf3fc695b0ae Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Hi, I upgrade my 3 servers to CentOS 7.2 and oVirt 3.6.1, the oVirt engine works fine and my first host upgraded too. However, my second and third host have a ovirt-ha-broker issue and is doesn't start. When i try to start the broker service, it failed with timeout. Here my logs of my second upgraded server : [root@ovirt01 ~]# systemctl status ovirt-ha-broker ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor preset: disabled) Active: activating (start) since jeu. 2015-12-17 17:41:46 GFT; 1min 3s ago Process: 15245 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS) CGroup: /system.slice/ovirt-ha-broker.service └─15259 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker déc. 17 17:41:46 ovirt01 systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker... déc. 17 17:41:46 ovirt01 systemd[1]: PID 15252 read from file /run/ovirt-hosted-engine-ha/broker.pid does not exist or is a zombie. [root@ovirt01 ~]# systemctl start ovirt-ha-broker Job for ovirt-ha-broker.service failed because a timeout was exceeded. See "systemctl status ovirt-ha-broker.service" and "journalctl -xe" for details. [root@ovirt01 ~]# tail -f /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2015-12-17 17:44:28,562::broker::57::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 1.3.3.4 started MainThread::INFO::2015-12-17 17:44:28,588::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors MainThread::INFO::2015-12-17 17:44:28,588::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2015-12-17 17:44:28,590::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2015-12-17 17:44:28,592::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2015-12-17 17:44:28,592::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2015-12-17 17:44:28,594::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::INFO::2015-12-17 17:44:28,595::listener::41::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__) Initializing SocketServer MainThread::INFO::2015-12-17 17:44:28,595::listener::56::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__) SocketServer ready MainThread::INFO::2015-12-17 17:45:59,215::broker::114::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Server shutting down [root@ovirt01 ~]# tail -f /var/log/vdsm/vdsm.log Reactor thread::INFO::2015-12-17 17:48:19,800::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 127.0.0.1:44498 Reactor thread::DEBUG::2015-12-17 17:48:19,808::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2015-12-17 17:48:19,808::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from 127.0.0.1:44498 Reactor thread::DEBUG::2015-12-17 17:48:19,809::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 44498) BindingXMLRPC::INFO::2015-12-17 17:48:19,809::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for 127.0.0.1:44498 Thread-185::INFO::2015-12-17 17:48:19,810::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:44498 started Thread-185::INFO::2015-12-17 17:48:19,811::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:44498 stopped Cordialement, Florent BELLO --=_f943a51baf446e98d98ebf3fc695b0ae Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"> <html><body> <p>Hi,</p> <p>I upgrade my 3 servers to CentOS 7.2 and oVirt 3.6.1, the oVirt engine w= orks fine and my first host upgraded too. However, my second and third host= have a ovirt-ha-broker issue and is doesn't start.<br />When i try to star= t the broker service, it failed with timeout.</p> <p>Here my logs of my second upgraded server :</p> <p>[root@ovirt01 ~]# systemctl status ovirt-ha-broker<br />=E2=97=8F = ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communicati= ons Broker<br /> Loaded: loaded (/usr/lib/systemd/system/ovirt-= ha-broker.service; enabled; vendor preset: disabled)<br /> Acti= ve: activating (start) since jeu. 2015-12-17 17:41:46 GFT; 1min 3s ago<br /= thread::DEBUG::2015-12-17 17:48:19,809::bindingxmlrpc::1297::XmlDetector::(= handle_socket) xml over http detected from ('127.0.0.1', 44498)<br />Bindin= gXMLRPC::INFO::2015-12-17 17:48:19,809::xmlrpc::73::vds.XMLRPCServer::(hand= le_request) Starting request handler for 127.0.0.1:44498<br />Thread-185::I= NFO::2015-12-17 17:48:19,810::xmlrpc::84::vds.XMLRPCServer::(_process_reque= sts) Request handler for 127.0.0.1:44498 started<br />Thread-185::INFO::201= 5-12-17 17:48:19,811::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Req= uest handler for 127.0.0.1:44498 stopped</p> <div>Cordialement,<br /> <p>Florent BELLO</p> </div> </body></html> --=_f943a51baf446e98d98ebf3fc695b0ae--
On Thu, Dec 17, 2015 at 9:49 PM, Bello Florent < florent.bello@ville-kourou.fr> wrote:
Hi,
I upgrade my 3 servers to CentOS 7.2 and oVirt 3.6.1, the oVirt engine works fine and my first host upgraded too. However, my second and third host have a ovirt-ha-broker issue and is doesn't start. When i try to start the broker service, it failed with timeout.
Here my logs of my second upgraded server :
[root@ovirt01 ~]# systemctl status ovirt-ha-broker ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor preset: disabled) Active: activating (start) since jeu. 2015-12-17 17:41:46 GFT; 1min 3s ago Process: 15245 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS) CGroup: /system.slice/ovirt-ha-broker.service └─15259 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker
déc. 17 17:41:46 ovirt01 systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker... déc. 17 17:41:46 ovirt01 systemd[1]: PID 15252 read from file /run/ovirt-hosted-engine-ha/broker.pid does not exist or is a zombie.
[root@ovirt01 ~]# systemctl start ovirt-ha-broker Job for ovirt-ha-broker.service failed because a timeout was exceeded. See "systemctl status ovirt-ha-broker.service" and "journalctl -xe" for details.
Thanks for the report Florent, just today I experimented it too: we are checking if it's related to this last minute patch from this morning: https://gerrit.ovirt.org/#/c/50662/ Adding Fabian and Martin.
[root@ovirt01 ~]# tail -f /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2015-12-17 17:44:28,562::broker::57::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 1.3.3.4 started MainThread::INFO::2015-12-17 17:44:28,588::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors MainThread::INFO::2015-12-17 17:44:28,588::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2015-12-17 17:44:28,590::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2015-12-17 17:44:28,591::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2015-12-17 17:44:28,592::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2015-12-17 17:44:28,592::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2015-12-17 17:44:28,593::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2015-12-17 17:44:28,594::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2015-12-17 17:44:28,594::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::INFO::2015-12-17 17:44:28,595::listener::41::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__) Initializing SocketServer MainThread::INFO::2015-12-17 17:44:28,595::listener::56::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__) SocketServer ready MainThread::INFO::2015-12-17 17:45:59,215::broker::114::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Server shutting down
[root@ovirt01 ~]# tail -f /var/log/vdsm/vdsm.log Reactor thread::INFO::2015-12-17 17:48:19,800::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from 127.0.0.1:44498 Reactor thread::DEBUG::2015-12-17 17:48:19,808::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2015-12-17 17:48:19,808::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from 127.0.0.1:44498 Reactor thread::DEBUG::2015-12-17 17:48:19,809::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 44498) BindingXMLRPC::INFO::2015-12-17 17:48:19,809::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for 127.0.0.1:44498 Thread-185::INFO::2015-12-17 17:48:19,810::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:44498 started Thread-185::INFO::2015-12-17 17:48:19,811::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for 127.0.0.1:44498 stopped Cordialement,
Florent BELLO
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Bello Florent -
Simone Tiraboschi