Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

This is a multi-part message in MIME format. --------------FAB04AB2FA5A32FE906AB6B9 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit After an additional yum clean all && yum update was updated some other rpms. Something changed..... My setup has engine storage on gluster, but mounted with NFS. Now gluster daemon don't automatically start at boot. After starting manually gluster the error is the same: ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-19::ERROR::2016-06-24 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable VDSM.log jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0 aborting False jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Following parameters ['type'] were not recogn ized jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Provided value "2" not defined in DiskType en um for Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter capacity is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Required property allocType is not provided w hen calling Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter mtime is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter ctime is not int type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter truesize is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter apparentsize is not uint type jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Volume.getInfo' in bridge with {'sta tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': '00000000-0000-00 00-0000-000000000000', 'format': 'RAW', 'image': '6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 'truesize ': '1048576', 'type': 'PREALLOCATED'} JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,807::betterAsyncore::132::vds.dispatcher::(send) SSL error during sending data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,959::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: bad write retry JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:43,959::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:47,859::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:47,860::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:51,725::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:51,726::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event Reactor thread::INFO::2016-06-24 10:10:53,851::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from ::ffff:1 92.168.1.50:48554 Reactor thread::DEBUG::2016-06-24 10:10:53,860::protocoldetector::85::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2016-06-24 10:10:53,861::protocoldetector::121::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::ffff:192 .168.1.50:48554 Reactor thread::INFO::2016-06-24 10:10:53,862::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2016-06-24 10:10:53,862::stompreactor::482::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::ffff:192.168.1.50', 48554) Il 24/06/2016 8.18, Sandro Bonazzola ha scritto: -- Stefano Danzi Responsabile sistemi informativi HAWAI ITALIA S.r.l. Via Forte Garofolo, 16 37057 S. Giovanni Lupatoto Verona Italia P. IVA 01680700232 tel. +39/045/8266400 fax +39/045/8266401 Web www.hawai.it --------------FAB04AB2FA5A32FE906AB6B9 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> After an additional yum clean all &amp;&amp; yum update was updated some other rpms.<br> <br> Something changed.....<br> My setup has engine storage on gluster, but mounted with NFS.<br> Now gluster daemon don't automatically start at boot. After starting manually gluster the error is the same:<br> <br> ==&gt; /var/log/ovirt-hosted-engine-ha/broker.log &lt;==<br> Thread-19::ERROR::2016-06-24 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection<br> Traceback (most recent call last):<br>   File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle<br>     data)<br>   File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch<br>     .set_storage_domain(client, sd_type, **options)<br>   File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain<br>     self._backends[client].connect()<br>   File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect<br>     volUUID=volume.volume_uuid<br>   File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path<br>     volUUID<br>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__<br>     return self.__send(self.__name, args)<br>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request<br>     verbose=self.__verbose<br>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request<br>     return self.single_request(host, handler, request_body, verbose)<br>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request<br>     self.send_content(h, request_body)<br>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content<br>     connection.endheaders(request_body)<br>   File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders<br>     self._send_output(message_body)<br>   File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output<br>     self.send(msg)<br>   File "/usr/lib64/python2.7/httplib.py", line 797, in send<br>     self.connect()<br>   File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect<br>     sock = socket.create_connection((self.host, self.port), self.timeout)<br>   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection<br>     raise err<br> error: [Errno 101] Network is unreachable<br> <br> <br> VDSM.log<br> <br> jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0<br> aborting False<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Following parameters ['type'] were not recogn<br> ized<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Provided value "2" not defined in DiskType en<br> um for Volume.getInfo<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter capacity is not uint type<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Required property allocType is not provided w<br> hen calling Volume.getInfo<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter mtime is not uint type<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter ctime is not int type<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter truesize is not uint type<br> jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter apparentsize is not uint type<br> jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Volume.getInfo' in bridge with {'sta<br> tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': '00000000-0000-00<br> 00-0000-000000000000', 'format': 'RAW', 'image': '6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 'disktype': '2', 'legality': 'LEGAL',<br> 'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 'truesize<br> ': '1048576', 'type': 'PREALLOCATED'}<br> JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r<br> eset by peer')<br> JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event<br> JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,807::betterAsyncore::132::vds.dispatcher::(send) SSL error during sending data: (104, 'Connection r<br> eset by peer')<br> JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,959::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: bad write retry<br> JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:43,959::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event<br> JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:47,859::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r<br> eset by peer')<br> JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:47,860::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event<br> JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:51,725::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r<br> eset by peer')<br> JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:51,726::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event<br> Reactor thread::INFO::2016-06-24 10:10:53,851::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from ::ffff:1<br> 92.168.1.50:48554<br> Reactor thread::DEBUG::2016-06-24 10:10:53,860::protocoldetector::85::ProtocolDetector.Detector::(__init__) Using required_size=11<br> Reactor thread::INFO::2016-06-24 10:10:53,861::protocoldetector::121::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::ffff:192<br> .168.1.50:48554<br> Reactor thread::INFO::2016-06-24 10:10:53,862::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request<br> Reactor thread::DEBUG::2016-06-24 10:10:53,862::stompreactor::482::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::ffff:192.168.1.50', 48554)<br> <br> <br> <br> <div class="moz-cite-prefix">Il 24/06/2016 8.18, Sandro Bonazzola ha scritto:<br> </div> <blockquote cite="mid:CAPQRNTnR-G=+gmG1YpuNvSuYRe_sgtyu-R=fyzX3kOa-MeiqMw@mail.gmail.com" type="cite"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Thu, Jun 23, 2016 at 11:46 PM, Stefano Danzi <span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:s.danzi@hawai.it" target=&quot;_blank&quot;&gt;s.danzi(a)hawai.it&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi!<br> After cleanin metadata yum do an update of vdsm:<br> <br> [root@ovirt01 ~]# rpm -qva | grep vdsm<br> vdsm-yajsonrpc-4.18.4.1-0.el7.centos.noarch<br> vdsm-infra-4.18.4.1-0.el7.centos.noarch<br> vdsm-cli-4.18.4.1-0.el7.centos.noarch<br> vdsm-python-4.18.4.1-0.el7.centos.noarch<br> vdsm-hook-vmfex-dev-4.18.4.1-0.el7.centos.noarch<br> vdsm-xmlrpc-4.18.4.1-0.el7.centos.noarch<br> vdsm-4.18.4.1-0.el7.centos.x86_64<br> vdsm-api-4.18.4.1-0.el7.centos.noarch<br> vdsm-gluster-4.18.4.1-0.el7.centos.noarch<br> vdsm-jsonrpc-4.18.4.1-0.el7.centos.noarch<br> <br> But this not solve the issue.<br> <br> - Host haven't default gateway after a reboot<br> - Self Hosted engine don't start.<br> </blockquote> <div><br> </div> <div>Martin, Dan, can you please look into this?</div> <div>Stefano, can you please share a full sos report from the host?</div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> vdsm.log:<br> <a moz-do-not-send="true" href="https://drive.google.com/file/d/0ByMG4sDqvlZcVEJ5YVI1UWxrdE0/v... rel="noreferrer" target="_blank">https://drive.google.com/file/d/0ByMG4sDqvlZ... <div class="HOEnZb"> <div class="h5"><br> <br> Il 2016-06-23 21:41 Sandro Bonazzola ha scritto:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi &lt;<a moz-do-not-send="true" href="mailto:s.danzi@hawai.it" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:s.danzi@hawai.it">s.danzi@hawai.it</a></a>&gt;<br> wrote:<br> <br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi!<br> I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start<br> the self hosted engine.<br> </blockquote> <br> Hi Stefano, can you please try "yum clean metadata" "yum update"<br> again?<br> You should get vdsm 4.18.4.1, please let us know if this solve your<br> issue.<br> </blockquote> </div> </div> </blockquote> </div> <br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail_signature" data-smartmail="gmail_signature"> <div dir="ltr"> <div> <div dir="ltr">Sandro Bonazzola<br> Better technology. Faster innovation. Powered by community collaboration.<br> See how it works at <a moz-do-not-send="true" href="http://redhat.com" target="_blank">redhat.com</a><br> </div> </div> </div> </div> </div> </div> </blockquote> <br> <pre class="moz-signature" cols="72">-- Stefano Danzi Responsabile sistemi informativi HAWAI ITALIA S.r.l. Via Forte Garofolo, 16 37057 S. Giovanni Lupatoto Verona Italia P. IVA 01680700232 tel. +39/045/8266400 fax +39/045/8266401 Web <a class="moz-txt-link-abbreviated" href="http://www.hawai.it">www.hawai.it</a> </pre> </body> </html> --------------FAB04AB2FA5A32FE906AB6B9--