[ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

Fri Jun 24 04:16:23 EDT 2016

After an additional yum clean all && yum update was updated some other rpms.

Something changed.....
My setup has engine storage on gluster, but mounted with NFS.
Now gluster daemon don't automatically start at boot. After starting 
manually gluster the error is the same:

==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-19::ERROR::2016-06-24 
10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) 
Error while serving connection
Traceback (most recent call last):
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle
     data)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch
     .set_storage_domain(client, sd_type, **options)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain
     self._backends[client].connect()
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 400, in connect
     volUUID=volume.volume_uuid
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 245, in _get_volume_path
     volUUID
   File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
     return self.__send(self.__name, args)
   File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
     verbose=self.__verbose
   File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
     return self.single_request(host, handler, request_body, verbose)
   File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
     self.send_content(h, request_body)
   File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
     connection.endheaders(request_body)
   File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
     self._send_output(message_body)
   File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
     self.send(msg)
   File "/usr/lib64/python2.7/httplib.py", line 797, in send
     self.connect()
   File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, 
in connect
     sock = socket.create_connection((self.host, self.port), self.timeout)
   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
     raise err
error: [Errno 101] Network is unreachable

VDSM.log

jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) 
Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0
aborting False
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Following parameters ['type'] were not recogn
ized
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Provided value "2" not defined in DiskType en
um for Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter capacity is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Required property allocType is not provided w
hen calling Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter mtime is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter ctime is not int type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter truesize is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter apparentsize is not uint type
jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) 
Return 'Volume.getInfo' in bridge with {'sta
tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 
'LEAF', 'description': 'hosted-engine.lockspace', 'parent': 
'00000000-0000-00
00-0000-000000000000', 'format': 'RAW', 'image': 
'6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 
'disktype': '2', 'legality': 'LEGAL',
'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 
'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 
'truesize
': '1048576', 'type': 'PREALLOCATED'}
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: (104, 'Connection r
eset by peer')
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled 
close event
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:43,807::betterAsyncore::132::vds.dispatcher::(send) SSL error 
during sending data: (104, 'Connection r
eset by peer')
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:43,959::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: bad write retry
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:43,959::betterAsyncore::154::vds.dispatcher::(log_info) unhandled 
close event
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:47,859::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: (104, 'Connection r
eset by peer')
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:47,860::betterAsyncore::154::vds.dispatcher::(log_info) unhandled 
close event
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:51,725::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: (104, 'Connection r
eset by peer')
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:51,726::betterAsyncore::154::vds.dispatcher::(log_info) unhandled 
close event
Reactor thread::INFO::2016-06-24 
10:10:53,851::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) 
Accepting connection from ::ffff:1
92.168.1.50:48554
Reactor thread::DEBUG::2016-06-24 
10:10:53,860::protocoldetector::85::ProtocolDetector.Detector::(__init__) 
Using required_size=11
Reactor thread::INFO::2016-06-24 
10:10:53,861::protocoldetector::121::ProtocolDetector.Detector::(handle_read) 
Detected protocol stomp from ::ffff:192
.168.1.50:48554
Reactor thread::INFO::2016-06-24 
10:10:53,862::stompreactor::101::Broker.StompAdapter::(_cmd_connect) 
Processing CONNECT request
Reactor thread::DEBUG::2016-06-24 
10:10:53,862::stompreactor::482::protocoldetector.StompDetector::(handle_socket) 
Stomp detected from ('::ffff:192.168.1.50', 48554)

Il 24/06/2016 8.18, Sandro Bonazzola ha scritto:
>
>
> On Thu, Jun 23, 2016 at 11:46 PM, Stefano Danzi <s.danzi at hawai.it 
> <mailto:s.danzi at hawai.it>> wrote:
>
>     Hi!
>     After cleanin metadata yum do an update of vdsm:
>
>     [root at ovirt01 ~]# rpm -qva | grep vdsm
>     vdsm-yajsonrpc-4.18.4.1-0.el7.centos.noarch
>     vdsm-infra-4.18.4.1-0.el7.centos.noarch
>     vdsm-cli-4.18.4.1-0.el7.centos.noarch
>     vdsm-python-4.18.4.1-0.el7.centos.noarch
>     vdsm-hook-vmfex-dev-4.18.4.1-0.el7.centos.noarch
>     vdsm-xmlrpc-4.18.4.1-0.el7.centos.noarch
>     vdsm-4.18.4.1-0.el7.centos.x86_64
>     vdsm-api-4.18.4.1-0.el7.centos.noarch
>     vdsm-gluster-4.18.4.1-0.el7.centos.noarch
>     vdsm-jsonrpc-4.18.4.1-0.el7.centos.noarch
>
>     But this not solve the issue.
>
>     - Host haven't default gateway after a reboot
>     - Self Hosted engine don't start.
>
>
> Martin, Dan, can you please look into this?
> Stefano, can you please share a full sos report from the host?
>
>
>     vdsm.log:
>     https://drive.google.com/file/d/0ByMG4sDqvlZcVEJ5YVI1UWxrdE0/view?usp=sharing
>
>
>
>     Il 2016-06-23 21:41 Sandro Bonazzola ha scritto:
>
>         On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi
>         <s.danzi at hawai.it <mailto:s.danzi at hawai.it>>
>         wrote:
>
>             Hi!
>             I've just upgrade oVirt from 3.6 to 4.0 and I'm not able
>             to start
>             the self hosted engine.
>
>
>         Hi Stefano, can you please try "yum clean metadata" "yum update"
>         again?
>         You should get vdsm 4.18.4.1, please let us know if this solve
>         your
>         issue.
>
>
>
>
> -- 
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com <http://redhat.com>

-- 

Stefano Danzi
Responsabile sistemi informativi

HAWAI ITALIA S.r.l.
Via Forte Garofolo, 16
37057 S. Giovanni Lupatoto Verona Italia

P. IVA 01680700232

tel. +39/045/8266400
fax +39/045/8266401
Web www.hawai.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160624/680b6774/attachment-0001.html>