[ovirt-users] 3.5 to 3.6 upgrade stuck

[Robert Story]

On Thu, 21 Jul 2016 14:43:50 -0400 Robert wrote:
RS> So after some debugging with Simone on irc, we've determined that the issue
RS> is the agent timing out trying to communicate with the broker. The problem
RS> is that we have no idea why.

So more detail attached. The agent is sending:

   MainThread::hosted_engine::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine
  ::(start_monitoring) Processing engine state <ovirt_hosted_engine_ha.agent.states.ReinitializeFSM object at 0x15d8c30> MainThread::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink
  ::(notify) Trying: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec' MainThread::brokerlink::273::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink
  ::(_communicate) Sending request: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec'

Which the broker sees:

Thread-1::util::69::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
     ::(socket_readline) socket_readline in blocking mode
Thread-1::listener::163::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
     ::(handle) Input: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec'

It then refreshes the local config file:

Thread-1::config::251::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
     ::(refresh_local_conf_file) Reading 'broker.conf' from '/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7

Which succeeds:

Thread-1::config::271::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
     ::(refresh_local_conf_file) Writing to '/var/lib/ovirt-hosted-engine-ha/broker.conf'
Thread-1::config::278::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
     ::(refresh_local_conf_file) local conf file was correctly written

And then .... nothing. It just hangs. Nothing more is logged Thread-1.



Robert

-- 
Senior Software Engineer @ Parsons



Robert

-- 
Senior Software Engineer @ Parsons
-------------- next part --------------
A non-text attachment was scrubbed...
Name: agent.log
Type: text/x-log
Size: 1902 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160721/90a140d7/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: broker.log
Type: text/x-log
Size: 3948 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160721/90a140d7/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160721/90a140d7/attachment-0001.sig>