[ovirt-devel] [vdsm] Connection refused when talking to jsonrpc

[Nir Soffer]

On Tue, Nov 8, 2016 at 10:31 PM, Piotr Kliczewski <pkliczew at redhat.com> wrote:
>
>
> On Tue, Nov 8, 2016 at 8:51 PM, Martin Perina <mperina at redhat.com> wrote:
>>
>> Adding Piotr
>>
>> On Tue, Nov 8, 2016 at 5:52 PM, Martin Sivak <msivak at redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> mom-vdsm.service contains:
>>>
>>> Requires=vdsmd.service
>>> After=vdsmd.service
>>>
>>> So when Shira restarted vdsm, mom was also restarted.
>>>
>>> [journalctl --unit vdsmd]
>>> Nov 08 18:25:27 RHEL7.2Server systemd[1]: Stopping Virtual Desktop
>>> Server Manager...
>>> Nov 08 18:25:27 RHEL7.2Server vdsmd_init_common.sh[3053]: vdsm:
>>> Running run_final_hooks
>>> Nov 08 18:25:27 RHEL7.2Server systemd[1]: Starting Virtual Desktop
>>> Server Manager...
>>>
>>> [journalctl --unit mom-vdsm]
>>> Nov 08 18:17:23 RHEL7.2Server systemd[1]: Starting MOM instance
>>> configured for VDSM purposes...
>>> Nov 08 18:25:16 RHEL7.2Server systemd[1]: Stopping MOM instance
>>> configured for VDSM purposes...
>>> Nov 08 18:25:29 RHEL7.2Server systemd[1]: Started MOM instance
>>> configured for VDSM purposes.
>>>
>>>
>>> But mom then immediately failed with:
>>>
>>> 2016-11-08 18:25:08,008 - mom.RPCServer - INFO - ping()
>>> 2016-11-08 18:25:08,010 - mom.RPCServer - INFO - getStatistics()
>>> 2016-11-08 18:25:17,028 - mom.RPCServer - INFO - RPC Server ending
>>> 2016-11-08 18:25:24,705 - mom.GuestManager - INFO - Guest Manager ending
>>> 2016-11-08 18:25:26,575 - mom.HostMonitor - INFO - Host Monitor ending
>>>
>>> 2016-11-08 18:25:29,869 - mom - INFO - MOM starting
>>> 2016-11-08 18:25:29,905 - mom.HostMonitor - INFO - Host Monitor starting
>>> 2016-11-08 18:25:29,905 - mom - INFO - hypervisor interface
>>> vdsmjsonrpcbulk
>>> 2016-11-08 18:25:30,029 - mom.vdsmInterface - ERROR - Cannot connect
>>> to VDSM! [Errno 111] Connection refused
>>> 2016-11-08 18:25:30,030 - mom - ERROR - Failed to initialize MOM threads
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in
>>> run
>>>     hypervisor_iface = self.get_hypervisor_interface()
>>>   File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217,
>>> in get_hypervisor_interface
>>>     return module.instance(self.config)
>>>   File
>>> "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
>>> line 47, in instance
>>>     return JsonRpcVdsmBulkInterface()
>>>   File
>>> "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py",
>>> line 29, in __init__
>>>     super(JsonRpcVdsmBulkInterface, self).__init__()
>>>   File
>>> "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py",
>>> line 43, in __init__
>>>     .orRaise(RuntimeError, 'No connection to VDSM.')
>>>   File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in
>>> orRaise
>>>     raise exception(*args, **kwargs)
>>> RuntimeError: No connection to VDSM.
>>>
>>>
>>> The question here is, how much time does VDSM need to allow jsonrpc to
>>> connect and request a ping and list of VMs?
>>>
>
> It depends on recovery logic in vdsm and it can take quite some time.

Recovery is done in another thread and does not delay startup in any way.
Same for storage init thread.

We can minimize the delay for clients by opening the acceptor first thing
during startup, so clients can connect immediately. The code is already
sending error 99 (recovering or initializing) until both vm recovery and storage
init is done.

Nir