[ovirt-devel] Testday: json rpc

Wed Jul 30 08:15:36 EDT 2014

Hi fellow oVirters,

On this test day I picked JSON RPC for testing:

My initial environment consisted of a hosted engine setup with three hosts.

Upgrading
===========
The first task, then, was to upgrade everything to 3.5.

My guide was http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine

One issue that I encountered was that after setting the 3.5 pre-release yum
repo on the engine VM and doing yum update, when doing
    engine-setup
To make it handle the upgrade, that failed complaining about missing a package
called patternfly1. I remembered that on the list Greg Sheremeta mentioned a
copr repository for it, so I went ahead, installed it and repeated the
    engine-setup

This time it succeeded, however I feel like patternfly1 should probably be an
ovirt-engine-3.5 dependency and it should be in the ovirt 3.5 repository.

I also noticed that after upgrading the hosts the amount of free system memory
is much lower, while the VMs continue to run fine. I opened:
    https://bugzilla.redhat.com/1124451

Another thing that happened was that restarting ovirt-ha-agent and
ovirt-ha-broker using systemd fails silently and it says that they were killed.
I only got it to work again by doing:

    /lib/systemd/systemd-ovirt-ha-broker restart
    /lib/systemd/systemd-ovirt-ha-agent restart

However, and obviously, doing it like that escapes from the advisable control
of systemd. Another bad thing is that after doing all this we get that the
current host (which has the engine running displays):

    --== Host 2 status ==--

    Status up-to-date                  : False
    Hostname                           : 10.34.63.180
    Host ID                            : 2
    Engine status                      : unknown stale-data
    Score                              : 2000
    Local maintenance                  : False
    Host timestamp                     : 1406640293
    Extra metadata (valid at timestamp):
         metadata_parse_version=1
         metadata_feature_version=1
         timestamp=1406640293 (Tue Jul 29 15:24:53 2014)
         host-id=2
         score=2000
         maintenance=False
         bridge=True
         cpu-load=0.0856
         engine-health={"health": "good", "vm": "up", "detail": "up"}
         gateway=True
         mem-free=3192

Why did the engine status go to unknown stale-data? (it also happened for the
other two hosts in the setup.

Changing hosts to use JSON RPC
===============================

After talking with Piotr and trying it with the webadmin UI, I found out that
there is no direct way to update a host's settings to use json rpc as its
connectivity mechanism with the engine. This constitutes a usabiltiy bug
which I filed:

    https://bugzilla.redhat.com/1124442

It is presumable that users will want to move to the newer and better RPC
mechanism and they should be able to do so by merely putting the host in
maintenance and ticking some checkbox in the 'edit host' dialog.

I did the workaround of removing the host and adding it again and that worked,
the host went up.

Network operations via JSON RPC
================================

After the host went up, I decided to send a setupNetworks command. The
operation worked out fine, but unfortunately we have a very serious gap that
makes it impossible for me to use jsonrpc for network operations/development.

**Logging**

When doing a network operation with xmlrpc we'd get the following in vdsm.log
    Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4]
    Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}

As you can see, we get the bare minimum logging one could ask for, an entry
with the command called and the data it received and another entry with the
return result data.

Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search
for "setupNetworks" the only thing I get is:
    Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge

And if I search for the data received, like 'STP', there is nothing whatsoever.
As I said, unless this is fixed and we get entries with the same amount of data
as before, it can't be used in production nor in development.

    https://bugzilla.redhat.com/1124813

TL;DR:
- lack of usability upgrading an environment to use jsonrpc
  https://bugzilla.redhat.com/1124442
- Failure on step 7 of upgrade steps:
  https://bugzilla.redhat.com/1124826
- JSON RPC logging excessive but insuficient for network call debugging
  https://bugzilla.redhat.com/1124813