[ovirt-devel] oVirt 3.5 test day 1 results

[Nir Soffer]

Hi all,

I tested today [RFE] replace XML-RPC communication (engine-vdsm) with json-rpc based on bidirectional transport

First I upgraded ovirt-3.4 stable engine to ovirt-3.5 - ok
Then I upgraded 4 hosts to latest vdsm - ok

I upgraded 2 data centers to cluster version 3.5:
- 2 Fedora 19 hosts with 30 ISCSI storage domains - ok
- 2 RHEL 6.5 hosts with 45 NFS storage domains - failed
  I had to remove the hosts and the virtual machines to complete
  the upgrade [1]

Then I removed the hosts and added them back (to configure jsonrpc), and
setup one host using jsonrpc and the other using xmlrpc - ok

After moving the hosts to maintenance mode and starting them back, I found
that the host using jsonrpc was stuck in "Unassigned" state [2],[3].

The errors in the vdsm log were not clear enough. After I improving this [4],
I could fix it in one line patch [5].

Finally when I had a working system, I run some sanity tests:
- start/stop vm - ok
- create vm from template - ok
- migrate vms between two hosts concurrenly (one host use xmlrpc, one using json) - ok

Then I tried to test create template from vm, but I had low disk space
on that storage domain. So I tried to extend the domain which would be
useful test as well.

But turns out that you cannot create or edit a block domain when using jsonrpc [6]

Looking at the logs, I found also that shutting down protocol detector fails [7]

Summary:

- upgrade is broken in some cases - critical
- jsonrpc is not ready yet
- jsonrpc needs lot of additional testing - for next test day I suggest one tester
  from each team (virt, storage, networking, sla?) to test jsonrpc with relevant
  flows.

[1] https://bugzilla.redhat.com/1114994
    Cannot edit cluster after upgrade from version 3.4 to 3.5 because cpu type (Intel Haswell) does not match

[2] https://bugzilla.redhat.com/1115033
    StoragePool_disconnect: disconnect() takes exactly 4 arguments

[3] https://bugzilla.redhat.com/1115044
    Host stuck in "Unassinged" state when using jsonrpc and disconnection from pool failed

[4] http://gerrit.ovirt.org/29457 
    bridge: Show more info when method call fail

[5] http://gerrit.ovirt.org/29465
    api: Make remove optional

[6] https://bugzilla.redhat.com/show_bug.cgi?id=1115152
    Cannot edit or create block storage doamin when using jsonrpc

[7] https://bugzilla.redhat.com/1115104
    Shuting down protocol detector fails

Nir