[ovirt-devel] oVirt 3.5 2 nd test day report

[Nir Soffer]

Hi all,

I checked again [RFE] replace XML-RPC communication (engine-vdsm) with json-rpc
based on bidirectional transport

I removed ovirt-release package from al 5 hosts, installed the ovirt-release3.5
package, and yum-updated all hosts (RHEL 6.5, Fedora 19, Fedora 20) (OK)

I removed ovirt-engine nightly and installed the test day version from the
ovirt-release35 repo. The installation was fine, and I could see all my data
centers.

When trying to connect to the hosts, all hosts were non responsive because of
SSL error in vdsm. I had to reinstall them from the engine - seems that
removing ovirt engine cause engine to create new certificates after
installation, invalidating existing hosts [1].

While trying to setup the system, I found [2]. Trying to take a screenshot, I
bumped into some annoying ui issues [3], [4], [5], [6], and [7].

I started with testing ISCSI data center with 30 storage domains and 2 hosts
running Fedora 19. I removed the hosts and added them back one host using
jsonrpc and the other using xmlrpc.

Both hosts added to engine without any issue, and one of them selected as the
spm.  I switched spm role between hosts several times. One time the hosts
switch to spm and then the other host became the spm again. I could not
reproduce this again.

Then I removed the hosts and added them back, this time both using jsonrpc. One
host was added successfully and was up, but it never became the spm, although
all storage domain were up.

The other host remained in "Initializing" state. After few minutes, the host
became non-operational. Activating the hosts made it "Unsigned". After few
minutes the host became non-operational again and I moved it into maintenance.
After activating the host, it was finally up. Both host were up but on host was
contending and there was no spm.

After putting both hosts to maintenance, all storage domain in the data center
remained up(!) - without any host in the data center - this seems like a useful
feature but I wonder how engine can tell that the storage domain are up without
any host monitoring the domains.

I removed the hosts again and added them back using xmlrpc. After both of them
came up, there was still no spm, and all storage domain were up

After putting the hosts to maintenance and restarting ovirt engine, all storage
domains became down. Looks like engine got into invalid state.

I removed both hosts and added back using jsonrpc again. This time both came
up, one become the spm, and I could switch spm role between them successfully
several times.

Having no spm in 3.5 is a little bit too early [8].

Trying to create or edit DATA/ISCSI domain failed like in test day 1 - seems
that [9] is not fixed yet. This time getDeviceList does not fail with exception
in engine side, but we don't see any trace for the reply. This looks like a
regression from the first test day.

Looking in engine.log, we can see that jsonrpc related logs add unwanted
newlines and unexpected characters [10].

Looking in vdsm logs, we have too much noise related to how jsonrpc server is
implemented. These logs should be eliminated - the operation of the server must
be verified using unit tests, not by looking at the logs [11].

This is similar to ioprocess excessive logging [12], which is not fixed yet in
this version. I added logs to the exiting bug.

Summary: unusable.

[1] https://bugzilla.redhat.com/1124737
    Removing ovirt-engine and installing it again invalidate all hosts certificates

[2] https://bugzilla.redhat.com/1124584
    Same disk appears multiple times (30!) in the Disks tab

[3] https://bugzilla.redhat.com/1124534
    Cannot resize columns in main tabs

[4] https://bugzilla.redhat.com/1124546
    Resized columns in main tabs do not save their user configured size

[5] https://bugzilla.redhat.com/1124580
    Missing title (or icon) for column 4 (vms?) in Disks tab

[6] https://bugzilla.redhat.com/1124587
    Missing margin in column 0 of main tabs

[7] https://bugzilla.redhat.com/1124593
    Same cluster appears twice in the clusters table with different info

[8] https://bugzilla.redhat.com/1124751
    Early removal of the spm when using jsonrpc

[9] https://bugzilla.redhat.com/1115152
    Cannot edit or create block storage doamin when using jsonrpc

[10] https://bugzilla.redhat.com/1124759
     JSONRPC logs add unwanted newlines and unexpected characters

[11] https://bugzilla.redhat.com/1124764
     extreme amount of irrelevant jsonrpc related log in vdsm log

[12] https://bugzilla.redhat.com/1114908
     Excessive ioprocess logging in vdsm.log

Nir