OST's basic suite UI sanity tests optimization
by Marcin Sobczyk
Hi,
_TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more
than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of
Chromium.
During the OST hackathon in Brno this year, I saw an opportunity to
optimize basic UI sanity tests from basic suite.
The way we currently run them, is by setting up a Selenium grid using 3
docker containers, with a dedicated network... that's insanity! (pun
intended).
Let's a look at the running time of '008_basic_ui_sanity.py' scenario
(https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te...):
01:31:50 @ Run test: 008_basic_ui_sanity.py:
01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_',
'^setup\\.py$']
01:31:50 # init:
01:31:50 # init: Success (in 0:00:00)
01:31:50 # start_grid:
01:34:05 # start_grid: Success (in 0:02:15)
01:34:05 # initialize_chrome:
01:34:18 # initialize_chrome: Success (in 0:00:13)
01:34:18 # login:
01:34:27 # login: Success (in 0:00:08)
01:34:27 # left_nav:
01:34:45 # left_nav: Success (in 0:00:18)
01:34:45 # close_driver:
01:34:46 # close_driver: Success (in 0:00:00)
01:34:46 # initialize_firefox:
01:35:02 # initialize_firefox: Success (in 0:00:16)
01:35:02 # login:
01:35:11 # login: Success (in 0:00:08)
01:35:11 # left_nav:
01:35:29 # left_nav: Success (in 0:00:18)
01:35:29 # cleanup:
01:35:36 # cleanup: Success (in 0:00:06)
01:35:36 # Results located at
/dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml
01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
* install 'chromium-headless' package on engine VM
* download 'chromedriver' and 'selenium hub' jar and deploy them in
'/var/opt/' on engine's VM
* run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by
using Lago's ssh
* connect to the Selenium instance running on the engine in
'008_basic_ui_sanity.py'
* make screenshots
This series of patches represent the changes:
https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+sta....
This is the new running time (https://jenkins.ovirt.org/view/oVirt
system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py:
20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_',
'^setup\\.py$']
20:13:26 # init:
20:13:26 # init: Success (in 0:00:00)
20:13:26 # make_screenshots:
20:13:27 * Retrying (Retry(total=2, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111]
Connection refused',)': /wd/hub
20:13:27 * Retrying (Retry(total=1, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111]
Connection refused',)': /wd/hub
20:13:27 * Retrying (Retry(total=0, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111]
Connection refused',)': /wd/hub
20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub ->
http://192.168.201.4:4444/wd/hub/static/resource/hub.html
20:14:02 # make_screenshots: Success (in 0:00:35)
20:14:02 # Results located at
/dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml
20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35)
(The 'NewConnectionErrors' is waiting for Selenium hub to be up and
running, I can silence these later).
And the screenshots are here:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te...
_The pros:_
* we cut the running time by more than 3 minutes
_The cons:_
* we don't get Firefox or Chrome screenshots - we get Chromium
screenshots (although AFAIK, QE has much more Selenium tests which
cover both Firefox and Chrome)
* we polute the engine VM with 'chromium-headless' package and deps
(in total: 'chromium-headless', 'chromium-common', 'flac-libs' and
'minizip'), although we can remove these after the tests
_Some design choices explained:_
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install
'chromium-headless' (and even other browsers) on our Jenkins executors,
but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of
https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the
engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical
Google Chromedriver builds, they contain a single binary, and they work
for us.
_What still needs to be polished with the patches:_
* Currently 'setup_engine_selenium.sh' script downloads each time
'selenium.jar' and 'chromedriver.zip' (even with these downloads we
get much faster set-up times) - we should bake these into the engine
VM image template.
* 'selenium_hub_running' function in 'selenium_on_engine.py' is
hackish - an ability to run an ssh command with a context manager
(and auto-terminate on it exits) should be part of Lago. Can be
refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
2 months, 1 week
Proposing Andrej Krejcir as a maintainer of engine core
by Arik Hadas
Dear engine-core maintainers,
I'd like to propose Andrej Krejcir as a maintainer of oVirt engine backend.
Since Jul 2015, Andrej has been contributing to ovirt-engine, mostly in SLA
areas (scheduling, quota, etc) and recently being more and more involved
also in the VIRT areas. This results in ~200 patches to engine master alone.
Personally, I reviewed some of Andrej's contributions and noticed reviews
made by Andrej to others. I sincerely believe Andrej would be a good
addition to the maintainers' group.
Thanks in advance for your response!
3 months, 2 weeks
mom install error on fc29
by Miguel Duarte de Mora Barroso
Hi,
I'm seeing the following error when I try to install
ovirt-provider-ovn-driver (which requires vdsm):
[MIRROR] mom-0.5.12-0.0.master.fc29.noarch.rpm: Interrupted by header
callback: Server reports Content-Length: 340 but expected size is:
133356
17:13:50 [FAILED] mom-0.5.12-0.0.master.fc29.noarch.rpm: No more
mirrors to try - All mirrors were already tried without success
17:13:50 Error: Error downloading packages:
17:13:50 Cannot download
noarch/mom-0.5.12-0.0.master.fc29.noarch.rpm: All mirrors were tried
Any clue ?.. it can be seen in [0].
Thanks in advance,
Miguel
[0] - https://jenkins.ovirt.org/job/ovirt-provider-ovn_standard-check-patch/230...
3 months, 2 weeks
Getting "Could not fetch dashboard data. Please ensure that data warehouse is properly installed and configured" after upgrading RHV 4.2 to 4.3
by Lynn Dixon
I attempted to upgrade a hosted engine environment that was running RHV 4.2
to RHV 4.3 and everything appeared to have worked fine. Except when I
login to the webui, I get the error in the dashboard that "Could not fetch
dashboard data...."
I see the dwhd service is running, and I noticed that there was a
postgresql upgrade in the engine-setup, but I did not see any errors or
oddness during the upgrade process.
Is there something I am missing?
*Lynn Dixon* | Red Hat Certified Architect #100-006-188
*Sr. Cloud Consultant* | Cloud Management Practice
Google Voice: 423-618-1414
Cell/Text: 423-774-3188
Click here to view my Certification Portfolio <http://red.ht/1XMX2Mi>
3 months, 2 weeks
USB on host support?
by Hetz Ben Hamo
Hi,
I'm in communication with a company that thinks about RHV.
One of the requirements is USB host support (which will be on the nodes,
not on the desktops). I haven't found any way to connect USB to a VM other
then through SPICE, but this method won't help as these dongles won't be on
desktops.
Is this in the road map of oVirt?
Thanks
3 months, 2 weeks
[Ovirt] [CQ weekly status] [28-06-2019]
by Dafna Ron
This mail is to provide the current status of CQ and allow people to review
status before and after the weekend.
Please refer to below colour map for further information on the meaning of
the colours.
*CQ-4.2*: GREEN (#1)
Last failure was on 28-06 for mom due to failed test:
002_bootstrap.add_master_storage_domain. this is a known issue which has a
fix which was probably not added to the upgrade suite. I re-triggered the
change to see if it passes
*CQ-4.3*: GREEN (#2)
Last failure was on 27-06-2019 for project ovirt-web-ui due to test
get_host_device which seemed to be a race. I re-triggered the patch and it
passed.
*CQ-Master:* GREEN (#1)
Last failure was on 28-06-3019 for project ovirt-node-ng-image on
build-artifacts for fc29
There is a ticket opened https://ovirt-jira.atlassian.net/browse/OVIRT-2747
Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found
here:
[1]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-...
[2]
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change...
[3]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
Happy week!
Dafna
-------------------------------------------------------------------------------------------------------------------
COLOUR MAP
Green = job has been passing successfully
** green for more than 3 days may suggest we need a review of our test
coverage
1.
1-3 days GREEN (#1)
2.
4-7 days GREEN (#2)
3.
Over 7 days GREEN (#3)
Yellow = intermittent failures for different projects but no lasting or
current regressions
** intermittent would be a healthy project as we expect a number of
failures during the week
** I will not report any of the solved failures or regressions.
1.
Solved job failures YELLOW (#1)
2.
Solved regressions YELLOW (#2)
Red = job has been failing
** Active Failures. The colour will change based on the amount of time the
project/s has been broken. Only active regressions would be reported.
1.
1-3 days RED (#1)
2.
4-7 days RED (#2)
3.
Over 7 days RED (#3)
3 months, 2 weeks
[VDSM] New random failure on Fedora 29
by Nir Soffer
I had this unrelated failure now, never seen it before.
======================================================================
ERROR: test_event_handler (client_test.VdsmClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py",
line 274, in wrapper
return f(*args, **kwargs)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/client_test.py",
line 220, in test_event_handler
client.Test.sendEvent()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/client.py",
line 294, in _call
raise TimeoutError(method, kwargs, timeout)
TimeoutError: Request Test.sendEvent with args {} timed out after 3 seconds
Build:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/6692//artifact/ch...
Looking at the captured logs, we see this alarming error:
2019-06-24 19:49:58,108 ERROR (Detector thread) [vds.dispatcher]
uncaptured python exception,closing channel
<yajsonrpc.betterAsyncore.Dispatcher ('::1', 34444, 0, 0) at
0x7f74983a5c20> (<type 'exceptions.AttributeError'>:'NoneType' object
has no attribute 'headers'
[/usr/lib64/python2.7/asyncore.py|readwrite|108]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[vdsm/lib/vdsm/protocoldetector.py|handle_read|129]
[vdsm/lib/yajsonrpc/stompserver.py|handle_socket|412]
[vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54]
[vdsm/lib/yajsonrpc/stompserver.py|createListener|379]
[vdsm/lib/yajsonrpc/stompserver.py|StompListener|345]
[vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47]
[vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86]
[vdsm/lib/yajsonrpc/stompserver.py|init|363]
[vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57]
[vdsm/lib/yajsonrpc/stomp.py|set_message_handler|635]
[/usr/lib64/python2.7/asyncore.py|handle_read_event|449]
[vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[vdsm/lib/yajsonrpc/stomp.py|handle_read|411]
[vdsm/lib/yajsonrpc/stomp.py|parse|313]
[vdsm/lib/yajsonrpc/stomp.py|_parse_header|249])
(betterAsyncore:179)
Unhandled exception means we did not implement some handle_error, and we
got the default error handling which give
very unhelpful tracebacks:
https://github.com/python/cpython/blob/6cbff564f00fbe17b5443f6986a44ce116...
So we have 2 issues:
- missing handle_error - fixing this first will help to fix the actual
error bellow
- the actual error: 'NoneType' object has no attribute 'headers', from:
yajsonrpc/stomp.py line 249
Nir
3 months, 3 weeks
yum x dnf on el8 host installation?
by Milan Zamazal
Hi, when I experiment with installing a RHEL 8 host from Engine (master
branch running on RHEL 7), I get the following error in host deploy:
2019-06-25 12:59:48,701+0200 DEBUG otopi.plugins.otopi.network.firewalld firewalld._get_firewalld_cmd_version:116 firewalld version: 0.6.3
2019-06-25 12:59:48,701+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventEnd STAGE customization METHOD otopi.plugins.otopi.network.firewalld.Plugin._customization (None)
2019-06-25 12:59:48,701+0200 DEBUG otopi.context context.dumpEnvironment:731 ENVIRONMENT DUMP - BEGIN
2019-06-25 12:59:48,702+0200 DEBUG otopi.context context.dumpEnvironment:741 ENV NETWORK/firewalldAvailable=bool:'True'
2019-06-25 12:59:48,702+0200 DEBUG otopi.context context.dumpEnvironment:745 ENVIRONMENT DUMP - END
2019-06-25 12:59:48,702+0200 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.otopi.core.config.Plugin._customize1
2019-06-25 12:59:48,702+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventStart STAGE customization METHOD otopi.plugins.otopi.core.config.Plugin._customize1 (None)
2019-06-25 12:59:48,703+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventEnd STAGE customization METHOD otopi.plugins.otopi.core.config.Plugin._customize1 (None)
2019-06-25 12:59:48,703+0200 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_host_deploy.kdump.packages.Plugin._customization
2019-06-25 12:59:48,703+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventStart STAGE customization METHOD otopi.plugins.ovirt_host_deploy.kdump.packages.Plugin._customization (None)
2019-06-25 12:59:48,704+0200 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
File "/tmp/ovirt-AX5hJa1Fcr/pythonlib/otopi/context.py", line 132, in _executeMethod
method['method']()
File "/tmp/ovirt-AX5hJa1Fcr/otopi-plugins/ovirt-host-deploy/kdump/packages.py", line 216, in _customization
self._kexec_tools_version_supported()
File "/tmp/ovirt-AX5hJa1Fcr/otopi-plugins/ovirt-host-deploy/kdump/packages.py", line 143, in _kexec_tools_version_supported
from rpmUtils.miscutils import compareEVR
ModuleNotFoundError: No module named 'rpmUtils'
2019-06-25 12:59:48,705+0200 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Environment customization': No module named 'rpmUtils'
It's apparently because `rpmUtils' module from `yum' package is no
longer present on RHEL 8 (in `yum' or another very easily guessable
package). I'm not sure how el7 x el8 host installation is supposed to
work and whether I can do anything at the moment to proceed a bit
further.
I suppose similar yum x dnf issues will need some attention; not only in
host deploy but everywhere where yum or rpm manipulation is used.
Thanks,
Milan
3 months, 3 weeks
