OST's basic suite UI sanity tests optimization
Hi, _TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario (https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...): 01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45) Starting the Selenium grid takes 2:15 out of 3:35 of total running time! I've investigated a lot of approaches and came up with something like this: * install 'chromium-headless' package on engine VM * download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM * run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh * connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' * make screenshots This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...). This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/): 20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... _The pros:_ * we cut the running time by more than 3 minutes _The cons:_ * we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) * we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests _Some design choices explained:_ Q: Why engine VM? A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot. Q: Why Chromium? A: Because it has a separate 'headless' package. Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds? A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us. _What still needs to be polished with the patches:_ * Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. * 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored. Questions, comments, reviews are welcome. Regards, Marcin
Hi, Thanks for trying to improve the tests! I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable. Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs? Best wishes, Greg On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
On Mon, Mar 4, 2019 at 8:08 PM Greg Sheremeta <gshereme@redhat.com> wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
Why are we running UI tests on OST? This is something that should run in the project tests, not in OST. You an have your own UI suite if you cannot run these tests in the CI, but why delay every run for testing compatibility with various browsers? Nir
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv. This approach has 2 big flaws: * it requires quite a lot of resources for the grid to always be there for us * it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI. Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
_TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium.
During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario (https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
* install 'chromium-headless' package on engine VM * download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM * run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh * connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' * make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...). This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35)
(The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
_The pros:_
* we cut the running time by more than 3 minutes
_The cons:_
* we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) * we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
_Some design choices explained:_
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
_What still needs to be polished with the patches:_
* Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. * 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave@ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox. If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them. Greg Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
-- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
Marcin, It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.] @Barak Korren <bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that? docker.io/selenium/node-chrome-debug 3.9.1-actinium 327adc897d23 13 months ago *904 MB* docker.io/selenium/node-firefox-debug 3.9.1-actinium 88649b420bd5 13 months ago *814 MB* Greg On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
-- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
Greg, that's a great finding and a very good starting point. If we want to stick with docker images and Firefox/Chrome testing, I still have some ideas, that would shorten the running time even more: * we do something like this: log("waiting %s sec for grid to initialize..." % GRID_STARTUP_DELAY) time.sleep(GRID_STARTUP_DELAY) this is very inefficient. We can change that to something like I wrote here (_wait_for_selenium_hub): https://gerrit.ovirt.org/#/c/98135/2/common/test-scenarios-files/selenium/na... This function probably needs some improvement (i.e. urllib3 spits out warnings on an unsuccessful connection attempt, so they would need to be silenced), but that's a far better approach than a simple sleep. * parallelize running Firefox and Chrome tests - there's no reason not to run them both at the same time. There's something called VectorThread in lago.utils. A simple example of usage can be found in '004_basic_sanity.py:955' (disk_operations function). This would have a nice side effect of getting rid of the ugly global ovirt_driver - each thread would have it's own. * maybe not a running-time improvement, but I think https://gerrit.ovirt.org/#/c/98127/ is still relevant - the way we call save_screenshot is ugly and much too verbose Right now, I have to switch my focus to some important stuff in VDSM - the OST patches were a continuation of a hackathon effort and something like a "side-project" ;) Still, I don't want the tread to die. I think there's a lot of room for improvements. I can rebase/improve some of my patches if you find them useful. Please keep me posted with your efforts! Regards, Marcin On 3/7/19 11:10 PM, Greg Sheremeta wrote:
Marcin,
It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.]
@Barak Korren <mailto:bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that?
docker.io/selenium/node-chrome-debug <http://docker.io/selenium/node-chrome-debug> 3.9.1-actinium 327adc897d23 13 months ago *904 MB* docker.io/selenium/node-firefox-debug <http://docker.io/selenium/node-firefox-debug> 3.9.1-actinium 88649b420bd5 13 months ago *814 MB*
Greg
On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com <mailto:gshereme@redhat.com>> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
* it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
* it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
_TL; DR_ Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium.
During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario (https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
* install 'chromium-headless' package on engine VM * download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM * run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh * connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' * make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...). This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35)
(The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
_The pros:_
* we cut the running time by more than 3 minutes
_The cons:_
* we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) * we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
_Some design choices explained:_
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
_What still needs to be polished with the patches:_
* Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. * 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave@ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com <mailto:gshereme@redhat.com> IRC: gshereme
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com <mailto:gshereme@redhat.com> IRC: gshereme
On Fri, Mar 8, 2019 at 12:36 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Greg,
that's a great finding and a very good starting point.
If we want to stick with docker images and Firefox/Chrome testing, I still have some ideas, that would shorten the running time even more:
- we do something like this:
log("waiting %s sec for grid to initialize..." % GRID_STARTUP_DELAY) time.sleep(GRID_STARTUP_DELAY)
this is very inefficient. We can change that to something like I wrote here (_wait_for_selenium_hub):
https://gerrit.ovirt.org/#/c/98135/2/common/test-scenarios-files/selenium/na...
This function probably needs some improvement (i.e. urllib3 spits out warnings on an unsuccessful connection attempt, so they would need to be silenced), but that's a far better approach than a simple sleep.
- parallelize running Firefox and Chrome tests - there's no reason not to run them both at the same time. There's something called VectorThread in lago.utils. A simple example of usage can be found in '004_basic_sanity.py:955' (disk_operations function). This would have a nice side effect of getting rid of the ugly global ovirt_driver - each thread would have it's own.
- maybe not a running-time improvement, but I think https://gerrit.ovirt.org/#/c/98127/ is still relevant - the way we call save_screenshot is ugly and much too verbose
Right now, I have to switch my focus to some important stuff in VDSM - the OST patches were a continuation of a hackathon effort and something like a "side-project" ;) Still, I don't want the tread to die. I think there's a lot of room for improvements. I can rebase/improve some of my patches if you find them useful. Please keep me posted with your efforts!
Regards, Marcin On 3/7/19 11:10 PM, Greg Sheremeta wrote:
Marcin,
It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.]
@Barak Korren <bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that?
Gal/Gallit/Barak, so is there any way how to store those docker containers within image of lago VM which runs OST tests?
docker.io/selenium/node-chrome-debug 3.9.1-actinium 327adc897d23 13 months ago *904 MB* docker.io/selenium/node-firefox-debug 3.9.1-actinium 88649b420bd5 13 months ago *814 MB*
Greg
On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/VH52GVG6V5SO34...
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
On Tue, 12 Mar 2019 at 10:20, Martin Perina <mperina@redhat.com> wrote:
On Fri, Mar 8, 2019 at 12:36 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Greg,
that's a great finding and a very good starting point.
If we want to stick with docker images and Firefox/Chrome testing, I still have some ideas, that would shorten the running time even more:
- we do something like this:
log("waiting %s sec for grid to initialize..." % GRID_STARTUP_DELAY) time.sleep(GRID_STARTUP_DELAY)
this is very inefficient. We can change that to something like I wrote here (_wait_for_selenium_hub):
https://gerrit.ovirt.org/#/c/98135/2/common/test-scenarios-files/selenium/na...
This function probably needs some improvement (i.e. urllib3 spits out warnings on an unsuccessful connection attempt, so they would need to be silenced), but that's a far better approach than a simple sleep.
- parallelize running Firefox and Chrome tests - there's no reason not to run them both at the same time. There's something called VectorThread in lago.utils. A simple example of usage can be found in '004_basic_sanity.py:955' (disk_operations function). This would have a nice side effect of getting rid of the ugly global ovirt_driver - each thread would have it's own.
- maybe not a running-time improvement, but I think https://gerrit.ovirt.org/#/c/98127/ is still relevant - the way we call save_screenshot is ugly and much too verbose
Right now, I have to switch my focus to some important stuff in VDSM - the OST patches were a continuation of a hackathon effort and something like a "side-project" ;) Still, I don't want the tread to die. I think there's a lot of room for improvements. I can rebase/improve some of my patches if you find them useful. Please keep me posted with your efforts!
Regards, Marcin On 3/7/19 11:10 PM, Greg Sheremeta wrote:
Marcin,
It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.]
@Barak Korren <bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that?
Gal/Gallit/Barak, so is there any way how to store those docker containers within image of lago VM which runs OST tests?
Reviving this now since we have some containerization support for CI.
Can we push this forward?
docker.io/selenium/node-chrome-debug 3.9.1-actinium 327adc897d23
13 months ago *904 MB* docker.io/selenium/node-firefox-debug 3.9.1-actinium 88649b420bd5 13 months ago *814 MB*
Greg
On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/VH52GVG6V5SO34...
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TP3EGSE65SNZXG...
On Wed, Aug 7, 2019 at 1:24 PM Doron Fediuck <dfediuck@redhat.com> wrote:
On Tue, 12 Mar 2019 at 10:20, Martin Perina <mperina@redhat.com> wrote:
On Fri, Mar 8, 2019 at 12:36 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Greg,
that's a great finding and a very good starting point.
If we want to stick with docker images and Firefox/Chrome testing, I still have some ideas, that would shorten the running time even more:
- we do something like this:
log("waiting %s sec for grid to initialize..." % GRID_STARTUP_DELAY) time.sleep(GRID_STARTUP_DELAY)
this is very inefficient. We can change that to something like I wrote here (_wait_for_selenium_hub):
https://gerrit.ovirt.org/#/c/98135/2/common/test-scenarios-files/selenium/na...
This function probably needs some improvement (i.e. urllib3 spits out warnings on an unsuccessful connection attempt, so they would need to be silenced), but that's a far better approach than a simple sleep.
- parallelize running Firefox and Chrome tests - there's no reason not to run them both at the same time. There's something called VectorThread in lago.utils. A simple example of usage can be found in '004_basic_sanity.py:955' (disk_operations function). This would have a nice side effect of getting rid of the ugly global ovirt_driver - each thread would have it's own.
- maybe not a running-time improvement, but I think https://gerrit.ovirt.org/#/c/98127/ is still relevant - the way we call save_screenshot is ugly and much too verbose
Right now, I have to switch my focus to some important stuff in VDSM - the OST patches were a continuation of a hackathon effort and something like a "side-project" ;) Still, I don't want the tread to die. I think there's a lot of room for improvements. I can rebase/improve some of my patches if you find them useful. Please keep me posted with your efforts!
Regards, Marcin On 3/7/19 11:10 PM, Greg Sheremeta wrote:
Marcin,
It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.]
@Barak Korren <bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that?
Gal/Gallit/Barak, so is there any way how to store those docker containers within image of lago VM which runs OST tests?
Reviving this now since we have some containerization support for CI.
Can we push this forward?
Adding Barak, Gal and Daniel, IIRC we do have cache for container images in STDCI, not sure if/how it works in OST.
docker.io/selenium/node-chrome-debug 3.9.1-actinium 327adc897d23
13 months ago *904 MB* docker.io/selenium/node-firefox-debug 3.9.1-actinium 88649b420bd5 13 months ago *814 MB*
Greg
On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
*TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor of Chromium. During the OST hackathon in Brno this year, I saw an opportunity to optimize basic UI sanity tests from basic suite. The way we currently run them, is by setting up a Selenium grid using 3 docker containers, with a dedicated network... that's insanity! (pun intended). Let's a look at the running time of '008_basic_ui_sanity.py' scenario ( https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...):
01:31:50 @ Run test: 008_basic_ui_sanity.py: 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 01:31:50 # init: 01:31:50 # init: Success (in 0:00:00) 01:31:50 # start_grid: 01:34:05 # start_grid: Success (in 0:02:15) 01:34:05 # initialize_chrome: 01:34:18 # initialize_chrome: Success (in 0:00:13) 01:34:18 # login: 01:34:27 # login: Success (in 0:00:08) 01:34:27 # left_nav: 01:34:45 # left_nav: Success (in 0:00:18) 01:34:45 # close_driver: 01:34:46 # close_driver: Success (in 0:00:00) 01:34:46 # initialize_firefox: 01:35:02 # initialize_firefox: Success (in 0:00:16) 01:35:02 # login: 01:35:11 # login: Success (in 0:00:08) 01:35:11 # left_nav: 01:35:29 # left_nav: Success (in 0:00:18) 01:35:29 # cleanup: 01:35:36 # cleanup: Success (in 0:00:06) 01:35:36 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45)
Starting the Selenium grid takes 2:15 out of 3:35 of total running time!
I've investigated a lot of approaches and came up with something like this:
- install 'chromium-headless' package on engine VM - download 'chromedriver' and 'selenium hub' jar and deploy them in '/var/opt/' on engine's VM - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' by using Lago's ssh - connect to the Selenium instance running on the engine in '008_basic_ui_sanity.py' - make screenshots
This series of patches represent the changes: https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) . This is the new running time (https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4195/):
20:13:26 @ Run test: 008_basic_ui_sanity.py: 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] 20:13:26 # init: 20:13:26 # init: Success (in 0:00:00) 20:13:26 # make_screenshots: 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /wd/hub 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> http://192.168.201.4:4444/wd/hub/static/resource/hub.html 20:14:02 # make_screenshots: Success (in 0:00:35) 20:14:02 # Results located at /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) (The 'NewConnectionErrors' is waiting for Selenium hub to be up and running, I can silence these later). And the screenshots are here: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
*The pros:*
- we cut the running time by more than 3 minutes
*The cons:*
- we don't get Firefox or Chrome screenshots - we get Chromium screenshots (although AFAIK, QE has much more Selenium tests which cover both Firefox and Chrome) - we polute the engine VM with 'chromium-headless' package and deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and 'minizip'), although we can remove these after the tests
*Some design choices explained:*
Q: Why engine VM?
A: Because the engine VM already has 'X11' libs. We could install 'chromium-headless' (and even other browsers) on our Jenkins executors, but that would mess them up a lot.
Q: Why Chromium?
A: Because it has a separate 'headless' package.
Q: Why not use 'chromedriver' RPM in favor of https://chromedriver.storage.googleapis.com Chromedriver builds?
A: Because the RPM version pulls a lot of extra dependencies even on the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical Google Chromedriver builds, they contain a single binary, and they work for us.
*What still needs to be polished with the patches:*
- Currently 'setup_engine_selenium.sh' script downloads each time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we get much faster set-up times) - we should bake these into the engine VM image template. - 'selenium_hub_running' function in 'selenium_on_engine.py' is hackish - an ability to run an ssh command with a context manager (and auto-terminate on it exits) should be part of Lago. Can be refactored.
Questions, comments, reviews are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/VH52GVG6V5SO34...
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TP3EGSE65SNZXG...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KNKSMINVCBW4XI...
-- Eyal edri He / Him / His MANAGER CONTINUOUS PRODUCTIZATION SYSTEM ENGINEERING Red Hat <https://www.redhat.com/> <https://red.ht/sig> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
On Wed, 7 Aug 2019 at 14:54, Eyal Edri <eedri@redhat.com> wrote:
On Wed, Aug 7, 2019 at 1:24 PM Doron Fediuck <dfediuck@redhat.com> wrote:
On Tue, 12 Mar 2019 at 10:20, Martin Perina <mperina@redhat.com> wrote:
On Fri, Mar 8, 2019 at 12:36 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Greg,
that's a great finding and a very good starting point.
If we want to stick with docker images and Firefox/Chrome testing, I still have some ideas, that would shorten the running time even more:
- we do something like this:
log("waiting %s sec for grid to initialize..." % GRID_STARTUP_DELAY) time.sleep(GRID_STARTUP_DELAY)
this is very inefficient. We can change that to something like I wrote here (_wait_for_selenium_hub):
https://gerrit.ovirt.org/#/c/98135/2/common/test-scenarios-files/selenium/na...
This function probably needs some improvement (i.e. urllib3 spits out warnings on an unsuccessful connection attempt, so they would need to be silenced), but that's a far better approach than a simple sleep.
- parallelize running Firefox and Chrome tests - there's no reason not to run them both at the same time. There's something called VectorThread in lago.utils. A simple example of usage can be found in '004_basic_sanity.py:955' (disk_operations function). This would have a nice side effect of getting rid of the ugly global ovirt_driver - each thread would have it's own.
- maybe not a running-time improvement, but I think https://gerrit.ovirt.org/#/c/98127/ is still relevant - the way we call save_screenshot is ugly and much too verbose
Right now, I have to switch my focus to some important stuff in VDSM - the OST patches were a continuation of a hackathon effort and something like a "side-project" ;) Still, I don't want the tread to die. I think there's a lot of room for improvements. I can rebase/improve some of my patches if you find them useful. Please keep me posted with your efforts!
Regards, Marcin On 3/7/19 11:10 PM, Greg Sheremeta wrote:
Marcin,
It just dawned on me that the main reason 008's start_grid takes so long is that the docker images are fresh pulled every time. Several hundred MB, every time (ugh, sorry). We can and should cache them. What do you think about trying this before doing anything else? [it would also be a good time to update from actinium to the latest, iron.]
@Barak Korren <bkorren@redhat.com> you once mentioned to me we should cache these if they are ok to cache (they are). How do we do that?
Gal/Gallit/Barak, so is there any way how to store those docker containers within image of lago VM which runs OST tests?
Reviving this now since we have some containerization support for CI.
Can we push this forward?
Adding Barak, Gal and Daniel, IIRC we do have cache for container images in STDCI, not sure if/how it works in OST.
I think this was already discussed somewhere else a long time ago - the Selenium images should already be cached on our system at this point
docker.io/selenium/node-chrome-debug 3.9.1-actinium 327adc897d23
13 months ago *904 MB* docker.io/selenium/node-firefox-debug 3.9.1-actinium 88649b420bd5 13 months ago *814 MB*
Greg
On Tue, Mar 5, 2019 at 6:15 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Tue, Mar 5, 2019 at 4:55 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi, On 3/4/19 7:07 PM, Greg Sheremeta wrote:
Hi,
Thanks for trying to improve the tests!
I'm reluctant to give up Firefox sanity tests on every commit, though. In fact, I wanted to add Edge and Safari, because those are also supported browsers. Just today a Firefox only issue was reported, so they are valuable.
Was the Firefox-only issue detected by basic suite or some other tests?
It was reported by a developer. Because GWT compiles permutations per browser, and each browser therefore loads completely separate JavaScript payloads, it's just too easy for it to break in one browser and be fine in the other, so I'm really not ok to remove Firefox.
If Admin Portal was React where there is a single JavaScript payload that's shared among all browsers, then I'd consider it.
Did you consider either leaving a grid up permanently or perhaps using a third party like saucelabs?
I did consider simply having our own grid for the OST. There's even a thread somewhere on ovirt-devel, where someone found OST trying to connect to one of my VMs in Tel Aviv, where my own grid was running :D I couldn't make a public demo though - OST executors couldn't see my VM in tlv.
This approach has 2 big flaws:
- it requires quite a lot of resources for the grid to always be there for us
What about Saucelabs or another third party free tool?
- it makes OST running times somehow undeterministic - situations, where WebDriver has to wait for Selenium hub/nodes to be free, will probably take place
The way I see basic suite's UI sanity tests, is that they're exactly what they're called - sanity tests. We do trivial checks like "can we log in to the webadmin site", "can we go to 'virtual machines' sub-page". I'm not in favor of dropping these completely - I think they make sense, but I also think we can live with a trimmed-down version that saves a lot of time. As I said - AFAIK QE have their own Selenium grid, where they run more complex tests on the UI.
Yes, OST basic_ui_sanity tests aren't "compatibility" tests. We're not checking pixels or look. They are super simple "does the app load" tests, are very valuable, and we're not dropping them.
Greg
Regards, Marcin
Best wishes, Greg
On Mon, Mar 4, 2019, 11:39 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
> Hi, > > *TL; DR* Let's cut the running time of '008_basic_ui_sanity.py' by > more than 3 minutes by sacrificing Firefox and Chrome screenshots in favor > of Chromium. > During the OST hackathon in Brno this year, I saw an opportunity to > optimize basic UI sanity tests from basic suite. > The way we currently run them, is by setting up a Selenium grid > using 3 docker containers, with a dedicated network... that's insanity! > (pun intended). > Let's a look at the running time of '008_basic_ui_sanity.py' > scenario ( > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...): > > > 01:31:50 @ Run test: 008_basic_ui_sanity.py: > 01:31:50 nose.config: INFO: Ignoring files matching ['^\\.', '^_', > '^setup\\.py$'] > 01:31:50 # init: > 01:31:50 # init: Success (in 0:00:00) > 01:31:50 # start_grid: > 01:34:05 # start_grid: Success (in 0:02:15) > 01:34:05 # initialize_chrome: > 01:34:18 # initialize_chrome: Success (in 0:00:13) > 01:34:18 # login: > 01:34:27 # login: Success (in 0:00:08) > 01:34:27 # left_nav: > 01:34:45 # left_nav: Success (in 0:00:18) > 01:34:45 # close_driver: > 01:34:46 # close_driver: Success (in 0:00:00) > 01:34:46 # initialize_firefox: > 01:35:02 # initialize_firefox: Success (in 0:00:16) > 01:35:02 # login: > 01:35:11 # login: Success (in 0:00:08) > 01:35:11 # left_nav: > 01:35:29 # left_nav: Success (in 0:00:18) > 01:35:29 # cleanup: > 01:35:36 # cleanup: Success (in 0:00:06) > 01:35:36 # Results located at > /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml > 01:35:36 @ Run test: 008_basic_ui_sanity.py: Success (in 0:03:45) > > Starting the Selenium grid takes 2:15 out of 3:35 of total running > time! > > I've investigated a lot of approaches and came up with something > like this: > > - install 'chromium-headless' package on engine VM > - download 'chromedriver' and 'selenium hub' jar and deploy them > in '/var/opt/' on engine's VM > - run 'selenium.jar' on engine VM from '008_basic_ui_sanity.py' > by using Lago's ssh > - connect to the Selenium instance running on the engine in > '008_basic_ui_sanity.py' > - make screenshots > > This series of patches represent the changes: > https://gerrit.ovirt.org/#/q/topic:selenium-on-engine+(status:open+OR+status...) > . > This is the new running time (https://jenkins.ovirt.org/view/oVirt > system tests/job/ovirt-system-tests_manual/4195/): > > 20:13:26 @ Run test: 008_basic_ui_sanity.py: > 20:13:26 nose.config: INFO: Ignoring files matching ['^\\.', '^_', > '^setup\\.py$'] > 20:13:26 # init: > 20:13:26 # init: Success (in 0:00:00) > 20:13:26 # make_screenshots: > 20:13:27 * Retrying (Retry(total=2, connect=None, read=None, > redirect=None, status=None)) after connection broken by > 'NewConnectionError('<urllib3.connection.HTTPConnection object at > 0x7fdb6004f8d0>: Failed to establish a new connection: [Errno 111] > Connection refused',)': /wd/hub > 20:13:27 * Retrying (Retry(total=1, connect=None, read=None, > redirect=None, status=None)) after connection broken by > 'NewConnectionError('<urllib3.connection.HTTPConnection object at > 0x7fdb6004fa10>: Failed to establish a new connection: [Errno 111] > Connection refused',)': /wd/hub > 20:13:27 * Retrying (Retry(total=0, connect=None, read=None, > redirect=None, status=None)) after connection broken by > 'NewConnectionError('<urllib3.connection.HTTPConnection object at > 0x7fdb6004fb50>: Failed to establish a new connection: [Errno 111] > Connection refused',)': /wd/hub > 20:13:28 * Redirecting http://192.168.201.4:4444/wd/hub -> > http://192.168.201.4:4444/wd/hub/static/resource/hub.html > 20:14:02 # make_screenshots: Success (in 0:00:35) > 20:14:02 # Results located at > /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml > 20:14:02 @ Run test: 008_basic_ui_sanity.py: Success (in 0:00:35) > (The 'NewConnectionErrors' is waiting for Selenium hub to be up and > running, I can silence these later). > And the screenshots are here: > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... > > *The pros:* > > - we cut the running time by more than 3 minutes > > *The cons:* > > - we don't get Firefox or Chrome screenshots - we get Chromium > screenshots (although AFAIK, QE has much more Selenium tests which cover > both Firefox and Chrome) > - we polute the engine VM with 'chromium-headless' package and > deps (in total: 'chromium-headless', 'chromium-common', 'flac-libs' and > 'minizip'), although we can remove these after the tests > > *Some design choices explained:* > > Q: Why engine VM? > > A: Because the engine VM already has 'X11' libs. We could install > 'chromium-headless' (and even other browsers) on our Jenkins executors, but > that would mess them up a lot. > > Q: Why Chromium? > > A: Because it has a separate 'headless' package. > > Q: Why not use 'chromedriver' RPM in favor of > https://chromedriver.storage.googleapis.com Chromedriver builds? > > A: Because the RPM version pulls a lot of extra dependencies even on > the engine VM ('gtk3', 'cairo' etc.). Builds from the URL are the offical > Google Chromedriver builds, they contain a single binary, and they work for > us. > > *What still needs to be polished with the patches:* > > - Currently 'setup_engine_selenium.sh' script downloads each > time 'selenium.jar' and 'chromedriver.zip' (even with these downloads we > get much faster set-up times) - we should bake these into the engine VM > image template. > - 'selenium_hub_running' function in 'selenium_on_engine.py' is > hackish - an ability to run an ssh command with a context manager (and > auto-terminate on it exits) should be part of Lago. Can be refactored. > > Questions, comments, reviews are welcome. > > Regards, Marcin > > > > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RLB2KSNJS4YKVM... >
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/VH52GVG6V5SO34...
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TP3EGSE65SNZXG...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KNKSMINVCBW4XI...
--
Eyal edri
He / Him / His
MANAGER
CONTINUOUS PRODUCTIZATION
SYSTEM ENGINEERING
Red Hat <https://www.redhat.com/> <https://red.ht/sig> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
participants (7)
-
Barak Korren -
Doron Fediuck -
Eyal Edri -
Greg Sheremeta -
Marcin Sobczyk -
Martin Perina -
Nir Soffer