December 2016 - Devel - oVirt List Archives

[VDSM] Tests leaking child processes --with-leaked-processes
by Nir Soffer 30 Dec '16

30 Dec '16

Hi all, The most common test failure on travis (and when running locally) is tests failing with EINTR: IOError: [Errno 4] Interrupted system call This happens because we are using zombiereaper, and it adds a signal handler for SIGCHLD. When a test leaks a child process and the child process terminates, the test runner get a SIGCHLD. If this happens when the test framework is writing to stdout, the write is interrupted and the test fail, failing the entire build. Here is an example failure on travis: https://travis-ci.org/oVirt/vdsm/jobs/187464715 To fix this we need to find the buggy tests (or tested code) leaking child processes. I added a new plugin for this here: https://gerrit.ovirt.org/69324 (mostly stolen from https://gerrit.ovirt.org/69040) Here is the result of running this plugin with current tests: $ make check NOSE_WITH_LEAKED_PROCESSES=1 ... ====================================================================== ERROR: test_running_repr (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3858" } ] -------------------- >> begin captured stdout << --------------------- <DirectioChecker /path running next_check=4321478.96 at 0x7f0f99812e90> --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,478 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,479 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,489 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616848> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_running_start_raises (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3861" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,513 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,514 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,524 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704979209232> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_repr (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3864" } ] -------------------- >> begin captured stdout << --------------------- <DirectioChecker /path stopping at 0x7f0f99812cd0> --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,548 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,548 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,557 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,558 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616592> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_start_raises (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3867" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,586 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,586 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,596 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,597 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616720> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_stop_ignored (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3870" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,621 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,622 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,632 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,632 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616912> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_taskset_missing (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3873" } ] -------------------- >> begin captured stdout << --------------------- [<CheckResult path=/var/tmp/tmpaIeNcE rc=127 err='Error starting process: [Errno 2] No such file or directory' time=4321479.14 elapsed=0.01 at 0x7f0f99a8bc90>] --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,658 DEBUG (MainThread) [storage.check] Checker '/var/tmp/tmpaIeNcE' started (check:233) 2016-12-30 02:21:32,658 DEBUG (MainThread) [storage.check] START check '/var/tmp/tmpaIeNcE' (delay=0.00) (check:283) 2016-12-30 02:21:32,662 DEBUG (MainThread) [storage.check] FINISH check '/var/tmp/tmpaIeNcE' (rc=127, elapsed=0.01) (check:328) 2016-12-30 02:21:32,663 INFO (MainThread) [storage.asyncevent] Stopping <EventLoop running=False closed=False at 0x139704976616272> (asyncevent:194) 2016-12-30 02:21:32,663 INFO (MainThread) [storage.asyncevent] Starting <EventLoop running=True closed=False at 0x139704976616272> (asyncevent:122) 2016-12-30 02:21:32,663 DEBUG (MainThread) [storage.asyncevent] <EventLoop running=False closed=False at 0x139704976616272> stopped (asyncevent:130) 2016-12-30 02:21:32,664 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616272> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testTimeout (storage_iscsi_test.RescanTimeoutTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "sleep 2", "pid": "4293" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.SamplingMethod] Trying to enter sampling method (vdsm.storage.iscsi.rescan) (misc:467) 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.SamplingMethod] Got in to sampling method (misc:470) 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.ISCSI] Performing SCSI scan, this will take up to 1 seconds (iscsi:457) 2016-12-30 02:21:37,279 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 2 (cwd None) (commands:69) 2016-12-30 02:21:38,285 DEBUG (MainThread) [storage.SamplingMethod] Returning last result (misc:477) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4367" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:39,392 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:39,402 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:241) 2016-12-30 02:21:39,403 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:244) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testCommunicate (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4370" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:39,426 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 /bin/dd (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testMutiWrite (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4373" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:40,463 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:256) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Writing more data to std out (storage_misc_test:258) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:261) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWaitCond (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4376" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:40,501 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 12 (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWaitTimeout (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4379" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:42,537 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 12 (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWriteLargeData (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4382" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:44,572 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:44,580 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:286) 2016-12-30 02:21:44,582 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:289) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that stop really stops the copying process. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4413" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:45,963 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/nice -n 19 /usr/bin/ionice -c 3 /usr/bin/dd if=/tmp/tmpUsJOxV of=/tmp/tmpjQOhqP bs=64 seek=0 skip=0 conv=notrunc,fdatasync count=1 (cwd None) (commands:69) 2016-12-30 02:21:45,974 INFO (MainThread) [DdWatchCopy] Looks like it stopped! (storage_misc_test:465) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testNoCommandWithoutAffinity (storage_misc_test.ExecCmd) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4449" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:02,309 DEBUG (MainThread) [root] I.DONT.EXIST (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that stopping the process really works. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4582" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:06,000 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 10 (cwd None) (commands:69) 2016-12-30 02:22:06,005 INFO (MainThread) [WatchCmd] Looks like task stopped! (storage_misc_test:897) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testAmountOfInstancesPerPoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 93 --write-pipe-fd 92 --max-threads 10 --max-queued-requests 10", "pid": "4861" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:18,447 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:18,447 INFO (MainThread) [IOProcessClient] Starting client ioprocess-14 (__init__:325) 2016-12-30 02:22:18,447 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-14 (__init__:334) 2016-12-30 02:22:18,452 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-14 (__init__:386) 2016-12-30 02:22:18,452 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-14 started (__init__:398) 2016-12-30 02:22:18,452 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:18,452 INFO (MainThread) [IOProcessClient] Starting client ioprocess-15 (__init__:325) 2016-12-30 02:22:18,453 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-15 (__init__:334) 2016-12-30 02:22:18,453 INFO (ioprocess communication (4848)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:18,453 DEBUG (ioprocess communication (4848)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:18,453 DEBUG (ioprocess communication (4848)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:18,457 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-15 (__init__:386) 2016-12-30 02:22:18,458 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-15 started (__init__:398) 2016-12-30 02:22:18,458 INFO (ioprocess communication (4854)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:18,458 DEBUG (ioprocess communication (4854)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:18,458 DEBUG (ioprocess communication (4854)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:24,464 DEBUG (MainThread) [storage.oop] Removing idle ioprocess A (outOfProcess:77) 2016-12-30 02:22:24,464 INFO (MainThread) [IOProcessClient] Closing client ioprocess-15 (__init__:593) 2016-12-30 02:22:24,464 DEBUG (MainThread) [storage.oop] Creating ioprocess B (outOfProcess:87) 2016-12-30 02:22:24,465 INFO (MainThread) [IOProcessClient] Starting client ioprocess-16 (__init__:325) 2016-12-30 02:22:24,465 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-16 (__init__:334) 2016-12-30 02:22:24,470 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-16 (__init__:386) 2016-12-30 02:22:24,470 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-16 started (__init__:398) 2016-12-30 02:22:24,515 INFO (ioprocess communication (4861)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:24,515 DEBUG (ioprocess communication (4861)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:24,516 DEBUG (ioprocess communication (4861)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,557 INFO (MainThread) [IOProcessClient] Closing client ioprocess-14 (__init__:593) 2016-12-30 02:22:25,557 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-14 (__init__:599) 2016-12-30 02:22:25,557 INFO (ioprocess communication (4848)) [IOProcessClient] shutdown requested (__init__:124) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testDifferentPoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 109 --write-pipe-fd 108 --max-threads 10 --max-queued-requests 10", "pid": "4882" }, { "command": "/usr/libexec/ioprocess --read-pipe-fd 101 --write-pipe-fd 99 --max-threads 10 --max-queued-requests 10", "pid": "4876" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:25,583 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:25,584 INFO (MainThread) [IOProcessClient] Starting client ioprocess-17 (__init__:325) 2016-12-30 02:22:25,584 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-17 (__init__:334) 2016-12-30 02:22:25,592 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-17 (__init__:386) 2016-12-30 02:22:25,593 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-17 started (__init__:398) 2016-12-30 02:22:25,593 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:25,593 INFO (ioprocess communication (4870)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,593 DEBUG (ioprocess communication (4870)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,594 INFO (MainThread) [IOProcessClient] Starting client ioprocess-18 (__init__:325) 2016-12-30 02:22:25,594 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-18 (__init__:334) 2016-12-30 02:22:25,594 DEBUG (ioprocess communication (4870)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,604 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-18 (__init__:386) 2016-12-30 02:22:25,604 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-18 started (__init__:398) 2016-12-30 02:22:25,605 DEBUG (MainThread) [storage.oop] Creating ioprocess B (outOfProcess:87) 2016-12-30 02:22:25,605 INFO (ioprocess communication (4876)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,605 DEBUG (ioprocess communication (4876)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,605 INFO (MainThread) [IOProcessClient] Starting client ioprocess-19 (__init__:325) 2016-12-30 02:22:25,605 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-19 (__init__:334) 2016-12-30 02:22:25,605 DEBUG (ioprocess communication (4876)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,610 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-19 (__init__:386) 2016-12-30 02:22:25,610 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-19 started (__init__:398) 2016-12-30 02:22:25,611 INFO (MainThread) [IOProcessClient] Closing client ioprocess-16 (__init__:593) 2016-12-30 02:22:25,611 INFO (MainThread) [IOProcessClient] Closing client ioprocess-17 (__init__:593) 2016-12-30 02:22:25,611 INFO (ioprocess communication (4882)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,611 DEBUG (ioprocess communication (4882)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,611 INFO (ioprocess communication (4870)) [IOProcessClient] shutdown requested (__init__:124) 2016-12-30 02:22:25,612 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-17 (__init__:599) 2016-12-30 02:22:25,612 DEBUG (ioprocess communication (4882)) [IOProcess] Opening communication channels... (__init__:445) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testSamePoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 96 --write-pipe-fd 94 --max-threads 10 --max-queued-requests 10", "pid": "4918" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:25,718 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:25,719 INFO (MainThread) [IOProcessClient] Starting client ioprocess-22 (__init__:325) 2016-12-30 02:22:25,719 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-22 (__init__:334) 2016-12-30 02:22:25,723 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-22 (__init__:386) 2016-12-30 02:22:25,724 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-22 started (__init__:398) 2016-12-30 02:22:25,724 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:25,724 INFO (MainThread) [IOProcessClient] Starting client ioprocess-23 (__init__:325) 2016-12-30 02:22:25,724 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-23 (__init__:334) 2016-12-30 02:22:25,724 INFO (ioprocess communication (4912)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,724 DEBUG (ioprocess communication (4912)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,725 DEBUG (ioprocess communication (4912)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,729 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-23 (__init__:386) 2016-12-30 02:22:25,730 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-23 started (__init__:398) 2016-12-30 02:22:25,730 INFO (MainThread) [IOProcessClient] Closing client ioprocess-18 (__init__:593) 2016-12-30 02:22:25,730 INFO (ioprocess communication (4918)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,730 DEBUG (ioprocess communication (4918)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,730 DEBUG (ioprocess communication (4918)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,731 INFO (MainThread) [IOProcessClient] Closing client ioprocess-22 (__init__:593) 2016-12-30 02:22:25,731 INFO (ioprocess communication (4912)) [IOProcessClient] shutdown requested (__init__:124) 2016-12-30 02:22:25,731 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-22 (__init__:599) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_get (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7676" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,549 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7676 (cwd None) (commands:69) 2016-12-30 02:22:54,555 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,555 DEBUG (MainThread) [root] /usr/bin/taskset --pid 567 (cwd None) (commands:69) 2016-12-30 02:22:54,562 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([0, 3])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7684" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,652 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7684 (cwd None) (commands:69) 2016-12-30 02:22:54,658 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([0])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7689" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,701 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7689 (cwd None) (commands:69) 2016-12-30 02:22:54,707 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([1, 2])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7694" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,752 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7694 (cwd None) (commands:69) 2016-12-30 02:22:54,763 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([0, 3])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7699" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,797 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 0,3 7699 (cwd None) (commands:69) 2016-12-30 02:22:54,808 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,808 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7699 (cwd None) (commands:69) 2016-12-30 02:22:54,813 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([0])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7704" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,849 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 0 7704 (cwd None) (commands:69) 2016-12-30 02:22:54,855 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,855 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7704 (cwd None) (commands:69) 2016-12-30 02:22:54,861 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([1, 2])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7710" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,909 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 1,2 7710 (cwd None) (commands:69) 2016-12-30 02:22:54,920 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,920 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7710 (cwd None) (commands:69) 2016-12-30 02:22:54,925 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '1'], ['sleep', '3'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9495" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:34,072 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 1 (cwd None) (v2v:1301) 2016-12-30 02:23:34,082 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:37,086 DEBUG (MainThread) [root] Killing pid=9494 (v2v:679) 2016-12-30 02:23:37,086 DEBUG (MainThread) [root] Killing pid=9495 (v2v:679) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '3'], ['sleep', '1'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9498" }, { "command": "", "pid": "9499" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:37,112 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:37,122 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 1 (cwd None) (v2v:1301) 2016-12-30 02:23:39,134 DEBUG (MainThread) [root] Killing pid=9498 (v2v:679) 2016-12-30 02:23:39,135 DEBUG (MainThread) [root] Killing pid=9499 (v2v:679) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '3'], ['sleep', '3'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9503" }, { "command": "", "pid": "9504" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:39,164 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:39,176 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:41,186 DEBUG (MainThread) [root] Killing pid=9503 (v2v:679) 2016-12-30 02:23:41,187 DEBUG (MainThread) [root] Killing pid=9504 (v2v:679) --------------------- >> end captured logging << ---------------------

1 0

Jenkins jobs failure
by Gil Shinar 29 Dec '16

29 Dec '16

Hi all, We have have a merged a patch yesterday that made a very important enhancement to the logs collection of the STD-CI jobs. This merge caused a small regression that we have already found and working on fixing it. Please be patient till this fix will be merged. Thanks Gil

1 1

Jenkins restart
by Gil Shinar 28 Dec '16

28 Dec '16

Hi all, Jenkins restart is planned today at 18:00. Patches that will be sent or merged up to an hour before that time might fail and would need to be re-triggered. If there's any urgent patch that needs to be sent or merged at that time, please let me know before 18:00. Thanks Gil

1 1

Re: [ovirt-devel] ost host addition failure
by Eyal Edri 27 Dec '16

27 Dec '16

Any updates? The tests are still failing on vdsmd won't start from Sunday... master repos havn't been refreshed for a few days due to this. from host deploy log: [1] basic-suite-master-engine/_var_log_ovirt-engine/host-deploy/ovirt-host-deploy-20161227012930-192.168.201.4-14af2bf0.log the job links [2] [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/lastComple… 016-12-27 01:29:29 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stdout: 2016-12-27 01:29:29 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stderr: A dependency job for vdsmd.service failed. See 'journalctl -xe' for details. 2016-12-27 01:29:29 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/tmp/ovirt-QZ1ucxWFfm/pythonlib/otopi/context.py", line 132, in _executeMethod method['method']() File "/tmp/ovirt-QZ1ucxWFfm/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 209, in _start self.services.state('vdsmd', True) File "/tmp/ovirt-QZ1ucxWFfm/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'vdsmd' 2016-12-27 01:29:29 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Closing up': Failed to start service 'vdsmd' 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/excep [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/lastComple… On Sun, Dec 25, 2016 at 11:31 AM, Eyal Edri <eedri(a)redhat.com> wrote: > We should see it fixed here hopefully [1] > > > [1] http://jenkins.ovirt.org/view/All%20Running%20jobs/job/ > test-repo_ovirt_experimental_master/4412/console > > On Sun, Dec 25, 2016 at 11:19 AM, Dan Kenigsberg <danken(a)redhat.com> > wrote: > >> On Sun, Dec 25, 2016 at 10:28 AM, Yaniv Kaul <ykaul(a)redhat.com> wrote: >> > >> > >> > On Sun, Dec 25, 2016 at 9:47 AM, Dan Kenigsberg <danken(a)redhat.com> >> wrote: >> >> >> >> Correct. https://gerrit.ovirt.org/#/c/69052/ >> >> >> >> Can you try adding >> >> lago shell "$vm_name" -c "mkdir -p /var/log/ovirt-imageio-daemon/ && >> >> chown vdsm:kvm /var/log/ovirt-imageio-daemon/" >> > >> > >> > How will it know what is the vdsm user before installing vdsm? >> >> You're right. a hack would have to `chmod a+rwx >> /var/log/ovirt-imageio-daemon/` instead. >> >> > Why not either: >> > 1. Fix it >> >> yes, that's why we've opened >> https://bugzilla.redhat.com/show_bug.cgi?id=1400003 ; now a fix is >> getting merged. I don't know when it is going to be ready in lago's >> repos. >> >> > -or- >> > 2. Revert the offending patch? >> >> I'm not aware of such patch. It's a race that has been there since >> ever, and I don't know why it suddenly pops up so often. >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

4 5

[VDSM] storage_iscsi_test.RescanTimeoutTests test failing
by Edward Haas 26 Dec '16

26 Dec '16

I got this test failing on one of the builds: ( http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5266/console ) *09:39:16* ======================================================================*09:39:16* FAIL: testTimeout (storage_iscsi_test.RescanTimeoutTests)*09:39:16* ----------------------------------------------------------------------*09:39:16* Traceback (most recent call last):*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/monkeypatch.py", line 133, in wrapper*09:39:16* return f(*args, **kw)*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/monkeypatch.py", line 133, in wrapper*09:39:16* return f(*args, **kw)*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_iscsi_test.py", line 45, in testTimeout*09:39:16* iscsi.rescan()*09:39:16* File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__*09:39:16* self.gen.next()*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_iscsi_test.py", line 33, in assertMaxDuration*09:39:16* (elapsed, maxtime))*09:39:16* AssertionError: Operation was too slow 1.32s > 1.20s*09:39:16* -------------------- >> begin captured logging << --------------------*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.SamplingMethod] Trying to enter sampling method (vdsm.storage.iscsi.rescan) (misc:467)*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.SamplingMethod] Got in to sampling method (misc:470)*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.ISCSI] Performing SCSI scan, this will take up to 1 seconds (iscsi:457)*09:39:16* 2016-12-26 09:36:46,858 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 2 (cwd None) (commands:69)*09:39:16* 2016-12-26 09:36:48,183 DEBUG (MainThread) [storage.SamplingMethod] Returning last result (misc:477)*09:39:16* --------------------- >> end captured logging << --------------------- I was surprised that it uses a time measurement to determine success. Edy.

1 0

Check-merged is broken since we added network test to it
by Yaniv Bronheim 26 Dec '16

26 Dec '16

Hi guys and Leon, https://gerrit.ovirt.org/#/c/68078/ broke the check-merged job with really tough exception: *15:23:34* sh: [17766: 1 (255)] tcsetattr: Inappropriate ioctl for device*15:23:34* Took 2586 seconds*15:23:34* Slave went offline during the build <http://jenkins.ovirt.org/computer/vm0136.workers-phx.ovirt.org/log>*15:23:34* ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel*15:23:34* at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)*15:23:34* Caused by: java.io.EOFException*15:23:34* at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)*15:23:34* at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)*15:23:34* at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)*15:23:34* at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)*15:23:34* at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)*15:23:34* at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)*15:23:34* at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)*15:23:34* *15:23:34* Build step 'Execute shell' marked build as failure*15:23:34* Performing Post build task... I have no clue what causes it.. we need to investigate in the tests code. you can see it in http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/772/consol… and this for a job run just before it got in, which worked well - http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/688/console I suggest to revert this patch (and backport the revert to ovirt-4.1 branch as well) until figuring what causes it -- *Yaniv Bronhaim.*

4 9

oVirt system tests currently failing to AddHost on master
by Barak Korren 25 Dec '16

25 Dec '16

It hard to tell currently when did this start b/c we had so package issues that made the tests fail before reaching that point most of the day. Since we currently have an issue in Lago with collecting AddHost logs (Hopefully we'll resolve this in the next release early next week), I`ve ran the tests locally and attached the bundle of generated logs to this message. Included in the attached file are engine logs, host-deploy logs and VDSM logs for both test hosts. >From a quick look inside it seems the issue is with VDSM failing to start. Thanks, -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

5 15

ovirt.org is back online
by Eyal Edri 25 Dec '16

25 Dec '16

The site is back online thanks for Duck in helping clearing the log files which caused the gear to get stuck. I understood that the site is planned to move to another location, with more storage so hopefully these kind of outage will be rare soon after. -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[Attention] Gerrit 'git://' protocol error
by Nadav Goldin 25 Dec '16

25 Dec '16

Hi, Currently git-daemon is not working on gerrit.ovirt.org, so all actions with remote configured with 'git://' prefix would fail(including Jenkins jobs). We are working on resolving the issue, until then you can use 'https://' for anonymous clones, or SSH(ssh://username@gerrit.ovirt.org:29418/repo.git). Thanks, Nadav.

1 1

[VDSM] Support for external leases - possible breakage with development version
by Nir Soffer 24 Dec '16

24 Dec '16

Hi all, I merged today the last patches for supporting external leases. In storage domain version 4 (the default in ovirt 4.1), there is now a new volume named "xleases". When upgrading older storage domain (version 3), we create and format the new volume. If you used a development version with storage domain version 4, your storage domain may not work with current master. You should recreated such storage domain. As a workaround, you can create the special volume manually: File storage: sudo -u vdsm truncate -s 1G /rhev/data-center/mnt/server:_path/domain_uuid/dom_md/xleases Block storage: lvcreate --name xleases --size 1g domain_uuid Note that the volume should be also be formatted to actually use external leases, but having it should be good enough to keep the system happy. Thanks, Nir

1 1

Re: [ovirt-devel] System tests for 4.1 currently failing to run VMs!
by Oved Ourfali 22 Dec '16

22 Dec '16

Why not run it via Jenkins for patches? Like, if you add a comment saying "run: ost" it will run it? It do it automatically based on another thing? On Dec 21, 2016 17:42, "Eyal Edri" <eedri(a)redhat.com> wrote: > > > > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >> >> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul(a)redhat.com> wrote: >>> >>> >>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >>>> >>>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >>>>> >>>>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <eedri(a)redhat.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra < vfeenstr(a)redhat.com> wrote: >>>>>>> >>>>>>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <bkorren(a)redhat.com> wrote: >>>>>>>> >>>>>>>> The test for running VMs had been failing since yesterday. >>>>>>>> >>>>>>>> The patch merged before the failures started was: >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) while running >>>>>>>> this API call: >>>>>>>> >>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up' >>>>>>> >>>>>>> >>>>>>> This is a REST API call, the patch above is Frontend. So this is unrelated. >>>>>>> >>>>>>> However on Host 0 I can see this: >>>>>>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process failed (vm:615) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: internal error: process exited while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) >>>>> >>>>> >>>>> it is likely related to the recent USB patches >>>>> investigating >>>> >>>> >>>> hm, there are multiple problems (features/bugs depending on prefferred point of view:) >>>> but there is an easy “fix” taking care of this particular problem, so we can start with that and figure out the proper approach later >>>> arik will push that and merge it soon, likely today >>> >>> >>> Thanks - if there is a quicker way to resolve this by reverting, I think it's a better option. >> >> >> I really need to talk you out of this approach:-) >> It does sound tempting and logical, but with our development model of large patch series combined with late detection it really is quite risky. Here it wouldn’t help much…and figuring out the right revert patch is more complicated then fixing it. > > > Can we start asking developers run OST before they merge so it will be early detection and not late detection? > We have video sessions on how to use OST, so it shouldn't be any issues in running it on a patch. > >> >> I believe the best is to identify it early and notify the maintainer who merged that patch ASAP, as that person is in the best position to asses if revert is safe or if there is a simple follow up patch he can push right away >> >> We can surely improve on reporting, so Barak, how/why did you point to that particular patch in your email? It should start failing on 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 01:56:07 2016 -0500) >> Even though it may be that it was hidden because of c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 15:16:40 2016 -0500) >> >> (fix is ready, waiting on CI now) >> >> Thanks, >> michal >> >>> Y. >>> >>>> >>>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: internal error: process exited while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping connection (guestagent:430) >>>>>>> >>>>>>> >>>>>>> >>>>>>> And on The engine loads of these: >>>>>>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call 2016-12-20 16:53:57,849-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [7971dfb4] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}� 2016-12-20 16:53:57,850-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:287) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1389) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2053) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:170) at org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(BackendHostResource.java:212) at org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:116) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM response: Internal timeout occured >>>>>>> >>>>>>> >>>>>>> This is definitely not related to the patch linked above. >>>>>>> >>>>>>> However I am also not quite sure what is the cause of this right now, it might be a breaking change in VDSM. >>>>>>> >>>>>> >>>>>> The DC level is 4.0, is it possible that is the reason for the failure? >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Full test code can be seen here: >>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-sui… >>>>>>>> >>>>>>>> Full test exception can be seen here: >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport… >>>>>>>> >>>>>>>> Further logs can be seen in Jenkins: >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/e… >>>>>>>> >>>>>>>> -- >>>>>>>> Barak Korren >>>>>>>> bkorren(a)redhat.com >>>>>>>> RHCE, RHCi, RHV-DevOps Team >>>>>>>> https://ifireball.wordpress.com/ >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list >>>>>>>> Devel(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Eyal Edri >>>>>> Associate Manager >>>>>> RHV DevOps >>>>>> EMEA ENG Virtualization R&D >>>>>> Red Hat Israel >>>>>> >>>>>> phone: +972-9-7692018 >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

6 11

Re: [ovirt-devel] [RFE] treat local NFS storage as localfs
by Pavel Gashev 22 Dec '16

22 Dec '16

On Wed, 2016-12-21 at 16:25 +0100, Martin Sivak wrote: > > So VMs on one host will get better IO performance and the others will > still use NFS as they do now. Performance is not an issue when all I/O bound VMs are co-located with disks. If all VMs are I/O bound, all hosts in a cluster can use local storage. > It is an interesting idea, I am just not sure if having poor-man's > hyperconverged setup with all the drawbacks of NFS is worth it. > Imagine for example what happens when that storage provider host > needs > to be fenced or put into maintenance. The whole cluster would go down > (all VMs would lose storage connection, not just the VMs from the > affected host). Losing a server together with a storage could be an issue, or not. It really depends on setup. At least it would be good to provide such feature :) > I will let someone from the storage team to respond to this, but I do > not think that trading performance (each host has its own local > storage) and resilience (well, at least one failing host does not > affect the others) for migrations is a good deal. > > -- > Martin Sivak > SLA / oVirt > > On Wed, Dec 21, 2016 at 2:18 PM, Sven Kieske <s.kieske(a)mittwald.de> > wrote: > > > > On 21/12/16 11:44, Pavel Gashev wrote: > > > > > > Hello, > > > > > > I'd like to introduce a RFE that allows to use a local storage in > > > multi > > > server environments https://bugzilla.redhat.com/show_bug.cgi?id=1 > > > 406412 > > > > > > Most servers have a local storage. Some servers have very > > > reliable > > > storages with hardware RAID controllers and battery units. > > > > > > Example user cases: > > > https://www.mail-archive.com/users@ovirt.org/msg36719.html > > > https://www.mail-archive.com/users@ovirt.org/msg36772.html > > > > > > The best way to use local storage in multi server "shared" > > > datacenters > > > is exporting it over NFS. Using NFS allows to move disks and VMs > > > among > > > servers. > > > > > > In order to improve performance, disk I/O bound VMs can be pinned > > > to > > > a host with local storage. However there still is a performance > > > drawback of NFS layers. Treating a local NFS storage as a local > > > storage > > > improves performance for VMs pinned to host. > > > > > > Currently setting up of NFS exports is out of scope of oVirt. > > > However > > > this would be a way to get rid of "Local/Shared" storage types of > > > datacenter. So that all storages are shared, but local storages > > > are > > > used as local. > > > > > > Any questions/comments are welcome. > > > > > > Specifically I'd like to request for comment on potential data > > > integrity issues during online VM or disk migration between NFS > > > and > > > localfs. > > > > > > > Just let me say that I really like this as an end user. > > > > Hope this get's in. This seems less overhead than a complete > > hyperconverged gluster setup. > > > > > > -- > > Mit freundlichen Grüßen / Regards > > > > Sven Kieske > > > > Systemadministrator > > Mittwald CM Service GmbH & Co. KG > > Königsberger Straße 6 > > 32339 Espelkamp > > T: +495772 293100 > > F: +495772 293333 > > https://www.mittwald.de > > Geschäftsführer: Robert Meyer > > St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad > > Oeynhausen > > Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad > > Oeynhausen > > > > > > _______________________________________________ > > Devel mailing list > > Devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

1 0

Re: [ovirt-devel] [RFE] treat local NFS storage as localfs
by Michal Skrivanek 22 Dec '16

22 Dec '16

> On 21 Dec 2016, at 16:26, Martin Sivak <msivak(a)redhat.com> wrote: > > Hi, > >> Hope this get's in. This seems less overhead than a complete >> hyperconverged gluster setup. > > But NFS still is a single point of failure. Hyperconverged is supposed > to address that. > >>> In order to improve performance, disk I/O bound VMs can be pinned to >>> a host with local storage. However there still is a performance >>> drawback of NFS layers. Treating a local NFS storage as a local storage >>> improves performance for VMs pinned to host. > > So VMs on one host will get better IO performance and the others will > still use NFS as they do now. > > It is an interesting idea, I am just not sure if having poor-man's > hyperconverged setup with all the drawbacks of NFS is worth it. > Imagine for example what happens when that storage provider host needs > to be fenced or put into maintenance. The whole cluster would go down > (all VMs would lose storage connection, not just the VMs from the > affected host). > > I will let someone from the storage team to respond to this, but I do > not think that trading performance (each host has its own local > storage) and resilience (well, at least one failing host does not > affect the others) for migrations is a good deal. If disk performance is critical then there is an option to use direct access on local host using either PCI passthrough of a local storage controller or SCSI passthrough of LUNs. > > -- > Martin Sivak > SLA / oVirt > >> On Wed, Dec 21, 2016 at 2:18 PM, Sven Kieske <s.kieske(a)mittwald.de> wrote: >>> On 21/12/16 11:44, Pavel Gashev wrote: >>> Hello, >>> >>> I'd like to introduce a RFE that allows to use a local storage in multi >>> server environments https://bugzilla.redhat.com/show_bug.cgi?id=1406412 >>> >>> Most servers have a local storage. Some servers have very reliable >>> storages with hardware RAID controllers and battery units. >>> >>> Example user cases: >>> https://www.mail-archive.com/users@ovirt.org/msg36719.html >>> https://www.mail-archive.com/users@ovirt.org/msg36772.html >>> >>> The best way to use local storage in multi server "shared" datacenters >>> is exporting it over NFS. Using NFS allows to move disks and VMs among >>> servers. >>> >>> In order to improve performance, disk I/O bound VMs can be pinned to >>> a host with local storage. However there still is a performance >>> drawback of NFS layers. Treating a local NFS storage as a local storage >>> improves performance for VMs pinned to host. >>> >>> Currently setting up of NFS exports is out of scope of oVirt. However >>> this would be a way to get rid of "Local/Shared" storage types of >>> datacenter. So that all storages are shared, but local storages are >>> used as local. >>> >>> Any questions/comments are welcome. >>> >>> Specifically I'd like to request for comment on potential data >>> integrity issues during online VM or disk migration between NFS and >>> localfs. >>> >> >> Just let me say that I really like this as an end user. >> >> Hope this get's in. This seems less overhead than a complete >> hyperconverged gluster setup. >> >> >> -- >> Mit freundlichen Grüßen / Regards >> >> Sven Kieske >> >> Systemadministrator >> Mittwald CM Service GmbH & Co. KG >> Königsberger Straße 6 >> 32339 Espelkamp >> T: +495772 293100 >> F: +495772 293333 >> https://www.mittwald.de >> Geschäftsführer: Robert Meyer >> St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen >> Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen >> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > >

3 3

Branching Vdsm to ovirt-4.1
by Yaniv Bronheim 22 Dec '16

22 Dec '16

Hello devels, I just introduced ovirt-4.1 branch in vdsm - which will be the base for ovirt-4.1 alpha build. Now master is tagged v4.20.0 and first ovirt-4.1 tag is v4.19.1. Basically it means that every fix that should be in 4.1 needs to be backported and merged to ovirt-4.1 branch by fromani or me. We both in #vdsm so ping us if needed and add us to the patches. Bare in mind that in most of the cases we will ask for Bug-Url link in the commit message and same commit-id as in the master branch patch. For any further questions or wonders feel free to reply. Greetings, -- *Yaniv Bronhaim.*

1 0

Fwd: Fedora 23 End Of Life
by Sandro Bonazzola 21 Dec '16

21 Dec '16

FYI. ---------- Messaggio inoltrato ---------- Da: "Mohan Boddu" <mboddu(a)redhat.com> Data: 21/Dic/2016 04:05 Oggetto: Fedora 23 End Of Life A: <announce(a)lists.fedoraproject.org>, < test-announce(a)lists.fedoraproject.org>, < devel-announce(a)lists.fedoraproject.org> Cc: As of the 20th of December 2016, Fedora 23 has reached its end of life for updates and support. No further updates, including security updates, will be available for Fedora 23. A previous reminder was sent on 28th of November 2016 [0]. Fedora 24 will continue to receive updates until approximately one month after the release of Fedora 26. The maintenance schedule of Fedora releases is documented on the Fedora Project wiki [1]. The Fedora Project wiki also contains instructions [2] on how to upgrade from a previous release of Fedora to a version receiving updates. Mohan Boddu. [0]https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org/thread/HLHKRTIB33EDZXP624GHF2OZLHWAGKSJ/# Q5O44X4BEBOYEKAEVLSXVI44DSNVHBYG [1]https://fedoraproject.org/wiki/Fedora_Release_Life_ Cycle#Maintenance_Schedule [2]https://fedoraproject.org/wiki/Upgrading?rd=DistributionUpgrades _______________________________________________ devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org

1 0

test mail
by Evgheni Dereveanchin 21 Dec '16

21 Dec '16

Hi, this is a test message to check that archiving is back to normal as part of OVIRT-949 [1] Please ignore. Regards, Evgheni Dereveanchin [1] https://ovirt-jira.atlassian.net/browse/OVIRT-949

1 0

Heads-up: moving Libvirt xml creation to the engine
by Arik Hadas 08 Dec '16

08 Dec '16

Hi All, We are working on something that is expected to have a big impact, hence this heads-up. First, we want you to be aware of this change and provide your feedback to make it as good as possible. Second, until the proposed mechanism is fully merged there will be a chase to cover all features unless new features are also implemented with the new mechanism. So please, if you are working on something that adds/changes something in the Libvirt's domain xml, do it with this new mechanism as well (first version would be merged soon). * Goal Creating Libvirt XML in the engine rather than in VDSM. ** Today's flow Engine: VM business entity -> VM properties map VDSM: VM properties map -> Libvirt XML ** Desired flow Engine: VM business entity -> Libvirt XML * Potential Benefits 1. Reduce the number of conversions from 2 to 1, reducing chances for mistakes in the process. 2. Reduce the amount of code in VDSM. 3. Make VM related changes easier - today many of these changes need to be reviewed in 2 projects, this will eliminate the one that tends to take longer. 4. Prevent shortcuts in the form of VDSM-only changes that should be better reflected in the engine. 5. Not to re-generate the XML on each rerun attempt of VM run/migration. 6. Future - not to re-generate the XML on each attempt to auto-start HA VM when using vm-leases (need to make sure we're using the up-to-date VM configuration though). 7. We already found improvements and cleanups that could be made while touching this area (e.g., remove the boot order from devices in the database). * Challenges 1. Not to move host-specific information to the engine. For example, path to storage domain or sockets of channels. The solution is to use place-holders that will be replaced by VDSM. 2. Backward compatibility. 3. The more challenging part is the other direction - that will be the next phase. * Status As a first step, we began with producing the Libvirt XML in the engine by converting the VM properties map to XML in the engine [1] And using the XML that is received as an input in VDSM [2] [1] https://gerrit.ovirt.org/#/c/64473/ [2] https://gerrit.ovirt.org/#/c/65182/ Regards, Arik

11 24

Red Hat Summit: oVirt Talk(s) Submission
by Brian Proffitt 08 Dec '16

08 Dec '16

One of the big events for the year is the Red Hat Summit, which has a dedicated open source community track for upstream projects. At the very least this is an opportunity to give a "state of the union" talk for the various communities, including oVirt. There are a significant number of downstream-related talks for Summit, but if you would like to submit an oVirt-related talk for the open source community track[1] before the CFP for Summit closes on Dec. 16, please do so! Peace, Brian [1] https://rh2017.smarteventscloud.com/portal/cfp/cfpLogin.ww?sc_cid=701600000… -- Brian Proffitt Principal Community Analyst Open Source and Standards @TheTechScribe 574.383.9BKP

1 0

ovirt-appliance (today) is 1.5G...
by Yaniv Kaul 08 Dec '16

08 Dec '16

I'm sure it was less just days ago, but now[1] it is again quite big. 4.0's is 'only' 1G in size[2]. Which anyway raises the question - should we provide delta RPMs for it? Thanks, Y. [1] http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ovirt-e… [2] http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/ovirt-engi…

3 4

set_modified_hook (update)
by Shlomo Ben David 08 Dec '16

08 Dec '16

Hi All, This email is an update for the set_modified hook hook: set_modified hook goal: when patch merged, changes the bug status from POST to MODIFIED. - The current version of the hook doesn't checks that all previous patches (on the external tracker) were closed before setting the bug status to MODIFIED. - I released this [1] patch that fixes it. - If you have any questions/issues about this/other hooks please fill free to send an email to me / infra.ovirt.org [1] - https://gerrit.ovirt.org/#/c/67512/1 Thanks for your cooperation, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

1 0

New test failure on travis
by Nir Soffer 08 Dec '16

08 Dec '16

Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this: See https://travis-ci.org/oVirt/vdsm/jobs/181933329 ====================================================================== ERROR: test suite for <module 'network.nmdbus_test' from '/vdsm/tests/network/nmdbus_test.py'> ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run self.setUp() File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp self.setupContext(ancestor) File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext try_run(context, names) File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run return func() File "/vdsm/tests/testValidation.py", line 191, in wrapper return f(*args, **kwargs) File "/vdsm/tests/testValidation.py", line 97, in wrapper return f(*args, **kwargs) File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module NMDbus.init() File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init NMDbus.bus = dbus.SystemBus() File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, in __new__ private=private) File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, in __new__ bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop) File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in __new__ bus = cls._new_for_bus(address_or_type, mainloop=mainloop) DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory -------------------- >> begin captured logging << -------------------- 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None) (commands:69) 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED: <err> = 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = 1 (commands:93) 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None) (commands:69) 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED: <err> = 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = 1 (commands:93) --------------------- >> end captured logging << ---------------------

4 7

[Feture discussion] Full vacuum tool
by Roy Golan 08 Dec '16

08 Dec '16

Hi all, This is a discussion on the RFE[1] to provide a tool to perform full vacuum on our DBs. First if you are not familiar with vacuum please read this [2] # Backgroud ovirt 'engine' DB have several busy table with 2 differnt usage patten. One is audit_log and the others are the 'v*_statistics' tables and the difference between them is mostly inserts vs mostly hot updates. Tables with tons of updates creates garbage or 'dead' records that should be removed, and for this postgres have the aforementioned autovacuum cleaner. It will make the db reuse its already allocated space to perform future updates/inserts and so on. Autovacuum is essential for a db to function optimally and tweaking it is out of the scope of the feature. Full vacuum is designed to reclaim the disk space and reset the table statistics. It is a heavy maintenance task, it takes an exclusive lock on the table and may take seconds to minutes. In some situations it is effectively a downtime due to the long table lock and should not be running when the engine is running. # Critiria Provide a way to reclaim disk space claimed by the garbage created over time by the engine db and dwh. # Usage Either use it as part of the upgrade procedure (after all dbscipts execution) or just provide the tool and admin will run in on demand - engine db credentials read from /etc/ovirt-engine/engine.conf.d/ - invocation: ``` tool: [dbname(default engine)] [table: (default all)] ``` - if we invoke it on upgrade than an installation plugin should be added to invoke with default, no interaction - since VACUUM ANALYZE is consider a recommended maintenance task we can to it by default and ask the user for FULL. - remote db is supported as well, doesn't have to be local # Questions - Will remote dwh have the credentials under /etc/ovirt-engine/engine.conf.d? - Should AAA schema be taken into account as well? Please review, thanks Roy [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430 [2] https://www.postgresql.org/docs/9.2/static/runtime-config-autovacuum.html [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html

6 11

The engine build job you probably didn't know exists
by Eyal Edri 08 Dec '16

08 Dec '16

FYI, After hearing that this might be useful to many developers, I wanted to bring to your attention the ovirt-engine master build job from patch [1] which allows you to build new rpms from an open ovirt-engine patch on Gerrit. Its was created as temp job for a team who needed it a few months ago, but seems it might useful for other teams as well, so we decided to publish it even though its not standardized yet as part of our standard CI system. I hope you can find it useful for your teams. [1] http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64… -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

6 9

Review request for vdsm patch https://gerrit.ovirt.org/#/c/67912/
by Ramesh Nachimuthu 08 Dec '16

08 Dec '16

Hi, I have sent the patch in subject to fix bz#1402248. It blocks gluster brick provisioning in oVirt 4.0.6. It will be helpful if someone can review this patch. Regards, Ramesh

1 0

[VDSM] Correct implementation of virt-sysprep job
by Shmuel Melamud 07 Dec '16

07 Dec '16

Hi! I'm currently working on integration of virt-sysprep into oVirt. Usually, if user creates a template from a regular VM, and then creates new VMs from this template, these new VMs inherit all configuration of the original VM, including SSH keys, UDEV rules, MAC addresses, system ID, hostname etc. It is unfortunate, because you cannot have two network devices with the same MAC address in the same network, for example. To avoid this, user must clean all machine-specific configuration from the original VM before creating a template from it. You can do this manually, but there is virt-sysprep utility that does this automatically. Ideally, virt-sysprep should be seamlessly integrated into template creation process. But the first step is to create a simple button: user selects a VM, clicks the button and oVirt executes virt-sysprep on the VM. virt-sysprep works directly on VM's filesystem. It accepts list of all disks of the VM as parameters: virt-sysprep -a disk1.img -a disk2.img -a disk3.img The architecture is as follows: command on the Engine side runs a job on VDSM side and tracks its success/failure. The job on VDSM side runs virt-sysprep. The question is how to implement the job correctly? I thought about using storage jobs, but they are designed to work only with a single volume, correct? Is is possible to use them with operation that is performed on multiple volumes? Or, alternatively, is it possible to use some kind of 'VM jobs' - that work on VM at whole? How v2v solves this problem? Any ideas? Shmuel

9 18

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job
by Oved Ourfali 07 Dec '16

07 Dec '16

On Dec 7, 2016 20:24, "Michal Skrivanek" <mskrivan(a)redhat.com> wrote: > > > > On 07 Dec 2016, at 09:17, Oved Ourfali <oourfali(a)redhat.com> wrote: > >> >> >> On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke <alitke(a)redhat.com> wrote: >>> >>> On 06/12/16 22:06 +0200, Arik Hadas wrote: >>>> >>>> Adam, >>> >>> >>> :) You seem upset. Sorry if I touched on a nerve... >>> >>>> Just out of curiosity: when you write "v2v has promised" - what exactly do you >>>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I that >>>> implemented the integration with virt-v2v? I'm not aware of such a promise by >>>> any of these options :) >>> >>> >>> Some history... >>> >>> Earlier this year Nir, Francesco (added), Shahar, and I began >>> discussing the similarities between what storage needed to do with >>> external commands and what was designed specifically for v2v. I am >>> not sure if you were involved in the project at that time. The plan >>> was to create common infrastructure that could be extended to fit the >>> unique needs of the verticals. The v2v code was going to be moved >>> over to the new infrastructure (see [1]) and the only thing that >>> stopped the initial patch was lack of a VMWare testing environment for >>> verification. >>> >>> At that time storage refocused on developing verbs that used the new >>> infrastructure and have been maintaining its suitability for general >>> use. Conversion of v2v -> Host Jobs is obviously a lower priority >>> item and much more difficult now due to the early missed opportunity. >>> >>>> Anyway, let's say that you were given such a promise by someone and thus >>>> consider that mechanism to be deprecated - it doesn't really matter. >>> >>> >>> I may be biased but I think my opinion does matter. >>> >>>> The current implementation doesn't well fit to this flow (it requires >>>> per-volume job, it creates leases that are not needed for template's disks, >>>> ...) and with the "next-gen API" with proper support for virt flows not even >>>> being discussed with us (and iiuc also not with the infra team) yet, I don't >>>> understand what do you suggest except for some strong, though irrelevant, >>>> statements. >>> >>> >>> If you are willing to engage in a good-faith technical discussion I am >>> sure I can help you to understand. These operations to storage demand >>> some form of locking protection. If volume leases aren't appropriate then >>> perhaps we should use the VM Leases / xleases that Nir is finishing >>> off for 4.1 now. >>> >>>> I suggest loud and clear to reuse (not to add dependencies, not to enhance, ..) >>>> an existing mechanism for a very similar flow of virt-v2v that works well and >>>> simple. >>> >>> >>> I clearly remember discussions involving infra (hello Oved), virt >>> (hola Michal), and storage where we decided that new APIs performing >>> async operations involving external commands should use the HostJobs >>> infrastructure instead of adding more information to Host Stats. >>> These were the "famous" entity polling meetings. >>> >>> Of course plans can change but I have never been looped into any such >>> discussions. >>> >> >> Well, I think that when someone builds a good infrastructure he first needs to talk to all consumers and make sure it fits. >> In this case it seems like most work was done to fit the storage use-case, and now you check whether it can fit others as well.... >> >> IMO it makes much more sense to use events where possible (and you've promised to use those as well, but I don't see you doing that...). v2v should use events for sure, and they have promised to do that in the past, instead of using the v2v jobs. The reason events weren't used originally with the v2v feature, was that it was too risky and the events infrastructure was added too late in the game. > > > Revisiting and refactoring code which is already in use is always a bit of luxury we can rarely prioritize. So indeed v2v is not using events. The generalization work has been done to some extent, but there is no incentive to rewrite it completely. > On the other hand we are now trying to add events to migration progress reporting and hand over since that area is being touched due to post-copy enhancements. > So, when there is a practical chance to improve functionality by utilizing events it indeed should be the first choice +1. Well understood. > >> >> >>>> >>>> Do you "promise" to implement your "next gen API" for 4.1 as an alternative? >>> >>> >>> I guess we need the design first. >>> >>> >>>> On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke <alitke(a)redhat.com> wrote: >>>> >>>> On 05/12/16 11:17 +0200, Arik Hadas wrote: >>>> >>>> >>>> >>>> On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer <nsoffer(a)redhat.com> wrote: >>>> >>>> On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud < smelamud(a)redhat.com> >>>> wrote: >>>> > >>>> > Hi! >>>> > >>>> > I'm currently working on integration of virt-sysprep into oVirt. >>>> > >>>> > Usually, if user creates a template from a regular VM, and then >>>> creates >>>> new VMs from this template, these new VMs inherit all configuration >>>> of the >>>> original VM, including SSH keys, UDEV rules, MAC addresses, system >>>> ID, >>>> hostname etc. It is unfortunate, because you cannot have two network >>>> devices with the same MAC address in the same network, for example. >>>> > >>>> > To avoid this, user must clean all machine-specific configuration >>>> from >>>> the original VM before creating a template from it. You can do this >>>> manually, but there is virt-sysprep utility that does this >>>> automatically. >>>> > >>>> > Ideally, virt-sysprep should be seamlessly integrated into >>>> template >>>> creation process. But the first step is to create a simple button: >>>> user >>>> selects a VM, clicks the button and oVirt executes virt-sysprep on >>>> the VM. >>>> > >>>> > virt-sysprep works directly on VM's filesystem. It accepts list of >>>> all >>>> disks of the VM as parameters: >>>> > >>>> > virt-sysprep -a disk1.img -a disk2.img -a disk3.img >>>> > >>>> > The architecture is as follows: command on the Engine side runs a >>>> job on >>>> VDSM side and tracks its success/failure. The job on VDSM side runs >>>> virt-sysprep. >>>> > >>>> > The question is how to implement the job correctly? >>>> > >>>> > I thought about using storage jobs, but they are designed to work >>>> only >>>> with a single volume, correct? >>>> >>>> New storage verbs are volume based. This make it easy to manage >>>> them on the engine side, and will allow parallelizing volume >>>> operations >>>> on single or multiple hosts. >>>> >>>> A storage volume job is using sanlock lease on the modified volume >>>> and volume generation number. If a host running pending jobs becomes >>>> non-responsive and cannot be fenced, we can detect the state of >>>> the job, fence the job, and start the job on another host. >>>> >>>> In the SPM task, if a host becomes non-responsive and cannot be >>>> fenced, the whole setup is stuck, there is no way to perform any >>>> storage operation. >>>> > Is is possible to use them with operation that is performed on >>>> multiple >>>> volumes? >>>> > Or, alternatively, is it possible to use some kind of 'VM jobs' - >>>> that >>>> work on VM at whole? >>>> >>>> We can do: >>>> >>>> 1. Add jobs with multiple volumes leases - can make error handling >>>> very >>>> complex. How do tell a job state if you have multiple leases? >>>> which >>>> volume generation you use? >>>> >>>> 2. Use volume job using one of the volumes (the boot volume?). This >>>> does >>>> not protect the other volumes from modification but engine is >>>> responsible >>>> for this. >>>> >>>> 3. Use new "vm jobs", using a vm lease (should be available this >>>> week >>>> on master). >>>> This protects a vm during sysprep from starting the vm. >>>> We still need a generation to detect the job state, I think we >>>> can >>>> use the sanlock >>>> lease generation for this. >>>> >>>> I like the last option since sysprep is much like running a vm. >>>> > How v2v solves this problem? >>>> >>>> It does not. >>>> >>>> v2v predates storage volume jobs. It does not use volume leases and >>>> generation >>>> and does have any way to recover if a host running v2v becomes >>>> non-responsive >>>> and cannot be fenced. >>>> >>>> It also does not use the jobs framework and does not use a thread >>>> pool for >>>> v2v jobs, so it has no limit on the number of storage operations on >>>> a host. >>>> >>>> >>>> Right, but let's be fair and present the benefits of v2v-jobs as well: >>>> 1. it is the simplest "infrastructure" in terms of LOC >>>> >>>> >>>> It is also deprecated. V2V has promised to adopt the richer Host Jobs >>>> API in the future. >>>> >>>> >>>> 2. it is the most efficient mechanism in terms of interactions between >>>> the >>>> engine and VDSM (it doesn't require new verbs/call, the data is >>>> attached to >>>> VdsStats; probably the easiest mechanism to convert to events) >>>> >>>> >>>> Engine is already polling the host jobs API so I am not sure I agree >>>> with you here. >>>> >>>> >>>> 3. it is the most efficient implementation in terms of interaction with >>>> the >>>> database (no date is persisted into the database, no polling is done) >>>> >>>> >>>> Again, we're already using the Host Jobs API. We'll gain efficiency >>>> by migrating away from the old v2v API and having a single, unified >>>> approach (Host Jobs). >>>> >>>> >>>> Currently we have 3 mechanisms to report jobs: >>>> 1. VM jobs - that is currently used for live-merge. This requires the >>>> VM entity >>>> to exist in VDSM, thus not suitable for virt-sysprep. >>>> >>>> >>>> Correct, not appropriate for this application. >>>> >>>> >>>> 2. storage jobs - complicated infrastructure, targeted for recovering >>>> from >>>> failures to maintain storage consistency. Many of the things this >>>> infrastructure knows to handle is irrelevant for virt-sysprep flow, and >>>> the >>>> fact that virt-sysprep is invoked on VM rather than particular disk >>>> makes it >>>> less suitable. >>>> >>>> >>>> These are more appropriately called HostJobs and the have the >>>> following semantics: >>>> - They represent an external process running on a single host >>>> - They are not persisted. If the host or vdsm restarts, the job is >>>> aborted >>>> - They operate on entities. Currently storage is the first adopter >>>> of the infrastructure but virt was going to adopt these for the >>>> next-gen API. Entities can be volumes, storage domains, vms, >>>> network interfaces, etc. >>>> - Job status and progress is reported by the Host Jobs API. If a job >>>> is not present, then the underlying entitie(s) must be polled by >>>> engine to determine the actual state. >>>> >>>> >>>> 3. V2V jobs - no mechanism is provided to resume failed jobs, no >>>> leases, etc >>>> >>>> >>>> This is the old infra upon which Host Jobs are built. v2v has >>>> promised to move to Host Jobs in the future so we should not add new >>>> dependencies to this code. >>>> >>>> >>>> I have some arguments for using V2V-like jobs [1]: >>>> 1. creating template from vm is rarely done - if host goes unresponsive >>>> or any >>>> other failure is detected we can just remove the template and report >>>> the error >>>> >>>> >>>> We can chose this error handling with Host Jobs as well. >>>> >>>> >>>> 2. the phase of virt-sysprep is, unlike typical storage operation, >>>> short - >>>> reducing the risk of failures during the process >>>> >>>> >>>> Reduced risk of failures is never an excuse to have lax error >>>> handling. The storage flavored host jobs provide tons of utilities >>>> for making error handling standardized, easy to implement, and >>>> correct. >>>> >>>> >>>> 3. during the operation the VM is down - by locking the VM/template and >>>> its >>>> disks on the engine side, we render leases-like mechanism redundant >>>> >>>> >>>> Eventually we want to protect all operations on storage with sanlock >>>> leases. This is safer and allows for a more distributed approach to >>>> management. Again, the use of leases correctly in host jobs requires >>>> about 5 lines of code. The benefits of standardization far outweigh >>>> any perceived simplification resulting from omitting it. >>>> >>>> >>>> 4. in the worst case - the disk will not be corrupted (only some of the >>>> data >>>> might be removed). >>>> >>>> >>>> Again, the way engine chooses to handle job failures is independent of >>>> the mechanism. Let's separate that from this discussion. >>>> >>>> >>>> So I think that the mechanism for storage jobs is an over-kill for this >>>> case. >>>> We can keep it simple by generalise the V2V-job for other virt-tools >>>> jobs, like >>>> virt-sysprep. >>>> >>>> >>>> I think we ought to standardize on the Host Jobs framework where we >>>> can collaborate on unit tests, standardized locking and error >>>> handling, abort logic, etc. When v2v moves to host jobs then we will >>>> have a unified method of handling ephemeral jobs that are tied to >>>> entities. >>>> >>>> -- >>>> Adam Litke >>>> >>>> >>> >>> -- >>> Adam Litke >> >>

1 0

ovirt-node_ovirt-3.6_create-iso-el7_merged job failed
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi, The [1] job failed with the following error: *10:21:33 Package qemu-kvm-tools is obsoleted by qemu-kvm-tools-ev, trying to install 10:qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64 instead 10:21:48 Error creating Live CD : Failed to build transaction : 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires usbredir >= 0.7.1 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires seavgabios-bin >= 1.9.1-4 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires ipxe-roms-qemu >= 20160127-4 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires libusbx >= 1.0.19 10:21:48 ERROR: ISO build failed.* [1] - http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/171… Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

2 1

Vdsm tests are 4X times faster on travis
by Nir Soffer 07 Dec '16

07 Dec '16

HI all, Watching vdsm travis builds in the last weeks, it is clear that vdsm tests on travis are about 4X times faster compared with jenkins builds. Here is a typical build: ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/conso… travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079 The build took 4:34 on travis, and 19:34 on ovirt ci. This has a huge impact on vdsm maintainers. Having to wait 20 minutes for each patch means that we must ignore the ci and merge and hope that previous tests without rebasing on master were good enough. The builds are mostly the same, expect: - In travis we don't check if the build system was changed and packages should be built takes 9:18 minutes in ovirt ci. - In travis we don't clean or install anything before the test, we use a container with all the available packages, pulled from dockerhub. takes about 3:52 minutes in ovirt ci - In travis we don't enable coverage. Running the tests with coverage may slow down the tests takes 5:04 minutes in ovirt ci creating the coverage report takes only 15 seconds, not interesting - In travis we don't cleanup anything after the test this takes 34 seconds in ovirt ci The biggest problem is the build system check taking 9:18 minutes. fixing it will cut the build time in half. This is how time is spent in ovirt ci: 1. Starting (1:28) 00:00:00.001 Triggered by Gerrit: https://gerrit.ovirt.org/67338 00:00:00.039 [EnvInject] - Loading node environment variables. 00:00:00.056 Building remotely on fc24-vm15.phx.ovirt.org (phx nested local_disk fc24) in workspace /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 2. Installing packages (2:24) 00:01:28.338 INFO: installing package(s): autoconf automake gdb git libguestfs-tools-c libselinux-python3 libvirt-python3 m2crypto make mom openvswitch policycoreutils-python PyYAML python-blivet python-coverage python2-decorator python-devel python-inotify python-ioprocess python-mock python-netaddr python-pthreading python-setuptools python-six python-requests python3-decorator python3-netaddr python3-nose python3-six python3-yaml rpm-build sanlock-python sudo yum yum-utils 3. Setup in check-patch.sh (00:04) 00:03:52.838 + export VDSM_AUTOMATION=1 4. Running the actual tests 00:03:56.670 + ./autogen.sh --system --enable-hooks --enable-vhostmd 5. Installing python-debugifo (00:19) 00:04:15.385 Yum-utils package has been deprecated, use dnf instead. 6. Running make check (05:04) 00:04:30.948 + TIMEOUT=600 00:04:30.948 + make check NOSE_WITH_COVERAGE=1 NOSE_COVER_PACKAGE=/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/vdsm,/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib ... 00:09:33.981 tests: commands succeeded 00:09:33.981 congratulations :) 7. Creating coverage report (00:15) 00:09:34.017 + coverage html -d /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/exported-artifacts/htmlcov 8. Finding modified files (09:18) 00:09:49.213 + git diff-tree --no-commit-id --name-only -r HEAD 00:09:49.213 + egrep --quiet 'vdsm.spec.in|Makefile.am' 9. Cleaing up (00:27) 00:19:07.994 Took 915 seconds 00:19:34.973 Finished: SUCCESS

3 13

VDSM review & development tools
by Martin Polednik 07 Dec '16

07 Dec '16

Hello developers, this e-mail is mostly aimed at VDSM contributors (current and potential) and should serve as continuation of the idea started at our weekly meeting. We've had a proposal of moving the project to github[1] and using reviewable[2] as a code review due to github's code review interface. I would like to start a *discussion* regarding how we would like to develop VDSM in future. So far, the suggested options were (1st implied): 1) gerrit (stay as we are), 2) github & reviewable, 3) mailing list. What would be your favorite review & development tool and why? Do you hate any of them? Let the flamewar^Wconstructive discussion begin! :) <subjective part> My preferred tool is mailing list with the main tree mirrored to github. Why mailing list in 2016 (almost 2017)? a) stack - We're built on top of libvirt, libvirt is built on top of qemu and qemu utilizes kvm which is a kernel module. Each of this projects uses mailing lists for development. b) tooling - Everyone is free to use tools of choice. Any sane e-mail client can handle mailing list patches, and it's up to reviewers to choose the best way to handle the review. As for sending the patches, there is the universal fallback in the form of git-send-email. c) freedom - It's up to us to decide how would we handle such development. As many system-level projects already use mailing lists (see a), there is enough inspiration for workflow design[3]. d) accessibility - VDSM is in unfortunate position between "cool", high level projects and "boring" low level projects. I believe that we should be more accessible to the developers from below the stack rather than to general public. Having unified workflow that doesn't require additional accounts and is compatible with their workflows makes that easier. </subjective part> [1] https://github.com/ [2] https://reviewable.io/ [3] e.g. https://www.kernel.org/doc/Documentation/SubmittingPatches Regards, mpolednik

3 2

ovirt_4.0_he-system-tests job failed
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi, The [1] job failed with the following error: 2016-12-07 02:40:43,805::ssh.py::ssh::96::lago.ssh::DEBUG::Command 8b25fff2 on lago-he-basic-suite-4-0-host0 errors: Error: Package: ovirt-hosted-engine-setup-2.0.4.2-0.0.master.20161201083322.gita375924.el7.centos.noarch (alocalsync) Requires: ovirt-engine-sdk-python >= 3.6.3.0 [1] - http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/584/console Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

3 2

test-repo_ovirt_experimental_4.0
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi All, The [1] job is failing with the following error: *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*' Error: Package: ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch (alocalsync) Requires: ovirt-engine-sdk-python >= 3.6.0.0 [0m [36m * [Thread-4] Deploy VM lago-basic-suite-4-0-engine: [31mERROR [0m (in 0:01:51) [0m [31mError while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name <http://host.name>(), RuntimeError: /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh failed with status 1 on lago-basic-suite-4-0-engine [0m [36m # Deploy environment: [31mERROR [0m (in 0:01:51) [0m [36m@ Deploy oVirt environment: [31mERROR [0m (in 0:01:51) [0m [31mError occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 67, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name <http://host.name>(), RuntimeError: /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh failed with status 1 on lago-basic-suite-4-0-engine [0m * [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3416/console Could you please check this issue? Thanks in advanced, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

3 6

Re: [ovirt-devel] ovirt-appliance (today) is 1.5G...
by Ryan Barry 06 Dec '16

06 Dec '16

We may also want to do this with Node/RHV-H eventually (since the updates there are already 500MB). I don't think we currently ship deltarpm on RHV-H, but it's a very small addition. The question is whether makedeltarpm can actually do anything meaningful with something which is already a sparsified image wrapped in an RPM, but I'll test... On Sun, Oct 9, 2016 at 6:35 AM, Doron Fediuck <dfediuck(a)redhat.com> wrote: > FYI > > ---------- Forwarded message ---------- > From: Yaniv Kaul <ykaul(a)redhat.com> > Date: Thu, Oct 6, 2016 at 11:13 AM > Subject: [ovirt-devel] ovirt-appliance (today) is 1.5G... > To: devel <devel(a)ovirt.org> > > > I'm sure it was less just days ago, but now[1] it is again quite big. > 4.0's is 'only' 1G in size[2]. > Which anyway raises the question - should we provide delta RPMs for it? > > Thanks, > Y. > > [1] http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm > /el7/noarch/ovirt-engine-appliance-4.1-20160927.1.el7.centos.noarch.rpm > [2] http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el > 7/noarch/ovirt-engine-appliance-4.0-20161003.1.el7.centos.noarch.rpm > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > >

2 2

master/4.1 now requires el7.3 for 4.1 Cluster Level
by Michal Skrivanek 06 Dec '16

06 Dec '16

Dear developers, patch requiring 7.3-based qemu-kvm machine type[1] has been merged to master For using the 4.1 Cluster Level please make sure you have qemu-kvm-rhev 2.6 and libvirt 2.0+ on RHEL 7.3, CentOS 7.3 prerelase repo ([2], I suppose[3] too, and [4]), or Fedora 24+virt-preview repo If you are unable to update your hosts please do use 4.0 compatibility clusters Also note that previous 4.1 or master deployments, including oVirt 4.1 beta from last week, do not properly update the machine type in 4.1 clusters, so please update that manually in the db Thanks, michal [1] https://gerrit.ovirt.org/#/c/65979/ [2] http://buildlogs.centos.org/c7.1611.00/ [3] http://buildlogs.centos.org/c7.1611.01/ [4] https://cbs.centos.org/koji/buildinfo?buildID=13884

2 3

planned jenkins restart
by Evgheni Dereveanchin 05 Dec '16

05 Dec '16

Hi everyone, I am going to restart Jenkins in order to update several plugins. No new builds will be scheduled during this maintenance period. I will announce you once it is finished. Regards, Evgheni Dereveanchin

1 1

master is broken - can't add host - due to vdsm/gluster/__init__.pyc conflicts
by Yaniv Kaul 05 Dec '16

05 Dec '16

Getting: file /usr/lib/python2.7/site-packages/vdsm/gluster/__init__.pyc conflicts between attempted installs of vdsm-python-4.18.999-1059.git63f9e2f.el7.centos.noarch and vdsm-4.18.999-1059.git63f9e2f.el7.centos.x86_64 Thanks, Y.

5 7

[Call to Action] packaging for your preferred distribution
by Sandro Bonazzola 05 Dec '16

05 Dec '16

Hi, we released oVirt 4.1 beta last week providing RPMs for latest Red Hat Enterprise Linux release and its derivatives like CentOS Linux and Scientific Linux and Fedora or similar distributions. Virt support for Fedora is provided as tech preview / best effort being Fedora so fast moving which is too difficult to stay aligned with. We have still 2 months to GA so if you'd like to get oVirt 4.1 on your preferred distribution and you've packaging skills, this is a good chance to start packaging it playing with 4.1 beta released code. You can find some information about porting oVirt to your preferred distribution starting from [1] [1] https://www.ovirt.org/develop/developer-guide/porting-ovirt/ -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

[Call to Action] oVirt 4.1 translation
by Sandro Bonazzola 05 Dec '16

05 Dec '16

Hi, we released 4.1 beta 1 last week and we're going fast toward the GA currently scheduled for January 24th[1]. We have a scheduled string freeze on December 21st to allow translators to complete the translations but looking at zanata[2] I think it's time to go there and start translating to your preferred language. [1] http://www.ovirt.org/develop/release-management/releases/4.1/release-manage… [2] https://translate.zanata.org/project/view/ovirt?dswid=9807 -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM
by Yaniv Kaul 04 Dec '16

04 Dec '16

On Dec 4, 2016 10:57 PM, "Eyal Edri" <eedri(a)redhat.com> wrote: tests are back to stable, but with a cost of not testing 4.1 CL. DC level. But we are not explicitly using any of the 4.1 features for the time being. (I'd like to believe implictly we did, qcow2v3 for example). Iets hope we get centos 7.3 soon. Indeed - but we also need virt-builder image for it. Y. On Dec 4, 2016 22:41, "Yaniv Kaul" <ykaul(a)redhat.com> wrote: > > > On Dec 4, 2016 6:42 PM, "Arik Hadas" <ahadas(a)redhat.com> wrote: > > Yaniv will try to lower the cluster level used in the system-tests to 4.0 > - this is supposed to solve the issue. > > > Done. > Y. > > If it won't help (we will know it in about an hour), we'll add a db-script > that changes the rng device of the blank template only. > > On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <eedri(a)redhat.com> wrote: > >> FYI, >> >> I opened a bug [1] to track this issue since I don't see any attempts to >> resolve the issue on the thread, hopefully a bug will get more attention. >> Opened on VDSM since we see the libvirt error there, feel free to move >> product/team. >> >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303 >> >> On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri(a)redhat.com> wrote: >> >>> Not sure if relevant, but Juan posted a fix for SDK4 last time it >>> happened ( but different failure on log-collector ): >>> >>> https://gerrit.ovirt.org/#/c/67213/ >>> >>> * Added `urandom` to the `RngSource` enumerated type. >>> >>> On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri(a)redhat.com> wrote: >>> >>>> And its still failing from Friday, >>>> Since we don't have official Centos 7.3 repos yet ( hopefully we'll >>>> have it this week, but as of this moment its not published yet ) , we have >>>> to either revert the offending patch >>>> or send a quick fix. >>>> >>>> Right now all experimental flows for master are not working and nightly >>>> rpms are not refreshed with new RPMs. >>>> >>>> >>>> >>>> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com> wrote: >>>>> >>>>> Hello Martin. >>>>> >>>>> Do by outdated you mean the old libvirt? If so that is that livirt >>>>> available in CentOS 7.2? There is no 7.3 yet. >>>>> >>>>> >>>>> Right, this is the issue. >>>>> Y. >>>>> >>>>> >>>>> Anton. >>>>> >>>>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com> >>>>> wrote: >>>>> >>>>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote: >>>>>> >>>>>>> Hello All. >>>>>>> >>>>>>> Engine log can be viewed here: >>>>>>> >>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi >>>>>>> rt-engine/engine.log >>>>>>> >>>>>>> I see the following exception there: >>>>>>> >>>>>>> 2016-12-02 04:29:24,030-05 DEBUG >>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id": >>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true} >>>>>>> 2016-12-02 04:29:24,030-05 ERROR >>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) >>>>>>> [83b6b5d] Not able to update response for >>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c" >>>>>>> 2016-12-02 04:29:24,041-05 DEBUG >>>>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] >>>>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling >>>>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref >>>>>>> reshLightWeightData#-9223372036854775775 >>>>>>> as there is no unfired trigger. >>>>>>> 2016-12-02 04:29:24,024-05 DEBUG >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception: >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >>>>>>> at org.ovirt.engine.core.vdsbroke >>>>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>>>>> [vdsbroker.jar:] >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> 2016-12-02 04:29:24,042-05 ERROR >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for >>>>>>> VDSM response: Internal timeout occured >>>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >>>>>>> SCommand] >>>>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START, >>>>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0, >>>>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >>>>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5', >>>>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9 >>>>>>> 188-685b6c64a2f5]'}), >>>>>>> log id: 58f448b8 >>>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND >>>>>>> destination:jms.topic.vdsm_requests >>>>>>> reply-to:jms.topic.vdsm_responses >>>>>>> content-length:105 >>>>>>> >>>>>>> >>>>>>> Please note that this runs on localhost with local bridge. So it is >>>>>>> not >>>>>>> likely to be network itself. >>>>>>> >>>>>> >>>>>> The main issue I see is that the VM run command has actually failed >>>>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was >>>>>> done as engine patch and according to git log, posted around Mon Nov >>>>>> 28. Also adding Jakub - this should either not happen from engine's >>>>>> point of view or the lago host is outdated. >>>>>> >>>>>> [1] >>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log >>>>>> >>>>>> >>>>>> Anton. >>>>>>> >>>>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov < >>>>>>> amarchuk(a)redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>> FYI. Experimental flow for master currently fails to run a VM. The >>>>>>>> tests >>>>>>>> times out while waiting for 180 seconds: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ >>>>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/ >>>>>>>> >>>>>>>> This is reproducible over 23 runs of this happened tonight, sounds >>>>>>>> like a >>>>>>>> regression to me: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ >>>>>>>> >>>>>>>> I will update here with additional information once I find it. >>>>>>>> >>>>>>>> Last successful run was with this patch: >>>>>>>> >>>>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters >>>>>>>> fixup >>>>>>>> in a method) >>>>>>>> >>>>>>>> Known to start failing around this patch: >>>>>>>> >>>>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string >>>>>>>> formatting) >>>>>>>> >>>>>>>> Please notes that we do not have gating implemented yet, so >>>>>>>> everything >>>>>>>> that was merged in between those patches might have caused this (not >>>>>>>> necessary in vdsm project). >>>>>>>> >>>>>>>> Anton. >>>>>>>> -- >>>>>>>> Anton Marchukov >>>>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Anton Marchukov >>>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Anton Marchukov >>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> >>>> -- >>>> Eyal Edri >>>> Associate Manager >>>> RHV DevOps >>>> EMEA ENG Virtualization R&D >>>> Red Hat Israel >>>> >>>> phone: +972-9-7692018 >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>> >>> -- >>> Eyal Edri >>> Associate Manager >>> RHV DevOps >>> EMEA ENG Virtualization R&D >>> Red Hat Israel >>> >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > >

1 0

Experimental Flow for Master Fails to Run a VM
by Anton Marchukov 04 Dec '16

04 Dec '16

FYI. Experimental flow for master currently fails to run a VM. The tests times out while waiting for 180 seconds: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/testR… This is reproducible over 23 runs of this happened tonight, sounds like a regression to me: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ I will update here with additional information once I find it. Last successful run was with this patch: https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters fixup in a method) Known to start failing around this patch: https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string formatting) Please notes that we do not have gating implemented yet, so everything that was merged in between those patches might have caused this (not necessary in vdsm project). Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat

6 13

test-repo_ovirt_experimental_master job fails
by Shlomo Ben David 04 Dec '16

04 Dec '16

Hi All, The [1] job is failing since yesterday (Dec 3, 2016 6:29 PM) with the following error: *15:35:52* [36m@ Run test: 004_basic_sanity.py: [31mERROR [0m (in 0:04:44) [0m*15:35:52* [31mError occured, aborting*15:35:52* Traceback (most recent call last):*15:35:52* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in do_run*15:35:52* self.cli_plugins[args.ovirtverb].do_run(args)*15:35:52* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*15:35:52* self._do_run(**vars(args))*15:35:52* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper*15:35:52* return func(*args, **kwargs)*15:35:52* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper*15:35:52* return func(*args, prefix=prefix, **kwargs)*15:35:52* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 105, in do_ovirt_runtest*15:35:52* raise RuntimeError('Some tests failed')*15:35:52* RuntimeError: Some tests failed [0m >From the test results: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 138, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 339, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 231, in assert_true_within_short allowed_exceptions=allowed_exceptions, File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 223, in assert_true_within raise AssertionError('Timed out after %s seconds' % timeout) AssertionError: Timed out after 180 seconds Could you guide me please who should handle this issue? [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

1 0

VDSM changes Linux memory dirty ratios - why?
by Yaniv Kaul 04 Dec '16

04 Dec '16

It appears that VDSM changes the following params: vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 Any idea why? Because we use cache=none it's irrelevant anyway? TIA, Y.

4 9

ovirt-guest-agent job not building rpms
by Sandro Bonazzola 04 Dec '16

04 Dec '16

Hi, please have a look at http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-el7-x… ovirt-guest-agent job not building rpms Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

4 6

Request for oVirt Ansible modules testing feedback
by Ondra Machacek 02 Dec '16

02 Dec '16

Hello all, I would like to kindly ask everyone who is Ansible or oVirt user for testing of the new Ansible oVirt modules. For everyone who is familiar with the Ansible and oVirt, this[1] describes the steps you need to do, to setup oVirt modules library and start using those modules (Most of those modules will be available in Ansible 2.3, some of them are already in 2.2). If you have any issue setting this up, please contact me, I will do the best to help you. If you have an issue, which you think is a bug, please open an issue here[2]. Please note that Ansible is merging it's repositories, so since next week it will actually be stored here[3]. If you are missing anything please open an issue as well, or just contact me, and I will do fix it. You are also very welcome to sent PR with fixes. For those who don't have testing environment which can test against, I've created an Vagrant project which will deploy you the oVirt instance using Ansible playbooks. You can find how to use it here[4]. The repository also contains few examples[5], so you don't have to copy-paste them from the source. Thanks all for reading this and any feedback, Ondra [1] https://github.com/machacekondra/ovirt-tests/releases/tag/0.1 [2] https://github.com/ansible/ansible-modules-extras/issues [3] https://github.com/ansible/ansible [4] https://github.com/machacekondra/ovirt-tests [5] https://github.com/machacekondra/ovirt-tests/tree/master/examples

2 2

oVirt Community Newsletter: November 2016
by Brian Proffitt 02 Dec '16

02 Dec '16

oVirt's development is continuing on pace, as the calendar year draws to a close and we get ready for a new year of development, evangelism, and making virtual machine management a simple process for everyone. Here's what happened in November of 2016: ----------------- Software Releases ----------------- oVirt 4.0.6 Third Release Candidate is now available http://bit.ly/2gOxaDm oVirt 4.1.0 First Beta Release is now available for testing http://bit.ly/2gOtRfa ---------------- In the Community ---------------- Testing ovirt-engine changes without a real cluster http://www.ovirt.org/blog/2016/11/testing-ovirt-changes-without-cluster/ Request for oVirt Ansible modules testing feedback http://bit.ly/2gOByCg ------------------------------------ Deep Dives and Technical Discussions ------------------------------------ Important Open Source Cloud Products [German] http://bit.ly/2gOvWbd Red Hat IT runs OpenShift Container Platform on Red Hat Virtualization and Ansible http://red.ht/2ekHcLV Keynote: Blurring the Lines: The Continuum Between Containers and VMs [Video] http://bit.ly/2gOzutT Quick Guide: How to Plan Your Red Hat Virtualization 4.0 Deployment http://red.ht/2emiQkN A Decade of KVM [Chinese] http://bit.ly/2gOAuyp Expansion of iptables Rules for oVirt 4.0 [Russian] http://bit.ly/2gOBaUf -- Brian Proffitt Principal Community Analyst Open Source and Standards @TheTechScribe 574.383.9BKP

1 0

python path conflicts issues while running python scripts from vdsm verbs
by Ramesh Nachimuthu 02 Dec '16

02 Dec '16

Hi, I am trying to run a python script '/sbin/gluster-eventsapi' in vdsm verb which internally imports some python modules from /usr/lib/python2.7/site-packages/gluster/cliutils/. But it fails with the import error. Following error is seen in the supervdsm log. MainProcess|Thread-1::DEBUG::2016-11-28 16:54:35,130::commands::93::root::(execCmd) FAILED: <err> = 'Traceback (most recent call last):\n File "/sbin/gluster-eventsapi", line 25, in <module>\n from gluster.cliutils.cliutils import (Cmd, execute, node_output_ok, node_output_notok,\nImportError: No module named cliutils.cliutils\n'; <rc> = 1 I think the import statement "from gluster.cliutils.cliutils import (Cmd, execute, node_output_ok, node_output_notok)" in the python script resolves to '/usr/share/vdsm/gluster' instead of /usr/lib/python2.7/site-packages/gluster/cliutils. I see the following in python sys.path while executing a python script from vdsm. ['/usr/libexec/glusterfs', '/usr/share/vdsm', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages'] Looks like '/usr/share/vdsm' take precedence over '/usr/lib64/python2.7/site-packages'. Can someone suggests a way to fix this issue? Note: '/sbin/gluster-eventsapi' works perfectly while running directly from CLI. Related vdsm patch: https://gerrit.ovirt.org/#/c/67168/2/vdsm/gluster/events.py Regards, Ramesh

4 11

Deprecating Defaults for to_all_distros Setting in repoman
by Anton Marchukov 01 Dec '16

01 Dec '16

Hello All. If you use repoman please be aware that it currently has hardcoded list of packages that defaults to being distroless. This is controlled by "to_all_distros" configuration setting, but if you do not set it than the defaults are used. For ovirt experimental we decided that we better have all packages to have distro specified in spec. This also inline with most distro packaging guidelines. But then with current defaults repoman will consider those package to be distroless even when they are not and specifying an empty list in configuration is a silly way to override this. This change provides an incompatibility, but you can still use this feature by specifying to_all_distros package list explicitly in the config. So it is just the change of default values at this point: https://gerrit.ovirt.org/#/c/67662/ If somebody is not ok with defaults change and need time to update configuration please let me know. Also if you are not sure we can go another way and have intermediate repoman version that does not do the change, but issue deprecation warning if defaults are being used, so everybody can test their setups. However I think that it is might not be actually needed in this case as it is a configuration thing. Please feel free to comment on change request. Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat

2 2

Gerrit parallel patch handling and CI (Or, why did my code fail post-merge)
by Barak Korren 01 Dec '16

01 Dec '16

Hi there, I would like to address a concernt that had been raised to us by multiple developers, and reach an agreement on how (and if) to remedy it. Lets assume the following situation: We have a Git repo in Gerrit with top commit C0 in master. On time t0 developers Alice and Bob push patches P1 and P2 respectively to master so that we end up with the following situation in git: C0 <= P1 (this is Alice`s patch) C0 <= P2 (this is Bob`s patch) On time t1 CI runs for both patches checking the code as it looks for each patch. Lets assume CI is successful for both. On time t2 Alice submits her patch and Gerrit merges it, resulting in the following situation in master: C0 <= P1 On time t2 Bob submits his patch. Gerrit, seeing master has changed, re-bases the patch and merges it, the resulting situation (If the rebase is successful) is: C0 <= P1 <= P2 This means that the resulting code was never tested in CI. This, in turn, causes various failures to show up post-merge despite having pre-merge CI run successfully. This situation is a result of the way our repos are currently configured. Most repos ATM are configured with the "Rebase If Necessary" submit type. This means that Gerrit tries to automatically rebase patches as mentioned in t2 above. We could, instead, configure the repos to use the "Fast Forward Only" submit type. In that case, when Bob submits on t2, Gerrit refuses to merge and asks Bob to rebase (While offering a convenient button to do it). When he does, a new patch set gets pushed, and subsequently checked by CI. I recommend we switch all projects to use the "Fast Forward Only" submit type. Thoughts? Concerns? -- Barak Korren bkorren(a)redhat.com RHEV-CI Team

7 17

performance improvements and gwt-rpc switch
by Greg Sheremeta 01 Dec '16

01 Dec '16

Hi everyone, For the upcoming oVirt 4.1, the UX team has been focused hard on webadmin performance improvements. There have been some reports [1] [2] of UI sluggishness in both 3.6 and 4.0, usually after the browser had been open some time, and usually in scale environments. After some research, we determined that the primary cause of this sluggishness was memory leaks. We embarked on several weeks of hunting down memory leak bugs and squashing them. Alexander Wels and Vojtech Szocs led this work, and I helped test the performance of each patch as they created them. As they created patches to squash leaks, performance kept getting better and better. Today we've merged the last of our patches [*], and I'm happy to announce that we are now seeing much better UI performance on 4.1-master and 4.0.6. Over the course of several hours with the browser window open, users should see no sluggishness at all. [*] This last patch switches oVirt from using de-rpc to gwt-rpc in the frontend. This improves performance, but it also allows us to upgrade to GWT 2.8. We'd been previously blocked on that. If you're interested in UI performance testing, continue reading. If not, you can stop here :) ..... To verify our performance improvements, we took some simple measurements using selenium webdriver. The tests were unscientific, but very helpful. We ran a webdriver flow through oVirt that clicked some buttons and tabs and refreshed some grids. We did it a few hundred or thousand times. The tests were run using stubbed hosts (ovirt-vdsmfake) so that only the engine and UI were under test. Below are the important takeaways. The x axis is time, and each point on a graph is a loop through the same webdriver flow. The (ms) y axes are response times, and memory is in MB. In this graph, we compare oVirt 4.1 with and without our most impactful patch applied. As you can see, with the patch applied, response time stays flat for 200 loops of my test script over the course of 18 and 43 minutes. Without the patch applied, response time quickly degraded such that 200 loops of my test script took 1 hr 2 minutes vs. 18 minutes with the patch applied -- a 66% improvement! [image: Inline image 1] In this graph [ignore the spike], we tested oVirt hard for 6 hours 25 minutes (2000 loops). As you can see, the response times stay relatively flat over 6 hours! This is a great improvement. Do note that the memory is still growing, albeit much more slowly now. You can see towards the end of this run, maybe around hour 5, that the deviation starts to go up (the line thickens). Takeaway: maybe refresh your browser after many hours of having webadmin open. But, this is a stress test -- I'm betting users won't notice this slowdown after even 6 hours of regular webadmin use or idling. Last, here is a graph that shows gwt-rpc performing slightly better than de-rpc. Memory consumption is about the same -- gwt-rpc is just a faster rpc implementation. Reply with any questions or concerns. Thanks! Best wishes, Greg [1] https://bugzilla.redhat.com/show_bug.cgi?id=1368101 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1388462 -- Greg Sheremeta, MBA Red Hat, Inc. Sr. Software Engineer gshereme(a)redhat.com

6 8