December 2016 - Devel - oVirt List Archives

[VDSM] Tests leaking child processes --with-leaked-processes
by Nir Soffer 30 Dec '16

30 Dec '16

Hi all, The most common test failure on travis (and when running locally) is tests failing with EINTR: IOError: [Errno 4] Interrupted system call This happens because we are using zombiereaper, and it adds a signal handler for SIGCHLD. When a test leaks a child process and the child process terminates, the test runner get a SIGCHLD. If this happens when the test framework is writing to stdout, the write is interrupted and the test fail, failing the entire build. Here is an example failure on travis: https://travis-ci.org/oVirt/vdsm/jobs/187464715 To fix this we need to find the buggy tests (or tested code) leaking child processes. I added a new plugin for this here: https://gerrit.ovirt.org/69324 (mostly stolen from https://gerrit.ovirt.org/69040) Here is the result of running this plugin with current tests: $ make check NOSE_WITH_LEAKED_PROCESSES=1 ... ====================================================================== ERROR: test_running_repr (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3858" } ] -------------------- >> begin captured stdout << --------------------- <DirectioChecker /path running next_check=4321478.96 at 0x7f0f99812e90> --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,478 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,479 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,489 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616848> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_running_start_raises (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3861" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,513 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,514 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,524 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704979209232> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_repr (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3864" } ] -------------------- >> begin captured stdout << --------------------- <DirectioChecker /path stopping at 0x7f0f99812cd0> --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,548 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,548 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,557 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,558 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616592> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_start_raises (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3867" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,586 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,586 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,596 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,597 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616720> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_stopping_stop_ignored (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3870" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,621 DEBUG (MainThread) [storage.check] Checker '/path' started (check:233) 2016-12-30 02:21:32,622 DEBUG (MainThread) [storage.check] START check '/path' (delay=0.00) (check:283) 2016-12-30 02:21:32,632 DEBUG (MainThread) [storage.check] Checker '/path' stopping (check:250) 2016-12-30 02:21:32,632 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616912> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_taskset_missing (storage_check_test.TestDirectioChecker) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "3873" } ] -------------------- >> begin captured stdout << --------------------- [<CheckResult path=/var/tmp/tmpaIeNcE rc=127 err='Error starting process: [Errno 2] No such file or directory' time=4321479.14 elapsed=0.01 at 0x7f0f99a8bc90>] --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:32,658 DEBUG (MainThread) [storage.check] Checker '/var/tmp/tmpaIeNcE' started (check:233) 2016-12-30 02:21:32,658 DEBUG (MainThread) [storage.check] START check '/var/tmp/tmpaIeNcE' (delay=0.00) (check:283) 2016-12-30 02:21:32,662 DEBUG (MainThread) [storage.check] FINISH check '/var/tmp/tmpaIeNcE' (rc=127, elapsed=0.01) (check:328) 2016-12-30 02:21:32,663 INFO (MainThread) [storage.asyncevent] Stopping <EventLoop running=False closed=False at 0x139704976616272> (asyncevent:194) 2016-12-30 02:21:32,663 INFO (MainThread) [storage.asyncevent] Starting <EventLoop running=True closed=False at 0x139704976616272> (asyncevent:122) 2016-12-30 02:21:32,663 DEBUG (MainThread) [storage.asyncevent] <EventLoop running=False closed=False at 0x139704976616272> stopped (asyncevent:130) 2016-12-30 02:21:32,664 DEBUG (MainThread) [storage.asyncevent] Closing <EventLoop running=False closed=False at 0x139704976616272> (asyncevent:215) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testTimeout (storage_iscsi_test.RescanTimeoutTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "sleep 2", "pid": "4293" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.SamplingMethod] Trying to enter sampling method (vdsm.storage.iscsi.rescan) (misc:467) 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.SamplingMethod] Got in to sampling method (misc:470) 2016-12-30 02:21:37,279 DEBUG (MainThread) [storage.ISCSI] Performing SCSI scan, this will take up to 1 seconds (iscsi:457) 2016-12-30 02:21:37,279 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 2 (cwd None) (commands:69) 2016-12-30 02:21:38,285 DEBUG (MainThread) [storage.SamplingMethod] Returning last result (misc:477) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4367" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:39,392 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:39,402 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:241) 2016-12-30 02:21:39,403 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:244) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testCommunicate (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4370" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:39,426 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 /bin/dd (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testMutiWrite (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4373" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:40,463 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:256) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Writing more data to std out (storage_misc_test:258) 2016-12-30 02:21:40,473 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:261) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWaitCond (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4376" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:40,501 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 12 (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWaitTimeout (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4379" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:42,537 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 12 (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWriteLargeData (storage_misc_test.AsyncProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4382" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:44,572 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 cat (cwd None) (commands:69) 2016-12-30 02:21:44,580 INFO (MainThread) [AsyncProcTests] Writing data to std out (storage_misc_test:286) 2016-12-30 02:21:44,582 INFO (MainThread) [AsyncProcTests] Written data reading (storage_misc_test:289) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that stop really stops the copying process. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4413" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:21:45,963 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/nice -n 19 /usr/bin/ionice -c 3 /usr/bin/dd if=/tmp/tmpUsJOxV of=/tmp/tmpjQOhqP bs=64 seek=0 skip=0 conv=notrunc,fdatasync count=1 (cwd None) (commands:69) 2016-12-30 02:21:45,974 INFO (MainThread) [DdWatchCopy] Looks like it stopped! (storage_misc_test:465) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testNoCommandWithoutAffinity (storage_misc_test.ExecCmd) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4449" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:02,309 DEBUG (MainThread) [root] I.DONT.EXIST (cwd None) (commands:69) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that stopping the process really works. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "4582" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:06,000 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 10 (cwd None) (commands:69) 2016-12-30 02:22:06,005 INFO (MainThread) [WatchCmd] Looks like task stopped! (storage_misc_test:897) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testAmountOfInstancesPerPoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 93 --write-pipe-fd 92 --max-threads 10 --max-queued-requests 10", "pid": "4861" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:18,447 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:18,447 INFO (MainThread) [IOProcessClient] Starting client ioprocess-14 (__init__:325) 2016-12-30 02:22:18,447 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-14 (__init__:334) 2016-12-30 02:22:18,452 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-14 (__init__:386) 2016-12-30 02:22:18,452 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-14 started (__init__:398) 2016-12-30 02:22:18,452 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:18,452 INFO (MainThread) [IOProcessClient] Starting client ioprocess-15 (__init__:325) 2016-12-30 02:22:18,453 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-15 (__init__:334) 2016-12-30 02:22:18,453 INFO (ioprocess communication (4848)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:18,453 DEBUG (ioprocess communication (4848)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:18,453 DEBUG (ioprocess communication (4848)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:18,457 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-15 (__init__:386) 2016-12-30 02:22:18,458 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-15 started (__init__:398) 2016-12-30 02:22:18,458 INFO (ioprocess communication (4854)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:18,458 DEBUG (ioprocess communication (4854)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:18,458 DEBUG (ioprocess communication (4854)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:24,464 DEBUG (MainThread) [storage.oop] Removing idle ioprocess A (outOfProcess:77) 2016-12-30 02:22:24,464 INFO (MainThread) [IOProcessClient] Closing client ioprocess-15 (__init__:593) 2016-12-30 02:22:24,464 DEBUG (MainThread) [storage.oop] Creating ioprocess B (outOfProcess:87) 2016-12-30 02:22:24,465 INFO (MainThread) [IOProcessClient] Starting client ioprocess-16 (__init__:325) 2016-12-30 02:22:24,465 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-16 (__init__:334) 2016-12-30 02:22:24,470 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-16 (__init__:386) 2016-12-30 02:22:24,470 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-16 started (__init__:398) 2016-12-30 02:22:24,515 INFO (ioprocess communication (4861)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:24,515 DEBUG (ioprocess communication (4861)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:24,516 DEBUG (ioprocess communication (4861)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,557 INFO (MainThread) [IOProcessClient] Closing client ioprocess-14 (__init__:593) 2016-12-30 02:22:25,557 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-14 (__init__:599) 2016-12-30 02:22:25,557 INFO (ioprocess communication (4848)) [IOProcessClient] shutdown requested (__init__:124) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testDifferentPoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 109 --write-pipe-fd 108 --max-threads 10 --max-queued-requests 10", "pid": "4882" }, { "command": "/usr/libexec/ioprocess --read-pipe-fd 101 --write-pipe-fd 99 --max-threads 10 --max-queued-requests 10", "pid": "4876" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:25,583 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:25,584 INFO (MainThread) [IOProcessClient] Starting client ioprocess-17 (__init__:325) 2016-12-30 02:22:25,584 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-17 (__init__:334) 2016-12-30 02:22:25,592 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-17 (__init__:386) 2016-12-30 02:22:25,593 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-17 started (__init__:398) 2016-12-30 02:22:25,593 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:25,593 INFO (ioprocess communication (4870)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,593 DEBUG (ioprocess communication (4870)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,594 INFO (MainThread) [IOProcessClient] Starting client ioprocess-18 (__init__:325) 2016-12-30 02:22:25,594 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-18 (__init__:334) 2016-12-30 02:22:25,594 DEBUG (ioprocess communication (4870)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,604 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-18 (__init__:386) 2016-12-30 02:22:25,604 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-18 started (__init__:398) 2016-12-30 02:22:25,605 DEBUG (MainThread) [storage.oop] Creating ioprocess B (outOfProcess:87) 2016-12-30 02:22:25,605 INFO (ioprocess communication (4876)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,605 DEBUG (ioprocess communication (4876)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,605 INFO (MainThread) [IOProcessClient] Starting client ioprocess-19 (__init__:325) 2016-12-30 02:22:25,605 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-19 (__init__:334) 2016-12-30 02:22:25,605 DEBUG (ioprocess communication (4876)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,610 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-19 (__init__:386) 2016-12-30 02:22:25,610 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-19 started (__init__:398) 2016-12-30 02:22:25,611 INFO (MainThread) [IOProcessClient] Closing client ioprocess-16 (__init__:593) 2016-12-30 02:22:25,611 INFO (MainThread) [IOProcessClient] Closing client ioprocess-17 (__init__:593) 2016-12-30 02:22:25,611 INFO (ioprocess communication (4882)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,611 DEBUG (ioprocess communication (4882)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,611 INFO (ioprocess communication (4870)) [IOProcessClient] shutdown requested (__init__:124) 2016-12-30 02:22:25,612 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-17 (__init__:599) 2016-12-30 02:22:25,612 DEBUG (ioprocess communication (4882)) [IOProcess] Opening communication channels... (__init__:445) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testSamePoolName (storage_outofprocess_test.OopWrapperTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "/usr/libexec/ioprocess --read-pipe-fd 96 --write-pipe-fd 94 --max-threads 10 --max-queued-requests 10", "pid": "4918" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:25,718 DEBUG (MainThread) [storage.oop] Creating ioprocess Global (outOfProcess:87) 2016-12-30 02:22:25,719 INFO (MainThread) [IOProcessClient] Starting client ioprocess-22 (__init__:325) 2016-12-30 02:22:25,719 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-22 (__init__:334) 2016-12-30 02:22:25,723 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-22 (__init__:386) 2016-12-30 02:22:25,724 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-22 started (__init__:398) 2016-12-30 02:22:25,724 DEBUG (MainThread) [storage.oop] Creating ioprocess A (outOfProcess:87) 2016-12-30 02:22:25,724 INFO (MainThread) [IOProcessClient] Starting client ioprocess-23 (__init__:325) 2016-12-30 02:22:25,724 DEBUG (MainThread) [IOProcessClient] Starting ioprocess for client ioprocess-23 (__init__:334) 2016-12-30 02:22:25,724 INFO (ioprocess communication (4912)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,724 DEBUG (ioprocess communication (4912)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,725 DEBUG (ioprocess communication (4912)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,729 DEBUG (MainThread) [IOProcessClient] Starting communication thread for client ioprocess-23 (__init__:386) 2016-12-30 02:22:25,730 DEBUG (MainThread) [IOProcessClient] Communication thread for client ioprocess-23 started (__init__:398) 2016-12-30 02:22:25,730 INFO (MainThread) [IOProcessClient] Closing client ioprocess-18 (__init__:593) 2016-12-30 02:22:25,730 INFO (ioprocess communication (4918)) [IOProcess] Starting ioprocess (__init__:447) 2016-12-30 02:22:25,730 DEBUG (ioprocess communication (4918)) [IOProcess] Closing unrelated FDs... (__init__:445) 2016-12-30 02:22:25,730 DEBUG (ioprocess communication (4918)) [IOProcess] Opening communication channels... (__init__:445) 2016-12-30 02:22:25,731 INFO (MainThread) [IOProcessClient] Closing client ioprocess-22 (__init__:593) 2016-12-30 02:22:25,731 INFO (ioprocess communication (4912)) [IOProcessClient] shutdown requested (__init__:124) 2016-12-30 02:22:25,731 DEBUG (MainThread) [IOProcessClient] Waiting for communication thread for ioprocess-22 (__init__:599) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_get (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7676" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,549 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7676 (cwd None) (commands:69) 2016-12-30 02:22:54,555 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,555 DEBUG (MainThread) [root] /usr/bin/taskset --pid 567 (cwd None) (commands:69) 2016-12-30 02:22:54,562 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([0, 3])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7684" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,652 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7684 (cwd None) (commands:69) 2016-12-30 02:22:54,658 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([0])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7689" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,701 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7689 (cwd None) (commands:69) 2016-12-30 02:22:54,707 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_child(frozenset([1, 2])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7694" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,752 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7694 (cwd None) (commands:69) 2016-12-30 02:22:54,763 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([0, 3])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7699" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,797 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 0,3 7699 (cwd None) (commands:69) 2016-12-30 02:22:54,808 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,808 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7699 (cwd None) (commands:69) 2016-12-30 02:22:54,813 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([0])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7704" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,849 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 0 7704 (cwd None) (commands:69) 2016-12-30 02:22:54,855 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,855 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7704 (cwd None) (commands:69) 2016-12-30 02:22:54,861 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: test_set_from_parent(frozenset([1, 2])) (tasksetTests.AffinityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "7710" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:22:54,909 DEBUG (MainThread) [root] /usr/bin/taskset --pid --cpu-list 1,2 7710 (cwd None) (commands:69) 2016-12-30 02:22:54,920 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 2016-12-30 02:22:54,920 DEBUG (MainThread) [root] /usr/bin/taskset --pid 7710 (cwd None) (commands:69) 2016-12-30 02:22:54,925 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '1'], ['sleep', '3'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9495" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:34,072 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 1 (cwd None) (v2v:1301) 2016-12-30 02:23:34,082 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:37,086 DEBUG (MainThread) [root] Killing pid=9494 (v2v:679) 2016-12-30 02:23:37,086 DEBUG (MainThread) [root] Killing pid=9495 (v2v:679) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '3'], ['sleep', '1'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9498" }, { "command": "", "pid": "9499" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:37,112 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:37,122 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 1 (cwd None) (v2v:1301) 2016-12-30 02:23:39,134 DEBUG (MainThread) [root] Killing pid=9498 (v2v:679) 2016-12-30 02:23:39,135 DEBUG (MainThread) [root] Killing pid=9499 (v2v:679) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: testWait(['sleep', '3'], ['sleep', '3'], False) (v2vTests.PipelineProcTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) AssertionError: Test leaked child processes: [ { "command": "", "pid": "9503" }, { "command": "", "pid": "9504" } ] -------------------- >> begin captured logging << -------------------- 2016-12-30 02:23:39,164 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:39,176 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 3 (cwd None) (v2v:1301) 2016-12-30 02:23:41,186 DEBUG (MainThread) [root] Killing pid=9503 (v2v:679) 2016-12-30 02:23:41,187 DEBUG (MainThread) [root] Killing pid=9504 (v2v:679) --------------------- >> end captured logging << ---------------------

1 0

Jenkins jobs failure
by Gil Shinar 29 Dec '16

29 Dec '16

Hi all, We have have a merged a patch yesterday that made a very important enhancement to the logs collection of the STD-CI jobs. This merge caused a small regression that we have already found and working on fixing it. Please be patient till this fix will be merged. Thanks Gil

1 1

Jenkins restart
by Gil Shinar 28 Dec '16

28 Dec '16

Hi all, Jenkins restart is planned today at 18:00. Patches that will be sent or merged up to an hour before that time might fail and would need to be re-triggered. If there's any urgent patch that needs to be sent or merged at that time, please let me know before 18:00. Thanks Gil

1 1

Re: [ovirt-devel] ost host addition failure
by Eyal Edri 27 Dec '16

27 Dec '16

Any updates? The tests are still failing on vdsmd won't start from Sunday... master repos havn't been refreshed for a few days due to this. from host deploy log: [1] basic-suite-master-engine/_var_log_ovirt-engine/host-deploy/ovirt-host-deploy-20161227012930-192.168.201.4-14af2bf0.log the job links [2] [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/lastComple… 016-12-27 01:29:29 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stdout: 2016-12-27 01:29:29 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stderr: A dependency job for vdsmd.service failed. See 'journalctl -xe' for details. 2016-12-27 01:29:29 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/tmp/ovirt-QZ1ucxWFfm/pythonlib/otopi/context.py", line 132, in _executeMethod method['method']() File "/tmp/ovirt-QZ1ucxWFfm/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 209, in _start self.services.state('vdsmd', True) File "/tmp/ovirt-QZ1ucxWFfm/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'vdsmd' 2016-12-27 01:29:29 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Closing up': Failed to start service 'vdsmd' 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-12-27 01:29:29 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/excep [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/lastComple… On Sun, Dec 25, 2016 at 11:31 AM, Eyal Edri <eedri(a)redhat.com> wrote: > We should see it fixed here hopefully [1] > > > [1] http://jenkins.ovirt.org/view/All%20Running%20jobs/job/ > test-repo_ovirt_experimental_master/4412/console > > On Sun, Dec 25, 2016 at 11:19 AM, Dan Kenigsberg <danken(a)redhat.com> > wrote: > >> On Sun, Dec 25, 2016 at 10:28 AM, Yaniv Kaul <ykaul(a)redhat.com> wrote: >> > >> > >> > On Sun, Dec 25, 2016 at 9:47 AM, Dan Kenigsberg <danken(a)redhat.com> >> wrote: >> >> >> >> Correct. https://gerrit.ovirt.org/#/c/69052/ >> >> >> >> Can you try adding >> >> lago shell "$vm_name" -c "mkdir -p /var/log/ovirt-imageio-daemon/ && >> >> chown vdsm:kvm /var/log/ovirt-imageio-daemon/" >> > >> > >> > How will it know what is the vdsm user before installing vdsm? >> >> You're right. a hack would have to `chmod a+rwx >> /var/log/ovirt-imageio-daemon/` instead. >> >> > Why not either: >> > 1. Fix it >> >> yes, that's why we've opened >> https://bugzilla.redhat.com/show_bug.cgi?id=1400003 ; now a fix is >> getting merged. I don't know when it is going to be ready in lago's >> repos. >> >> > -or- >> > 2. Revert the offending patch? >> >> I'm not aware of such patch. It's a race that has been there since >> ever, and I don't know why it suddenly pops up so often. >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

4 5

[VDSM] storage_iscsi_test.RescanTimeoutTests test failing
by Edward Haas 26 Dec '16

26 Dec '16

I got this test failing on one of the builds: ( http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5266/console ) *09:39:16* ======================================================================*09:39:16* FAIL: testTimeout (storage_iscsi_test.RescanTimeoutTests)*09:39:16* ----------------------------------------------------------------------*09:39:16* Traceback (most recent call last):*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/monkeypatch.py", line 133, in wrapper*09:39:16* return f(*args, **kw)*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/monkeypatch.py", line 133, in wrapper*09:39:16* return f(*args, **kw)*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_iscsi_test.py", line 45, in testTimeout*09:39:16* iscsi.rescan()*09:39:16* File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__*09:39:16* self.gen.next()*09:39:16* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_iscsi_test.py", line 33, in assertMaxDuration*09:39:16* (elapsed, maxtime))*09:39:16* AssertionError: Operation was too slow 1.32s > 1.20s*09:39:16* -------------------- >> begin captured logging << --------------------*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.SamplingMethod] Trying to enter sampling method (vdsm.storage.iscsi.rescan) (misc:467)*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.SamplingMethod] Got in to sampling method (misc:470)*09:39:16* 2016-12-26 09:36:46,857 DEBUG (MainThread) [storage.ISCSI] Performing SCSI scan, this will take up to 1 seconds (iscsi:457)*09:39:16* 2016-12-26 09:36:46,858 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 sleep 2 (cwd None) (commands:69)*09:39:16* 2016-12-26 09:36:48,183 DEBUG (MainThread) [storage.SamplingMethod] Returning last result (misc:477)*09:39:16* --------------------- >> end captured logging << --------------------- I was surprised that it uses a time measurement to determine success. Edy.

1 0

Check-merged is broken since we added network test to it
by Yaniv Bronheim 26 Dec '16

26 Dec '16

Hi guys and Leon, https://gerrit.ovirt.org/#/c/68078/ broke the check-merged job with really tough exception: *15:23:34* sh: [17766: 1 (255)] tcsetattr: Inappropriate ioctl for device*15:23:34* Took 2586 seconds*15:23:34* Slave went offline during the build <http://jenkins.ovirt.org/computer/vm0136.workers-phx.ovirt.org/log>*15:23:34* ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel*15:23:34* at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)*15:23:34* Caused by: java.io.EOFException*15:23:34* at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)*15:23:34* at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)*15:23:34* at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)*15:23:34* at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)*15:23:34* at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)*15:23:34* at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)*15:23:34* at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)*15:23:34* *15:23:34* Build step 'Execute shell' marked build as failure*15:23:34* Performing Post build task... I have no clue what causes it.. we need to investigate in the tests code. you can see it in http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/772/consol… and this for a job run just before it got in, which worked well - http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/688/console I suggest to revert this patch (and backport the revert to ovirt-4.1 branch as well) until figuring what causes it -- *Yaniv Bronhaim.*

4 9

oVirt system tests currently failing to AddHost on master
by Barak Korren 25 Dec '16

25 Dec '16

It hard to tell currently when did this start b/c we had so package issues that made the tests fail before reaching that point most of the day. Since we currently have an issue in Lago with collecting AddHost logs (Hopefully we'll resolve this in the next release early next week), I`ve ran the tests locally and attached the bundle of generated logs to this message. Included in the attached file are engine logs, host-deploy logs and VDSM logs for both test hosts. >From a quick look inside it seems the issue is with VDSM failing to start. Thanks, -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

5 15

ovirt.org is back online
by Eyal Edri 25 Dec '16

25 Dec '16

The site is back online thanks for Duck in helping clearing the log files which caused the gear to get stuck. I understood that the site is planned to move to another location, with more storage so hopefully these kind of outage will be rare soon after. -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[Attention] Gerrit 'git://' protocol error
by Nadav Goldin 25 Dec '16

25 Dec '16

Hi, Currently git-daemon is not working on gerrit.ovirt.org, so all actions with remote configured with 'git://' prefix would fail(including Jenkins jobs). We are working on resolving the issue, until then you can use 'https://' for anonymous clones, or SSH(ssh://username@gerrit.ovirt.org:29418/repo.git). Thanks, Nadav.

1 1

[VDSM] Support for external leases - possible breakage with development version
by Nir Soffer 24 Dec '16

24 Dec '16

Hi all, I merged today the last patches for supporting external leases. In storage domain version 4 (the default in ovirt 4.1), there is now a new volume named "xleases". When upgrading older storage domain (version 3), we create and format the new volume. If you used a development version with storage domain version 4, your storage domain may not work with current master. You should recreated such storage domain. As a workaround, you can create the special volume manually: File storage: sudo -u vdsm truncate -s 1G /rhev/data-center/mnt/server:_path/domain_uuid/dom_md/xleases Block storage: lvcreate --name xleases --size 1g domain_uuid Note that the volume should be also be formatted to actually use external leases, but having it should be good enough to keep the system happy. Thanks, Nir

1 1

Re: [ovirt-devel] System tests for 4.1 currently failing to run VMs!
by Oved Ourfali 22 Dec '16

22 Dec '16

Why not run it via Jenkins for patches? Like, if you add a comment saying "run: ost" it will run it? It do it automatically based on another thing? On Dec 21, 2016 17:42, "Eyal Edri" <eedri(a)redhat.com> wrote: > > > > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >> >> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul(a)redhat.com> wrote: >>> >>> >>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >>>> >>>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: >>>>> >>>>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <eedri(a)redhat.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra < vfeenstr(a)redhat.com> wrote: >>>>>>> >>>>>>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <bkorren(a)redhat.com> wrote: >>>>>>>> >>>>>>>> The test for running VMs had been failing since yesterday. >>>>>>>> >>>>>>>> The patch merged before the failures started was: >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) while running >>>>>>>> this API call: >>>>>>>> >>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up' >>>>>>> >>>>>>> >>>>>>> This is a REST API call, the patch above is Frontend. So this is unrelated. >>>>>>> >>>>>>> However on Host 0 I can see this: >>>>>>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process failed (vm:615) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: internal error: process exited while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) >>>>> >>>>> >>>>> it is likely related to the recent USB patches >>>>> investigating >>>> >>>> >>>> hm, there are multiple problems (features/bugs depending on prefferred point of view:) >>>> but there is an easy “fix” taking care of this particular problem, so we can start with that and figure out the proper approach later >>>> arik will push that and merge it soon, likely today >>> >>> >>> Thanks - if there is a quicker way to resolve this by reverting, I think it's a better option. >> >> >> I really need to talk you out of this approach:-) >> It does sound tempting and logical, but with our development model of large patch series combined with late detection it really is quite risky. Here it wouldn’t help much…and figuring out the right revert patch is more complicated then fixing it. > > > Can we start asking developers run OST before they merge so it will be early detection and not late detection? > We have video sessions on how to use OST, so it shouldn't be any issues in running it on a patch. > >> >> I believe the best is to identify it early and notify the maintainer who merged that patch ASAP, as that person is in the best position to asses if revert is safe or if there is a simple follow up patch he can push right away >> >> We can surely improve on reporting, so Barak, how/why did you point to that particular patch in your email? It should start failing on 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 01:56:07 2016 -0500) >> Even though it may be that it was hidden because of c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 15:16:40 2016 -0500) >> >> (fix is ready, waiting on CI now) >> >> Thanks, >> michal >> >>> Y. >>> >>>> >>>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: internal error: process exited while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping connection (guestagent:430) >>>>>>> >>>>>>> >>>>>>> >>>>>>> And on The engine loads of these: >>>>>>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call 2016-12-20 16:53:57,849-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [7971dfb4] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}� 2016-12-20 16:53:57,850-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:287) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1389) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2053) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:170) at org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(BackendHostResource.java:212) at org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:116) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM response: Internal timeout occured >>>>>>> >>>>>>> >>>>>>> This is definitely not related to the patch linked above. >>>>>>> >>>>>>> However I am also not quite sure what is the cause of this right now, it might be a breaking change in VDSM. >>>>>>> >>>>>> >>>>>> The DC level is 4.0, is it possible that is the reason for the failure? >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Full test code can be seen here: >>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-sui… >>>>>>>> >>>>>>>> Full test exception can be seen here: >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport… >>>>>>>> >>>>>>>> Further logs can be seen in Jenkins: >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/e… >>>>>>>> >>>>>>>> -- >>>>>>>> Barak Korren >>>>>>>> bkorren(a)redhat.com >>>>>>>> RHCE, RHCi, RHV-DevOps Team >>>>>>>> https://ifireball.wordpress.com/ >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list >>>>>>>> Devel(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Eyal Edri >>>>>> Associate Manager >>>>>> RHV DevOps >>>>>> EMEA ENG Virtualization R&D >>>>>> Red Hat Israel >>>>>> >>>>>> phone: +972-9-7692018 >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

6 11

Re: [ovirt-devel] [RFE] treat local NFS storage as localfs
by Pavel Gashev 22 Dec '16

22 Dec '16

On Wed, 2016-12-21 at 16:25 +0100, Martin Sivak wrote: > > So VMs on one host will get better IO performance and the others will > still use NFS as they do now. Performance is not an issue when all I/O bound VMs are co-located with disks. If all VMs are I/O bound, all hosts in a cluster can use local storage. > It is an interesting idea, I am just not sure if having poor-man's > hyperconverged setup with all the drawbacks of NFS is worth it. > Imagine for example what happens when that storage provider host > needs > to be fenced or put into maintenance. The whole cluster would go down > (all VMs would lose storage connection, not just the VMs from the > affected host). Losing a server together with a storage could be an issue, or not. It really depends on setup. At least it would be good to provide such feature :) > I will let someone from the storage team to respond to this, but I do > not think that trading performance (each host has its own local > storage) and resilience (well, at least one failing host does not > affect the others) for migrations is a good deal. > > -- > Martin Sivak > SLA / oVirt > > On Wed, Dec 21, 2016 at 2:18 PM, Sven Kieske <s.kieske(a)mittwald.de> > wrote: > > > > On 21/12/16 11:44, Pavel Gashev wrote: > > > > > > Hello, > > > > > > I'd like to introduce a RFE that allows to use a local storage in > > > multi > > > server environments https://bugzilla.redhat.com/show_bug.cgi?id=1 > > > 406412 > > > > > > Most servers have a local storage. Some servers have very > > > reliable > > > storages with hardware RAID controllers and battery units. > > > > > > Example user cases: > > > https://www.mail-archive.com/users@ovirt.org/msg36719.html > > > https://www.mail-archive.com/users@ovirt.org/msg36772.html > > > > > > The best way to use local storage in multi server "shared" > > > datacenters > > > is exporting it over NFS. Using NFS allows to move disks and VMs > > > among > > > servers. > > > > > > In order to improve performance, disk I/O bound VMs can be pinned > > > to > > > a host with local storage. However there still is a performance > > > drawback of NFS layers. Treating a local NFS storage as a local > > > storage > > > improves performance for VMs pinned to host. > > > > > > Currently setting up of NFS exports is out of scope of oVirt. > > > However > > > this would be a way to get rid of "Local/Shared" storage types of > > > datacenter. So that all storages are shared, but local storages > > > are > > > used as local. > > > > > > Any questions/comments are welcome. > > > > > > Specifically I'd like to request for comment on potential data > > > integrity issues during online VM or disk migration between NFS > > > and > > > localfs. > > > > > > > Just let me say that I really like this as an end user. > > > > Hope this get's in. This seems less overhead than a complete > > hyperconverged gluster setup. > > > > > > -- > > Mit freundlichen Grüßen / Regards > > > > Sven Kieske > > > > Systemadministrator > > Mittwald CM Service GmbH & Co. KG > > Königsberger Straße 6 > > 32339 Espelkamp > > T: +495772 293100 > > F: +495772 293333 > > https://www.mittwald.de > > Geschäftsführer: Robert Meyer > > St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad > > Oeynhausen > > Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad > > Oeynhausen > > > > > > _______________________________________________ > > Devel mailing list > > Devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

1 0

Re: [ovirt-devel] [RFE] treat local NFS storage as localfs
by Michal Skrivanek 22 Dec '16

22 Dec '16

> On 21 Dec 2016, at 16:26, Martin Sivak <msivak(a)redhat.com> wrote: > > Hi, > >> Hope this get's in. This seems less overhead than a complete >> hyperconverged gluster setup. > > But NFS still is a single point of failure. Hyperconverged is supposed > to address that. > >>> In order to improve performance, disk I/O bound VMs can be pinned to >>> a host with local storage. However there still is a performance >>> drawback of NFS layers. Treating a local NFS storage as a local storage >>> improves performance for VMs pinned to host. > > So VMs on one host will get better IO performance and the others will > still use NFS as they do now. > > It is an interesting idea, I am just not sure if having poor-man's > hyperconverged setup with all the drawbacks of NFS is worth it. > Imagine for example what happens when that storage provider host needs > to be fenced or put into maintenance. The whole cluster would go down > (all VMs would lose storage connection, not just the VMs from the > affected host). > > I will let someone from the storage team to respond to this, but I do > not think that trading performance (each host has its own local > storage) and resilience (well, at least one failing host does not > affect the others) for migrations is a good deal. If disk performance is critical then there is an option to use direct access on local host using either PCI passthrough of a local storage controller or SCSI passthrough of LUNs. > > -- > Martin Sivak > SLA / oVirt > >> On Wed, Dec 21, 2016 at 2:18 PM, Sven Kieske <s.kieske(a)mittwald.de> wrote: >>> On 21/12/16 11:44, Pavel Gashev wrote: >>> Hello, >>> >>> I'd like to introduce a RFE that allows to use a local storage in multi >>> server environments https://bugzilla.redhat.com/show_bug.cgi?id=1406412 >>> >>> Most servers have a local storage. Some servers have very reliable >>> storages with hardware RAID controllers and battery units. >>> >>> Example user cases: >>> https://www.mail-archive.com/users@ovirt.org/msg36719.html >>> https://www.mail-archive.com/users@ovirt.org/msg36772.html >>> >>> The best way to use local storage in multi server "shared" datacenters >>> is exporting it over NFS. Using NFS allows to move disks and VMs among >>> servers. >>> >>> In order to improve performance, disk I/O bound VMs can be pinned to >>> a host with local storage. However there still is a performance >>> drawback of NFS layers. Treating a local NFS storage as a local storage >>> improves performance for VMs pinned to host. >>> >>> Currently setting up of NFS exports is out of scope of oVirt. However >>> this would be a way to get rid of "Local/Shared" storage types of >>> datacenter. So that all storages are shared, but local storages are >>> used as local. >>> >>> Any questions/comments are welcome. >>> >>> Specifically I'd like to request for comment on potential data >>> integrity issues during online VM or disk migration between NFS and >>> localfs. >>> >> >> Just let me say that I really like this as an end user. >> >> Hope this get's in. This seems less overhead than a complete >> hyperconverged gluster setup. >> >> >> -- >> Mit freundlichen Grüßen / Regards >> >> Sven Kieske >> >> Systemadministrator >> Mittwald CM Service GmbH & Co. KG >> Königsberger Straße 6 >> 32339 Espelkamp >> T: +495772 293100 >> F: +495772 293333 >> https://www.mittwald.de >> Geschäftsführer: Robert Meyer >> St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen >> Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen >> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > >

3 3

Branching Vdsm to ovirt-4.1
by Yaniv Bronheim 22 Dec '16

22 Dec '16

Hello devels, I just introduced ovirt-4.1 branch in vdsm - which will be the base for ovirt-4.1 alpha build. Now master is tagged v4.20.0 and first ovirt-4.1 tag is v4.19.1. Basically it means that every fix that should be in 4.1 needs to be backported and merged to ovirt-4.1 branch by fromani or me. We both in #vdsm so ping us if needed and add us to the patches. Bare in mind that in most of the cases we will ask for Bug-Url link in the commit message and same commit-id as in the master branch patch. For any further questions or wonders feel free to reply. Greetings, -- *Yaniv Bronhaim.*

1 0

Fwd: Fedora 23 End Of Life
by Sandro Bonazzola 21 Dec '16

21 Dec '16

FYI. ---------- Messaggio inoltrato ---------- Da: "Mohan Boddu" <mboddu(a)redhat.com> Data: 21/Dic/2016 04:05 Oggetto: Fedora 23 End Of Life A: <announce(a)lists.fedoraproject.org>, < test-announce(a)lists.fedoraproject.org>, < devel-announce(a)lists.fedoraproject.org> Cc: As of the 20th of December 2016, Fedora 23 has reached its end of life for updates and support. No further updates, including security updates, will be available for Fedora 23. A previous reminder was sent on 28th of November 2016 [0]. Fedora 24 will continue to receive updates until approximately one month after the release of Fedora 26. The maintenance schedule of Fedora releases is documented on the Fedora Project wiki [1]. The Fedora Project wiki also contains instructions [2] on how to upgrade from a previous release of Fedora to a version receiving updates. Mohan Boddu. [0]https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org/thread/HLHKRTIB33EDZXP624GHF2OZLHWAGKSJ/# Q5O44X4BEBOYEKAEVLSXVI44DSNVHBYG [1]https://fedoraproject.org/wiki/Fedora_Release_Life_ Cycle#Maintenance_Schedule [2]https://fedoraproject.org/wiki/Upgrading?rd=DistributionUpgrades _______________________________________________ devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org

1 0

test mail
by Evgheni Dereveanchin 21 Dec '16

21 Dec '16

Hi, this is a test message to check that archiving is back to normal as part of OVIRT-949 [1] Please ignore. Regards, Evgheni Dereveanchin [1] https://ovirt-jira.atlassian.net/browse/OVIRT-949

1 0

Heads-up: moving Libvirt xml creation to the engine
by Arik Hadas 08 Dec '16

08 Dec '16

Hi All, We are working on something that is expected to have a big impact, hence this heads-up. First, we want you to be aware of this change and provide your feedback to make it as good as possible. Second, until the proposed mechanism is fully merged there will be a chase to cover all features unless new features are also implemented with the new mechanism. So please, if you are working on something that adds/changes something in the Libvirt's domain xml, do it with this new mechanism as well (first version would be merged soon). * Goal Creating Libvirt XML in the engine rather than in VDSM. ** Today's flow Engine: VM business entity -> VM properties map VDSM: VM properties map -> Libvirt XML ** Desired flow Engine: VM business entity -> Libvirt XML * Potential Benefits 1. Reduce the number of conversions from 2 to 1, reducing chances for mistakes in the process. 2. Reduce the amount of code in VDSM. 3. Make VM related changes easier - today many of these changes need to be reviewed in 2 projects, this will eliminate the one that tends to take longer. 4. Prevent shortcuts in the form of VDSM-only changes that should be better reflected in the engine. 5. Not to re-generate the XML on each rerun attempt of VM run/migration. 6. Future - not to re-generate the XML on each attempt to auto-start HA VM when using vm-leases (need to make sure we're using the up-to-date VM configuration though). 7. We already found improvements and cleanups that could be made while touching this area (e.g., remove the boot order from devices in the database). * Challenges 1. Not to move host-specific information to the engine. For example, path to storage domain or sockets of channels. The solution is to use place-holders that will be replaced by VDSM. 2. Backward compatibility. 3. The more challenging part is the other direction - that will be the next phase. * Status As a first step, we began with producing the Libvirt XML in the engine by converting the VM properties map to XML in the engine [1] And using the XML that is received as an input in VDSM [2] [1] https://gerrit.ovirt.org/#/c/64473/ [2] https://gerrit.ovirt.org/#/c/65182/ Regards, Arik

11 24

Red Hat Summit: oVirt Talk(s) Submission
by Brian Proffitt 08 Dec '16

08 Dec '16

One of the big events for the year is the Red Hat Summit, which has a dedicated open source community track for upstream projects. At the very least this is an opportunity to give a "state of the union" talk for the various communities, including oVirt. There are a significant number of downstream-related talks for Summit, but if you would like to submit an oVirt-related talk for the open source community track[1] before the CFP for Summit closes on Dec. 16, please do so! Peace, Brian [1] https://rh2017.smarteventscloud.com/portal/cfp/cfpLogin.ww?sc_cid=701600000… -- Brian Proffitt Principal Community Analyst Open Source and Standards @TheTechScribe 574.383.9BKP

1 0

ovirt-appliance (today) is 1.5G...
by Yaniv Kaul 08 Dec '16

08 Dec '16

I'm sure it was less just days ago, but now[1] it is again quite big. 4.0's is 'only' 1G in size[2]. Which anyway raises the question - should we provide delta RPMs for it? Thanks, Y. [1] http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ovirt-e… [2] http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/ovirt-engi…

3 4

set_modified_hook (update)
by Shlomo Ben David 08 Dec '16

08 Dec '16

Hi All, This email is an update for the set_modified hook hook: set_modified hook goal: when patch merged, changes the bug status from POST to MODIFIED. - The current version of the hook doesn't checks that all previous patches (on the external tracker) were closed before setting the bug status to MODIFIED. - I released this [1] patch that fixes it. - If you have any questions/issues about this/other hooks please fill free to send an email to me / infra.ovirt.org [1] - https://gerrit.ovirt.org/#/c/67512/1 Thanks for your cooperation, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

1 0

New test failure on travis
by Nir Soffer 08 Dec '16

08 Dec '16

Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this: See https://travis-ci.org/oVirt/vdsm/jobs/181933329 ====================================================================== ERROR: test suite for <module 'network.nmdbus_test' from '/vdsm/tests/network/nmdbus_test.py'> ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run self.setUp() File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp self.setupContext(ancestor) File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext try_run(context, names) File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run return func() File "/vdsm/tests/testValidation.py", line 191, in wrapper return f(*args, **kwargs) File "/vdsm/tests/testValidation.py", line 97, in wrapper return f(*args, **kwargs) File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module NMDbus.init() File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init NMDbus.bus = dbus.SystemBus() File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, in __new__ private=private) File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, in __new__ bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop) File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in __new__ bus = cls._new_for_bus(address_or_type, mainloop=mainloop) DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory -------------------- >> begin captured logging << -------------------- 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None) (commands:69) 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED: <err> = 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = 1 (commands:93) 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None) (commands:69) 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED: <err> = 'Failed to get D-Bus connection: Operation not permitted\n'; <rc> = 1 (commands:93) --------------------- >> end captured logging << ---------------------

4 7

[Feture discussion] Full vacuum tool
by Roy Golan 08 Dec '16

08 Dec '16

Hi all, This is a discussion on the RFE[1] to provide a tool to perform full vacuum on our DBs. First if you are not familiar with vacuum please read this [2] # Backgroud ovirt 'engine' DB have several busy table with 2 differnt usage patten. One is audit_log and the others are the 'v*_statistics' tables and the difference between them is mostly inserts vs mostly hot updates. Tables with tons of updates creates garbage or 'dead' records that should be removed, and for this postgres have the aforementioned autovacuum cleaner. It will make the db reuse its already allocated space to perform future updates/inserts and so on. Autovacuum is essential for a db to function optimally and tweaking it is out of the scope of the feature. Full vacuum is designed to reclaim the disk space and reset the table statistics. It is a heavy maintenance task, it takes an exclusive lock on the table and may take seconds to minutes. In some situations it is effectively a downtime due to the long table lock and should not be running when the engine is running. # Critiria Provide a way to reclaim disk space claimed by the garbage created over time by the engine db and dwh. # Usage Either use it as part of the upgrade procedure (after all dbscipts execution) or just provide the tool and admin will run in on demand - engine db credentials read from /etc/ovirt-engine/engine.conf.d/ - invocation: ``` tool: [dbname(default engine)] [table: (default all)] ``` - if we invoke it on upgrade than an installation plugin should be added to invoke with default, no interaction - since VACUUM ANALYZE is consider a recommended maintenance task we can to it by default and ask the user for FULL. - remote db is supported as well, doesn't have to be local # Questions - Will remote dwh have the credentials under /etc/ovirt-engine/engine.conf.d? - Should AAA schema be taken into account as well? Please review, thanks Roy [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430 [2] https://www.postgresql.org/docs/9.2/static/runtime-config-autovacuum.html [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html

6 11

The engine build job you probably didn't know exists
by Eyal Edri 08 Dec '16

08 Dec '16

FYI, After hearing that this might be useful to many developers, I wanted to bring to your attention the ovirt-engine master build job from patch [1] which allows you to build new rpms from an open ovirt-engine patch on Gerrit. Its was created as temp job for a team who needed it a few months ago, but seems it might useful for other teams as well, so we decided to publish it even though its not standardized yet as part of our standard CI system. I hope you can find it useful for your teams. [1] http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64… -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

6 9

Review request for vdsm patch https://gerrit.ovirt.org/#/c/67912/
by Ramesh Nachimuthu 08 Dec '16

08 Dec '16

Hi, I have sent the patch in subject to fix bz#1402248. It blocks gluster brick provisioning in oVirt 4.0.6. It will be helpful if someone can review this patch. Regards, Ramesh

1 0

[VDSM] Correct implementation of virt-sysprep job
by Shmuel Melamud 07 Dec '16

07 Dec '16

Hi! I'm currently working on integration of virt-sysprep into oVirt. Usually, if user creates a template from a regular VM, and then creates new VMs from this template, these new VMs inherit all configuration of the original VM, including SSH keys, UDEV rules, MAC addresses, system ID, hostname etc. It is unfortunate, because you cannot have two network devices with the same MAC address in the same network, for example. To avoid this, user must clean all machine-specific configuration from the original VM before creating a template from it. You can do this manually, but there is virt-sysprep utility that does this automatically. Ideally, virt-sysprep should be seamlessly integrated into template creation process. But the first step is to create a simple button: user selects a VM, clicks the button and oVirt executes virt-sysprep on the VM. virt-sysprep works directly on VM's filesystem. It accepts list of all disks of the VM as parameters: virt-sysprep -a disk1.img -a disk2.img -a disk3.img The architecture is as follows: command on the Engine side runs a job on VDSM side and tracks its success/failure. The job on VDSM side runs virt-sysprep. The question is how to implement the job correctly? I thought about using storage jobs, but they are designed to work only with a single volume, correct? Is is possible to use them with operation that is performed on multiple volumes? Or, alternatively, is it possible to use some kind of 'VM jobs' - that work on VM at whole? How v2v solves this problem? Any ideas? Shmuel

9 18