February 2018 - Devel - oVirt List Archives

[ OST Failure Report ] [ oVirt $VER (ovirt-log-collector) ] [ 21-02-2018 ] [ 098_ovirt_provider_ovn.use_ovn_provider ]
by Dafna Ron 21 Feb '18

21 Feb '18

Hi, We failed test on duplicate network name. I don't think that the change is related. I can see that we are importing network_1 and then trying to create a network by the same name. I don't think there was a change made to ovn before this failure that could have cause this to fail but adding Dan in case he is aware of any changes. *Link and headline of suspected patches: failed change: * inventory: handle to postgresql dumps from sosreport - https://gerrit.ovirt.org/#/c/88010/ *Change reported as root cause of failure: *collector: fix list function - https://gerrit.ovirt.org/#/c/87798/ *Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805>Link to all logs:* http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805/artifacts *(Relevant) error snippet from the log: <error>* 2018-02-20 23:51:03,179-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Compiled stored procedure. Call string is [{call getdcidbyexternalnetworkid(?)}] 2018-02-20 23:51:03,179-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] SqlCall for procedure [GetDcIdByExternalNetworkId] compiled 2018-02-20 23:51:03,180-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] method: runQuery, params: [GetAllExternalNetworksOnProvider, IdQueryParameters:{refresh='false', filtered='false'}], timeElapsed: 266ms 2018-02-20 23:51:03,187-05 INFO [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[network_1=NETWORK, 94fc81b9-fbf2-4262-a877-32a066a573c2=PROVIDER]', sharedLocks=''}' 2018-02-20 23:51:03,190-05 WARN [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Validation of action 'AddNetwork' failed for user admin@internal-authz. Reasons: VAR__TYPE__NETWORK,VAR__ACTION__ADD,ACTION_TYPE_FAILED_NETWORK_NAME_IN_USE,$NetworkName network_1 2018-02-20 23:51:03,190-05 INFO [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Lock freed to object 'EngineLock:{exclusiveLocks='[network_1=NETWORK, 94fc81b9-fbf2-4262-a877-32a066a573c2=PROVIDER]', sharedLocks=''}' 2018-02-20 23:51:03,191-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] method: runAction, params: [AddNetwork, AddNetworkStoragePoolParameters:{commandId='41c44658-04e5-4aeb-9f65-cc2ad76e0c42', user='null', commandType='Unknown'}], timeElapsed: 10ms 2018-02-20 23:51:03,196-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-19) [] Operation Failed: [Cannot add Network. The name of the logical network 'network_1' is already used by an existing logical network in the same data-center. -Please choose a different name.] 2018-02-20 23:51:03,346-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] method: get, params: [da0bebdb-cb83-415f-bcba-8920126e31a4], timeElapsed: 6ms 2018-02-20 23:51:03,346-05 DEBUG [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] vdsManager::refreshVdsStats entered, host='lago-basic-suite-master-host-1'(da0bebdb-cb83-415f-bcba-8920126e31a4) 2018-02-20 23:51:03,365-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] START, GetStatsAsyncVDSCommand(HostName = lago-basic-suite-master-host-1, VdsIdAndVdsVDSCommandParametersBase:{hostId='da0bebdb-cb83-415f-bcba-8920126e31a4', vds='Host[lago-basic-suite-master-host-1,da0bebdb-cb83-415f-bcba-8920126e31a4]'}), log id: 279de334 2018-02-20 23:51:03,365-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] SEND destination:jms.topic.vdsm_requests reply-to:jms.topic.vdsm_responses content-length:98 *</error>*

1 0

Documenting HA VMs on Wiki
by Milan Zamazal 20 Feb '18

20 Feb '18

Hi, to document and clarify some facts about HA VMs to users, especially regarding I/O errors, I work on a HOWTO document to be added to oVirt Wiki. If you know how HA VMs or related mechanisms (such as VM handling on I/O errors) work and would like to help review the document, please look at the pull request: https://github.com/oVirt/ovirt-site/pull/1530 Thanks, Milan

1 0

[vdsm][network] false test failiures in CI
by Francesco Romani 20 Feb '18

20 Feb '18

Hi there, recently some patches of mine failed on CI with this error which is totally unrelated and which seems bogus. Could some network developer please have a look and improve the current state? :) https://gerrit.ovirt.org/#/c/87881/ 19:43:33 ====================================================================== 19:43:33 ERROR: test_list_ipv4_ipv6 (network.ip_address_test.IPAddressTest) 19:43:33 ---------------------------------------------------------------------- 19:43:33 Traceback (most recent call last): 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/testValidation.py", line 330, in wrapper 19:43:33 return f(*args, **kwargs) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/network/ip_address_test.py", line 289, in test_list_ipv4_ipv6 19:43:33 ipv6_addresses=[IPV6_B_WITH_PREFIXLEN] 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/network/ip_address_test.py", line 297, in _test_list 19:43:33 address.IPAddressData(addr, device=nic)) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 40, in add 19:43:33 addr_data.address, addr_data.prefixlen, addr_data.family 19:43:33 File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ 19:43:33 self.gen.throw(type, value, traceback) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 91, in _translate_iproute2_exception 19:43:33 new_exception, new_exception(str(address_data), error_message), tb) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 86, in _translate_iproute2_exception 19:43:33 yield 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 40, in add 19:43:33 addr_data.address, addr_data.prefixlen, addr_data.family 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 561, in addrAdd 19:43:33 _exec_cmd(command) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 482, in _exec_cmd 19:43:33 raise exc(returnCode, error.splitlines()) 19:43:33 IPAddressAddError: ("IPAddressData(device='dummy_ZVN3L' address=IPv6Interface(u'2002:99::1/64') scope=None flags=None)", 'RTNETLINK answers: Permission denied') I had other failures, but they look like known issues (mkimage, no port on protocol detector) -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

2 1

[ OST Failure Report ] [ oVirt 4.2 (ovirt-engine-metrics) ] [ 20-02-2018 ] [ 003_00_metrics_bootstrap.metrics_and_log_collector ]
by Dafna Ron 20 Feb '18

20 Feb '18

Hi, We have a failed test for project ovirt-engine-metrics. It seems that CQ is reporting that the root cause of the failure was a patch from 6 days ago. Shirly, can you please check the changes? *Link and headline of suspected patches: change reported as failed: * replace vdsm stats with collectd virt plugin - https://gerrit.ovirt.org/#/c/87310/ *change reported as root cause: * ansible: add role to copy required files - https://gerrit.ovirt.org/#/c/87430/ TASK [oVirt.ovirt-initial-validations/validate-config-yml : Validate viaq_metrics_store parameter] *** fatal: [localhost]: FAILED! => {"msg": "The conditional check 'viaq_metrics_store != true' failed. The error was: error while evaluating conditional (viaq_metrics_store != true): 'viaq_metrics_store' is undefined\n\nThe error appears to have been in '/usr/share/ansible/roles/oVirt.metrics/roles/oVirt.ovirt-initial-validations/validate-config-yml/tasks/main.yml': line 49, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Validate viaq_metrics_store parameter\"\n ^ here\n"} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* localhost : ok=4 changed=0 unreachable=0 failed=1 lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/003_00_metrics_bootstrap.py", line 54, in configure_metrics ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) AssertionError: Configuring ovirt machines for metrics failed. Exit code is 2 lago.ssh: DEBUG: Command 59b597f6 on lago-basic-suite-4-2-engine returned with 0 lago.ssh: DEBUG: Command 59b597f6 on lago-basic-suite-4-2-engine output: This command will collect system configuration and diagnostic information from this system. The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. INFO: /dev/shm/log does not exist. It will be created. DEBUG: API Vendor(None) API Version(4.2.0) WARNING: This ovirt-log-collector call will collect logs from all available hosts. This may take long time, depending on the size of your deployment INFO: Gathering oVirt Engine information... DEBUG: loading config '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf' DEBUG: calling(['sosreport', '--list-plugins']) DEBUG: returncode(0) DEBUG: STDOUT( sosreport (version 3.4) *Link to Job:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/ <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/artifac… <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/artifacts>(Relevant) error snippet from the log: <error></error>*

3 4

CI outage notice
by Evgheni Dereveanchin 20 Feb '18

20 Feb '18

Hi everyone, After the Jenkins update tonight it started behaving unstably and there were several restarts recorded throughout the night. The root cause is now found and mitigated [1], Jenkins is up and stable again. There may still be slaves around that were running jobs when the master restarted which caused leftover files in the workspace. If you had any CI jobs fail with errors unrelated to code between midnight and 8:15 GMT today - please re-trigger those. If you still get failures - please inform me or other members of the infra team to offline such slaves and perform a cleanup. -- Regards, Evgheni Dereveanchin [1] https://ovirt-jira.atlassian.net/browse/OVIRT-1904

1 0

planned Jenkins restart
by Evgheni Dereveanchin 20 Feb '18

20 Feb '18

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. Regards, Evgheni Dereveanchin

1 1

OST Failure - Weekly update [10/02/2018-16/02/2018]
by Dafna Ron 18 Feb '18

18 Feb '18

Hello, I would like to update on this week's failure and OST current status. At the end of the week I am happy to say that we are no longer failing on any issues and the testers are completely green. We had a great week of many people rallying together to fix OST and resolve all outstanding issues. During the week we saw and resolved the following failures: *Master: * 002_bootstrap.get_host_devices - this was related to timing issues with the test and was fixed by: https://gerrit.ovirt.org/87526 001_initialize_engine.initialize_engine - dwh service failed to start - this seemed liked a packaging issue and was resolved. *4.2:* 002_bootstrap.add_cluster - cluster.cpu.type error - this was resolved a week ago and was resolved by patch: https://gerrit.ovirt.org/#/c/87126/ 003_00_metrics_bootstrap.metrics_and_log_collector - was resolved by Shirly. 002_bootstrap.get_host_devices - this was related to timing issues with the test and was fixed by: https://gerrit.ovirt.org/87526 *4.1* 001_initialize_engine.initialize_engine - dwh service failed to start - this seemed liked a packaging issue and was resolved. Build-artifacts jobs failing has a high percentage but* many of the build-artifacts failures were caused because of fcraw issues *and I am hoping this would be resolved over time. *below you can see the chart of OST resolved issues based on cause and failures: * * Based on feedback, I made some changes to the definitions Code= regression of working components/functionalities Configurations - package related issues Other = failed build artifacts Infra = infrastructure/OST/Lago related issues [image: Inline image 2] [image: Inline image 1] *Below is a chart of resolved failures based on Ovirt Version: * [image: Inline image 4] [image: Inline image 3] *Below is a chart showing failures by suite type: * * Suite type None means that it was a failure that did not result in a failed test such artifacts related or packaging related. [image: Inline image 8] [image: Inline image 7] Thanks, Dafna

3 2

[ OST Failure Report ] [ oVirt 4.2 +Master (ovirt-engine-metrics) ] [ 14-02-2018 ] [ 003_00_metrics_bootstrap.metrics_and_log_collector ]
by Dafna Ron 15 Feb '18

15 Feb '18

Hi, Test 003_00_metrics_bootstrap.metrics_and_log_collector failed on basic suite. The only error I could find was in lago log about failing to configure metrics. *Link and headline of suspected patches: https://gerrit.ovirt.org/#/c/87489/7 <https://gerrit.ovirt.org/#/c/87489/7> - ansible: update ovirt-metrics-store-setup roleLink to Job:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/775/ <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/775/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/775/artifact <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/775/artifact>(Relevant) error snippet from the log: <error>* 018-02-13 17:06:38,250::utils.py::_ret_via_queue::60::lago.utils::ERROR::Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/003_00_metrics_bootstrap.py", line 54, in configure_metrics ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) AssertionError: Configuring ovirt machines for metrics failed. Exit code is 2

2 1

OST 004_basic_sanity.runVm()
by Eitan Raviv 14 Feb '18

14 Feb '18

In a manual jenkins run [1] run vm once fails because of 'CPU-Level' with traceback [2]. engine.log [3]: 2018-02-14 07:11:02,272-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] Running command: RunVmOnceCommand internal: false. Entities affected : ID: cedb8d45-a3f3-4475-b494-fb9e9aaaeb93 Type: VMAction group RUN_VM with role type USER, ID: cedb8d45-a3f3-4475-b494-fb9e9aaaeb93 Type: VMAction group EDIT_ADMIN_VM_PROPERTIES with role type ADMIN 2018-02-14 07:11:02,276-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] method: getVmManager, params: [cedb8d45-a3f3-4475-b494-fb9e9aaaeb93], timeElapsed: 4ms 2018-02-14 07:11:02,288-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] method: getAllForClusterWithStatus, params: [91d42c1f-07f5-4ef8-ae09-8da2481f6e65, Up], timeElapsed: 5ms 2018-02-14 07:11:02,289-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] Candidate host 'lago-basic-suite-master-host-0' ('ad4e3763-233c-4ae1-95c8-b23ba4ffd60a') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 7d3b5606-16d7-4749-858c-8e6859135999) 2018-02-14 07:11:02,289-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] Can't find VDS to run the VM 'cedb8d45-a3f3-4475-b494-fb9e9aaaeb93' on, so this VM will not be run. 2018-02-14 07:11:02,295-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-20) [7d3b5606-16d7-4749-858c-8e6859135999] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm0 (User: admin@internal-authz). anybody? thanks [1] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2176/ [2] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2176/artifact/export… [3] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2176/artifact/export… -- Eitan Raviv IRC: erav (#ovirt #vdsm #devel #rhev-dev)

2 1

Packages for Python SDK
by Tomáš Golembiovský 13 Feb '18

13 Feb '18

Hi, are there any plans to have the packages for Python SDK in base repositories of CentOS and Fedora? They seem to be available only from oVirt and Virt SIG repos. I saw that Juan used to build the v3 packages for Fedora. The rationale behind is that virt-v2v is considering to add support for the disk upload API and would like to use Python snippets so as not to re-implement the calls in curl. Tomas -- Tomáš Golembiovský <tgolembi(a)redhat.com>

4 5

[ OST Failure Report ] [ oVirt Master (vdsmt) ] [ 12-0202018 ] [ 002_bootstrap.get_host_devices ] - Reminder
by Dafna Ron 12 Feb '18

12 Feb '18

hi, I saw this failure on Friday and I also see it in 4.2 branch as well as Master. host-0 did not finish installation on misconfiguration (which I can only asume has to do with the block device list) but I cannot see anything in host-0 or engine to suggest what was the actual problem. The patches its reporting are not related. *Link and headline of suspected patches: Not relatedLink to Job:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5613/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5613/>Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5613/artifacts <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5613/>(Relevant) error snippet from the log: <error>Api error: *Error Message Could not find block_vda1 device in host devices: Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1208, in get_host_devices raise RuntimeError('Could not find block_vda1 device in host devices: {}'.format(device_list)) RuntimeError: Could not find block_vda1 device in host devices: *eng*ine log: [dron@dron post-002_bootstrap.py]$ less lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log |grep host-0 2018-02-12 07:20:17,184-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] method: getByName, params: [lago-basic-suite-master-host-0], timeElapsed: 18ms 2018-02-12 07:20:17,201-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] method: getAllForHostname, params: [lago-basic-suite-master-host-0], timeElapsed: 17ms 2018-02-12 07:20:17,220-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] Connecting 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:17,414-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] Connected: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:17,427-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] Authenticating: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:17,490-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-9) [ebedf4a6-07be-45b3-9062-7996c362f747] Authenticated: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:18,247-05 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (default task-9) [21f04b96] START, RemoveVdsVDSCommand(HostName = lago-basic-suite-master-host-0, RemoveVdsVDSCommandParameters:{hostId='468d8314-e2e0-416b-af9d-295f8ab5a522'}), log id: 9f1017d 2018-02-12 07:20:18,262-05 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (default task-9) [21f04b96] START, AddVdsVDSCommand(HostName = lago-basic-suite-master-host-0, AddVdsVDSCommandParameters:{hostId='468d8314-e2e0-416b-af9d-295f8ab5a522'}), log id: a336fc4 2018-02-12 07:20:18,288-05 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (default task-9) [21f04b96] Initialize vdsBroker 'lago-basic-suite-master-host-0:54321' 2018-02-12 07:20:18,345-05 DEBUG [org.ovirt.engine.core.vdsbroker.ResourceManager] (default task-9) [21f04b96] subscribe called with subscription id: lago-basic-suite-master-host-0|*|VM_status|* 2018-02-12 07:20:18,346-05 DEBUG [org.ovirt.engine.core.vdsbroker.ResourceManager] (default task-9) [21f04b96] subscribe called with subscription id: lago-basic-suite-master-host-0|net|host_conn|no_id 2018-02-12 07:20:18,369-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-9) [21f04b96] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Failed to verify Power Management configuration for Host lago-basic-suite-master-host-0. 2018-02-12 07:20:18,398-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-9) [21f04b96] EVENT_ID: USER_ADD_VDS(42), Host lago-basic-suite-master-host-0 was added by admin@internal-authz. 2018-02-12 07:20:18,486-05 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Before Installation host 468d8314-e2e0-416b-af9d-295f8ab5a522, lago-basic-suite-masterhost-0 2018-02-12 07:20:18,549-05 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='468d8314-e2e0-416b-af9d-295f8ab5a522', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a95e559 2018-02-12 07:20:18,567-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] connect enter (lago-basic-suite-master-host-0:22, 1800000, 600000) 2018-02-12 07:20:18,567-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Connecting 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:18,751-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Connected: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:18,767-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Connected to host lago-basic-suite-master-host-0 with SSH key fingerprint: SHA256:efkJTmSmltPF5S4VbnQ1avsb7whAX0dgLgxZJTxFgqo. 2018-02-12 07:20:18,767-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Authenticating: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:18,835-05 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Authenticated: 'root@lago-basic-suite-master-host-0' 2018-02-12 07:20:18,835-05 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] Installation of lago-basic-suite-master-host-0. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2018-02-12 07:20:18,888-05 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [7389a3fe] SSH execute 'root@lago-basic-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-02-12 07:20:19,933-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Initializing. 2018-02-12 07:20:20,094-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Environment setup. 2018-02-12 07:20:20,112-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Environment packages setup. 2018-02-12 07:20:24,084-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Programs detection. 2018-02-12 07:20:24,199-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Environment setup. 2018-02-12 07:20:24,205-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Environment customization. 2018-02-12 07:20:24,617-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Kdump supported. 2018-02-12 07:20:24,775-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Logs at host located at: '/tmp/ovirt-host-deploy-20180212072019-56s96x.log'. 2018-02-12 07:20:24,890-05 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [7389a3fe] Host lago-basic-suite-master-host-0 reports unique id A29E23CD-AF59-423B-9FCB-D3E4A0CD6AA1 2018-02-12 07:20:24,908-05 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployVdsmUnit] (VdsDeploy) [7389a3fe] Assigning unique id A29E23CD-AF59-423B-9FCB-D3E4A0CD6AA1 to Host lago-basic-suite-master-host-0 2018-02-12 07:20:24,987-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Disabling Kdump integration. 2018-02-12 07:20:25,070-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Setup validation. 2018-02-12 07:20:25,280-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Hardware supports virtualization. 2018-02-12 07:20:25,383-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Transaction setup. 2018-02-12 07:20:25,429-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Misc configuration. 2018-02-12 07:20:25,433-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Package installation. 2018-02-12 07:20:27,065-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Status: Downloading Packages. 2018-02-12 07:20:27,069-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Download/Verify: tuned-2.8.0-5.el7_4.2.noarch. 2018-02-12 07:20:27,074-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Status: Check Package Signatures. 2018-02-12 07:20:27,078-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Status: Running Test Transaction. 2018-02-12 07:20:27,192-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Status: Running Transaction. 2018-02-12 07:20:27,309-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum update: 1/2: tuned-2.8.0-5.el7_4.2.noarch. 2018-02-12 07:20:27,524-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum updated: 2/2: tuned. 2018-02-12 07:20:29,174-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Verify: 1/2: tuned.noarch 0:2.8.0-5.el7_4.2 - u. 2018-02-12 07:20:29,182-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Yum Verify: 2/2: tuned.noarch 0:2.8.0-5.el7 - ud. 2018-02-12 07:20:29,266-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7389a3fe] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-0. Stage: Misc configuration. *</error>*

2 1

[OST] Some tests failed?
by Nir Soffer 12 Feb '18

12 Feb '18

I got this error when running OST with engine 4.2 - is this a test error? environment error? Looks like some error message needs work :-) http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2157/console *00:10:32.263* @ Run test: 002_bootstrap.py: *00:10:32.281* nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']*00:10:32.301* # add_dc: *00:10:53.839* # add_dc: Success (in 0:00:21)*00:10:53.843* # add_cluster: *00:10:54.123* * Collect artifacts: *00:10:55.995* * Collect artifacts: Success (in 0:00:01)*00:10:56.000* # add_cluster: Success (in 0:00:02)*00:10:56.003* # Results located at /dev/shm/ost/deployment-basic-suite-4.2/default/002_bootstrap.py.junit.xml*00:10:56.005* @ Run test: 002_bootstrap.py: Success (in 0:00:23)*00:10:56.296* Error occured, aborting*00:10:56.296* Traceback (most recent call last):*00:10:56.296* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in do_run*00:10:56.297* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:56.298* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*00:10:56.299* self._do_run(**vars(args))*00:10:56.299* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 505, in wrapper*00:10:56.300* return func(*args, **kwargs)*00:10:56.300* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 516, in wrapper*00:10:56.301* return func(*args, prefix=prefix, **kwargs)*00:10:56.301* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest*00:10:56.302* raise RuntimeError('Some tests failed')*00:10:56.303* RuntimeError: Some tests failed

2 1

[VDSM] network tests failing the build again
by Nir Soffer 11 Feb '18

11 Feb '18

Did we changes something in the CI or the network code? We see the following failures since yesterday. I'm ignoring the failures since they are not related to the patch, but checking these failures waste lot time. Can someone from network look at this? http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc27-x86_64/1283/conso… *00:04:37.280* ======================================================================*00:04:37.280* ERROR: test_identify_non_vlan_base_device (network.link_vlan_test.LinkIfaceTests)*00:04:37.281* ----------------------------------------------------------------------*00:04:37.281* Traceback (most recent call last):*00:04:37.281* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/link_vlan_test.py", line 46, in test_identify_non_vlan_base_device*00:04:37.282* self.assertFalse(vlan.is_base_device(nic))*00:04:37.283* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/vlan.py", line 47, in is_base_device*00:04:37.283* return any(get_vlans_on_base_device(dev_name))*00:04:37.284* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/vlan.py", line 41, in <genexpr>*00:04:37.284* return (iface_properties['name'] for iface_properties in iface.list()*00:04:37.285* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 227, in list*00:04:37.285* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.286* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 47, in get_dpdk_devices*00:04:37.287* dpdk_devices = _get_dpdk_devices()*00:04:37.287* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.287* value = self.func(*args)*00:04:37.288* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.288* devices = _lshw_command()*00:04:37.289* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.289* return _normalize_lshw_result(out)*00:04:37.290* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.290* return json.loads('[' + result + ']')*00:04:37.291* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.291* return _default_decoder.decode(s)*00:04:37.291* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.292* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.292* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.292* obj, end = self.scan_once(s, idx)*00:04:37.293* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.293* -------------------- >> begin captured logging << --------------------*00:04:37.294* 2018-02-08 11:43:47,659 DEBUG (MainThread) [root] /sbin/ip link add name dummy_Afi2m type dummy (cwd None) (cmdutils:150)*00:04:37.294* 2018-02-08 11:43:47,675 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.295* 2018-02-08 11:43:47,678 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb6687290>>, args=(), kwargs={}) (concurrent:192)*00:04:37.296* 2018-02-08 11:43:47,680 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_Afi2m up (cwd None) (cmdutils:150)*00:04:37.297* 2018-02-08 11:43:47,700 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.298* 2018-02-08 11:43:47,704 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.298* 2018-02-08 11:43:47,706 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.300* 2018-02-08 11:43:47,866 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.300* 2018-02-08 11:43:47,868 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_Afi2m (cwd None) (cmdutils:150)*00:04:37.301* 2018-02-08 11:43:47,896 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.301* --------------------- >> end captured logging << ---------------------*00:04:37.302* *00:04:37.302* ======================================================================*00:04:37.302* ERROR: test_identify_vlan_base_device (network.link_vlan_test.LinkIfaceTests)*00:04:37.303* ----------------------------------------------------------------------*00:04:37.303* Traceback (most recent call last):*00:04:37.303* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/link_vlan_test.py", line 42, in test_identify_vlan_base_device*00:04:37.304* self.assertTrue(vlan.is_base_device(nic))*00:04:37.304* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/vlan.py", line 47, in is_base_device*00:04:37.305* return any(get_vlans_on_base_device(dev_name))*00:04:37.306* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/vlan.py", line 41, in <genexpr>*00:04:37.306* return (iface_properties['name'] for iface_properties in iface.list()*00:04:37.307* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 227, in list*00:04:37.307* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.308* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.309* dpdk_devices = _get_dpdk_devices()*00:04:37.309* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.309* value = self.func(*args)*00:04:37.310* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.310* devices = _lshw_command()*00:04:37.311* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.311* return _normalize_lshw_result(out)*00:04:37.312* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.313* return json.loads('[' + result + ']')*00:04:37.313* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.313* return _default_decoder.decode(s)*00:04:37.313* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.314* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.314* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.315* obj, end = self.scan_once(s, idx)*00:04:37.315* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.315* -------------------- >> begin captured logging << --------------------*00:04:37.316* 2018-02-08 11:43:47,902 DEBUG (MainThread) [root] /sbin/ip link add name dummy_tNDrx type dummy (cwd None) (cmdutils:150)*00:04:37.316* 2018-02-08 11:43:47,917 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.317* 2018-02-08 11:43:47,926 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb61e8a90>>, args=(), kwargs={}) (concurrent:192)*00:04:37.318* 2018-02-08 11:43:47,928 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_tNDrx up (cwd None) (cmdutils:150)*00:04:37.319* 2018-02-08 11:43:47,943 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.320* 2018-02-08 11:43:47,946 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.321* 2018-02-08 11:43:47,950 DEBUG (MainThread) [root] /sbin/ip link add link dummy_tNDrx name dummy_tNDrx.999 type vlan id 999 (cwd None) (cmdutils:150)*00:04:37.322* 2018-02-08 11:43:47,971 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.322* 2018-02-08 11:43:47,972 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_tNDrx.999 up (cwd None) (cmdutils:150)*00:04:37.323* 2018-02-08 11:43:47,993 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.325* 2018-02-08 11:43:47,995 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.328* 2018-02-08 11:43:48,143 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.329* 2018-02-08 11:43:48,146 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb65cef90>>, args=(), kwargs={}) (concurrent:192)*00:04:37.338* 2018-02-08 11:43:48,148 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_tNDrx.999 down (cwd None) (cmdutils:150)*00:04:37.339* 2018-02-08 11:43:48,169 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.340* 2018-02-08 11:43:48,170 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.342* 2018-02-08 11:43:48,171 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_tNDrx.999 (cwd None) (cmdutils:150)*00:04:37.344* 2018-02-08 11:43:48,192 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.345* 2018-02-08 11:43:48,193 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_tNDrx (cwd None) (cmdutils:150)*00:04:37.346* 2018-02-08 11:43:48,214 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.347* --------------------- >> end captured logging << ---------------------*00:04:37.348* *00:04:37.348* ======================================================================*00:04:37.349* ERROR: test_list_vlans_on_base_device (network.link_vlan_test.LinkIfaceTests)*00:04:37.351* ----------------------------------------------------------------------*00:04:37.352* Traceback (most recent call last):*00:04:37.353* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/link_vlan_test.py", line 37, in test_list_vlans_on_base_device*00:04:37.355* list(vlan.get_vlans_on_base_device(nic)))*00:04:37.356* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/vlan.py", line 41, in <genexpr>*00:04:37.358* return (iface_properties['name'] for iface_properties in iface.list()*00:04:37.360* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/iface.py", line 227, in list*00:04:37.362* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.363* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.365* dpdk_devices = _get_dpdk_devices()*00:04:37.365* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.367* value = self.func(*args)*00:04:37.367* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.369* devices = _lshw_command()*00:04:37.369* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.371* return _normalize_lshw_result(out)*00:04:37.371* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.373* return json.loads('[' + result + ']')*00:04:37.373* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.374* return _default_decoder.decode(s)*00:04:37.375* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.375* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.376* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.377* obj, end = self.scan_once(s, idx)*00:04:37.377* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.378* -------------------- >> begin captured logging << --------------------*00:04:37.379* 2018-02-08 11:43:48,218 DEBUG (MainThread) [root] /sbin/ip link add name dummy_Ecsrx type dummy (cwd None) (cmdutils:150)*00:04:37.381* 2018-02-08 11:43:48,230 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.382* 2018-02-08 11:43:48,237 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb65cec50>>, args=(), kwargs={}) (concurrent:192)*00:04:37.385* 2018-02-08 11:43:48,238 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_Ecsrx up (cwd None) (cmdutils:150)*00:04:37.386* 2018-02-08 11:43:48,255 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.387* 2018-02-08 11:43:48,260 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.389* 2018-02-08 11:43:48,261 DEBUG (MainThread) [root] /sbin/ip link add link dummy_Ecsrx name dummy_Ecsrx.999 type vlan id 999 (cwd None) (cmdutils:150)*00:04:37.391* 2018-02-08 11:43:48,273 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.392* 2018-02-08 11:43:48,275 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_Ecsrx.999 up (cwd None) (cmdutils:150)*00:04:37.393* 2018-02-08 11:43:48,292 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.394* 2018-02-08 11:43:48,294 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.397* 2018-02-08 11:43:48,456 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.398* 2018-02-08 11:43:48,459 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb7faab90>>, args=(), kwargs={}) (concurrent:192)*00:04:37.401* 2018-02-08 11:43:48,461 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_Ecsrx.999 down (cwd None) (cmdutils:150)*00:04:37.402* 2018-02-08 11:43:48,488 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.403* 2018-02-08 11:43:48,490 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.405* 2018-02-08 11:43:48,491 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_Ecsrx.999 (cwd None) (cmdutils:150)*00:04:37.406* 2018-02-08 11:43:48,515 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.408* 2018-02-08 11:43:48,517 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_Ecsrx (cwd None) (cmdutils:150)*00:04:37.409* 2018-02-08 11:43:48,541 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.410* --------------------- >> end captured logging << ---------------------*00:04:37.411* *00:04:37.411* ======================================================================*00:04:37.412* ERROR: test_fake_nics (network.netinfo_test.TestNetinfo)*00:04:37.413* ----------------------------------------------------------------------*00:04:37.414* Traceback (most recent call last):*00:04:37.414* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:04:37.416* return f(*args, **kwargs)*00:04:37.417* File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched*00:04:37.418* return func(*args, **keywargs)*00:04:37.418* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/netinfo_test.py", line 201, in test_fake_nics*00:04:37.420* _nics = nics.nics()*00:04:37.420* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/misc.py", line 57, in visible_devs*00:04:37.422* return [dev.name for dev in getLinks() if predicate(dev) and*00:04:37.422* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.424* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.424* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.426* dpdk_devices = _get_dpdk_devices()*00:04:37.427* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.428* value = self.func(*args)*00:04:37.428* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.430* devices = _lshw_command()*00:04:37.430* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.432* return _normalize_lshw_result(out)*00:04:37.433* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.434* return json.loads('[' + result + ']')*00:04:37.435* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.436* return _default_decoder.decode(s)*00:04:37.437* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.437* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.438* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.439* obj, end = self.scan_once(s, idx)*00:04:37.439* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.440* -------------------- >> begin captured logging << --------------------*00:04:37.441* 2018-02-08 11:43:52,941 DEBUG (MainThread) [root] /sbin/ip link add name veth_Kb2e2cTujn type veth peer name veth_mevoaYSIn2 (cwd None) (cmdutils:150)*00:04:37.443* 2018-02-08 11:43:52,968 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.445* 2018-02-08 11:43:52,969 DEBUG (MainThread) [root] /sbin/ip link add name dummy_5VJln type dummy (cwd None) (cmdutils:150)*00:04:37.446* 2018-02-08 11:43:53,005 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.447* 2018-02-08 11:43:53,029 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb661f790>>, args=(), kwargs={}) (concurrent:192)*00:04:37.450* 2018-02-08 11:43:53,031 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_5VJln up (cwd None) (cmdutils:150)*00:04:37.452* 2018-02-08 11:43:53,060 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.453* 2018-02-08 11:43:53,064 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.455* 2018-02-08 11:43:53,065 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.457* 2018-02-08 11:43:53,222 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.458* 2018-02-08 11:43:53,224 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_5VJln (cwd None) (cmdutils:150)*00:04:37.459* 2018-02-08 11:43:53,248 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.461* 2018-02-08 11:43:53,249 DEBUG (MainThread) [root] /sbin/ip link del dev veth_Kb2e2cTujn (cwd None) (cmdutils:150)*00:04:37.462* 2018-02-08 11:43:53,271 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.463* --------------------- >> end captured logging << ---------------------*00:04:37.464* *00:04:37.464* ======================================================================*00:04:37.465* ERROR: test_get_non_existing_bridge_info (network.netinfo_test.TestNetinfo)*00:04:37.466* ----------------------------------------------------------------------*00:04:37.466* Traceback (most recent call last):*00:04:37.467* File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched*00:04:37.468* return func(*args, **keywargs)*00:04:37.468* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/netinfo_test.py", line 108, in test_get_non_existing_bridge_info*00:04:37.470* get()*00:04:37.470* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 137, in get*00:04:37.472* return _get(vdsmnets)*00:04:37.472* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 62, in _get*00:04:37.473* devices_info = _devices_report(ipaddrs, routes)*00:04:37.474* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in _devices_report*00:04:37.476* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.477* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in <genexpr>*00:04:37.478* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.479* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.481* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.481* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.483* dpdk_devices = _get_dpdk_devices()*00:04:37.483* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.485* value = self.func(*args)*00:04:37.485* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.487* devices = _lshw_command()*00:04:37.487* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.489* return _normalize_lshw_result(out)*00:04:37.489* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.491* return json.loads('[' + result + ']')*00:04:37.491* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.492* return _default_decoder.decode(s)*00:04:37.493* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.493* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.494* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.496* obj, end = self.scan_once(s, idx)*00:04:37.497* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.497* -------------------- >> begin captured logging << --------------------*00:04:37.498* 2018-02-08 11:43:53,427 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.501* 2018-02-08 11:43:53,573 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.502* --------------------- >> end captured logging << ---------------------*00:04:37.503* *00:04:37.503* ======================================================================*00:04:37.503* ERROR: test_speed_in_range (network.netinfo_test.TestNetinfo)*00:04:37.504* ----------------------------------------------------------------------*00:04:37.505* Traceback (most recent call last):*00:04:37.506* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/netinfo_test.py", line 71, in test_speed_in_range*00:04:37.507* for d in nics.nics():*00:04:37.507* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/misc.py", line 57, in visible_devs*00:04:37.509* return [dev.name for dev in getLinks() if predicate(dev) and*00:04:37.510* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.511* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.512* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.513* dpdk_devices = _get_dpdk_devices()*00:04:37.514* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.515* value = self.func(*args)*00:04:37.516* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.517* devices = _lshw_command()*00:04:37.518* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.519* return _normalize_lshw_result(out)*00:04:37.520* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.521* return json.loads('[' + result + ']')*00:04:37.522* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.523* return _default_decoder.decode(s)*00:04:37.523* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.524* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.524* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.525* obj, end = self.scan_once(s, idx)*00:04:37.526* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.527* -------------------- >> begin captured logging << --------------------*00:04:37.527* 2018-02-08 11:43:53,906 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.530* 2018-02-08 11:43:54,060 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.531* --------------------- >> end captured logging << ---------------------*00:04:37.532* *00:04:37.532* ======================================================================*00:04:37.533* ERROR: test_multiple_vlans (network.tc_test.TestConfigureOutbound)*00:04:37.534* ----------------------------------------------------------------------*00:04:37.535* Traceback (most recent call last):*00:04:37.535* File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched*00:04:37.536* return func(*args, **keywargs)*00:04:37.537* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/tc_test.py", line 454, in test_multiple_vlans*00:04:37.539* self.device_name, v.tag)*00:04:37.539* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 48, in configure_outbound*00:04:37.541* qosOutbound)*00:04:37.541* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 163, in _qdisc_conf_out*00:04:37.542* if not _is_explicit_defined_default_class(dev):*00:04:37.543* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 208, in _is_explicit_defined_default_class*00:04:37.545* netinfo = NetInfo(cache_get())*00:04:37.545* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 137, in get*00:04:37.547* return _get(vdsmnets)*00:04:37.547* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 62, in _get*00:04:37.549* devices_info = _devices_report(ipaddrs, routes)*00:04:37.549* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in _devices_report*00:04:37.551* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.552* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in <genexpr>*00:04:37.553* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.554* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.556* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.556* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.558* dpdk_devices = _get_dpdk_devices()*00:04:37.558* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.560* value = self.func(*args)*00:04:37.560* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.562* devices = _lshw_command()*00:04:37.562* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.563* return _normalize_lshw_result(out)*00:04:37.564* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.566* return json.loads('[' + result + ']')*00:04:37.566* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.567* return _default_decoder.decode(s)*00:04:37.568* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.568* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.569* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.570* obj, end = self.scan_once(s, idx)*00:04:37.570* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.571* -------------------- >> begin captured logging << --------------------*00:04:37.572* 2018-02-08 11:44:00,920 DEBUG (MainThread) [root] /sbin/ip link add name dummy_wVPHq type dummy (cwd None) (cmdutils:150)*00:04:37.573* 2018-02-08 11:44:00,932 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.575* 2018-02-08 11:44:00,933 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_wVPHq up (cwd None) (cmdutils:150)*00:04:37.576* 2018-02-08 11:44:00,962 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.577* 2018-02-08 11:44:00,964 DEBUG (MainThread) [root] /sbin/ip link add link dummy_wVPHq name dummy_wVPHq.16 type vlan id 16 (cwd None) (cmdutils:150)*00:04:37.579* 2018-02-08 11:44:00,977 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.580* 2018-02-08 11:44:00,978 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_wVPHq.16 up (cwd None) (cmdutils:150)*00:04:37.581* 2018-02-08 11:44:01,002 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.582* 2018-02-08 11:44:01,003 DEBUG (MainThread) [root] /sbin/ip link add link dummy_wVPHq name dummy_wVPHq.17 type vlan id 17 (cwd None) (cmdutils:150)*00:04:37.584* 2018-02-08 11:44:01,020 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.585* 2018-02-08 11:44:01,022 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_wVPHq.17 up (cwd None) (cmdutils:150)*00:04:37.587* 2018-02-08 11:44:01,047 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.588* 2018-02-08 11:44:01,048 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_wVPHq (cwd None) (cmdutils:150)*00:04:37.589* 2018-02-08 11:44:01,062 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.590* 2018-02-08 11:44:01,064 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_wVPHq root (cwd None) (cmdutils:150)*00:04:37.592* 2018-02-08 11:44:01,075 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:158)*00:04:37.594* 2018-02-08 11:44:01,076 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_wVPHq ingress (cwd None) (cmdutils:150)*00:04:37.595* 2018-02-08 11:44:01,087 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:158)*00:04:37.597* 2018-02-08 11:44:01,089 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_wVPHq root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:150)*00:04:37.599* 2018-02-08 11:44:01,100 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.600* 2018-02-08 11:44:01,101 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_wVPHq ingress (cwd None) (cmdutils:150)*00:04:37.601* 2018-02-08 11:44:01,112 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.602* 2018-02-08 11:44:01,113 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_wVPHq parent 1389: classid 1389:10 hfsc ul m2 8000000bit ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.605* 2018-02-08 11:44:01,124 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.606* 2018-02-08 11:44:01,126 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_wVPHq parent 1389: classid 1389:1388 hfsc ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.608* 2018-02-08 11:44:01,137 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.609* 2018-02-08 11:44:01,138 DEBUG (MainThread) [root] /sbin/tc filter replace dev dummy_wVPHq protocol all parent 1389: pref 5000 u32 match u8 0 0 flowid 0x1388 (cwd None) (cmdutils:150)*00:04:37.611* 2018-02-08 11:44:01,149 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.612* 2018-02-08 11:44:01,150 DEBUG (MainThread) [root] /sbin/tc filter replace dev dummy_wVPHq protocol all parent 1389: pref 16 basic match 'meta(vlan eq 16)' flowid 1389:10 (cwd None) (cmdutils:150)*00:04:37.616* 2018-02-08 11:44:01,163 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.617* 2018-02-08 11:44:01,164 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_wVPHq parent 1389:10 handle 10: fq_codel (cwd None) (cmdutils:150)*00:04:37.619* 2018-02-08 11:44:01,176 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.621* 2018-02-08 11:44:01,177 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_wVPHq parent 1389:1388 handle 1388: fq_codel (cwd None) (cmdutils:150)*00:04:37.623* 2018-02-08 11:44:01,188 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.624* 2018-02-08 11:44:01,189 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_wVPHq (cwd None) (cmdutils:150)*00:04:37.625* 2018-02-08 11:44:01,201 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.627* 2018-02-08 11:44:01,202 DEBUG (MainThread) [root] /sbin/tc filter show dev dummy_wVPHq parent 1389: (cwd None) (cmdutils:150)*00:04:37.629* 2018-02-08 11:44:01,214 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.630* 2018-02-08 11:44:01,215 DEBUG (MainThread) [root] /sbin/tc class del dev dummy_wVPHq classid 1389:11 (cwd None) (cmdutils:150)*00:04:37.632* 2018-02-08 11:44:01,226 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:158)*00:04:37.634* 2018-02-08 11:44:01,227 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_wVPHq parent 1389: classid 1389:11 hfsc ul m2 8000000bit ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.637* 2018-02-08 11:44:01,239 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.639* 2018-02-08 11:44:01,248 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.641* 2018-02-08 11:44:01,405 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.643* 2018-02-08 11:44:01,408 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb6436450>>, args=(), kwargs={}) (concurrent:192)*00:04:37.646* 2018-02-08 11:44:01,410 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_wVPHq.17 down (cwd None) (cmdutils:150)*00:04:37.647* 2018-02-08 11:44:01,430 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.648* 2018-02-08 11:44:01,431 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.650* 2018-02-08 11:44:01,432 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_wVPHq.17 (cwd None) (cmdutils:150)*00:04:37.651* 2018-02-08 11:44:01,452 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.652* 2018-02-08 11:44:01,454 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb6436b90>>, args=(), kwargs={}) (concurrent:192)*00:04:37.655* 2018-02-08 11:44:01,456 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_wVPHq.16 down (cwd None) (cmdutils:150)*00:04:37.657* 2018-02-08 11:44:01,475 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.658* 2018-02-08 11:44:01,477 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.660* 2018-02-08 11:44:01,478 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_wVPHq.16 (cwd None) (cmdutils:150)*00:04:37.661* 2018-02-08 11:44:01,498 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.662* --------------------- >> end captured logging << ---------------------*00:04:37.663* *00:04:37.663* ======================================================================*00:04:37.664* ERROR: test_single_vlan(2) (network.tc_test.TestConfigureOutbound)*00:04:37.665* ----------------------------------------------------------------------*00:04:37.666* Traceback (most recent call last):*00:04:37.666* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/testlib.py", line 143, in wrapper*00:04:37.667* return f(self, *args)*00:04:37.668* File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched*00:04:37.668* return func(*args, **keywargs)*00:04:37.669* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/network/tc_test.py", line 433, in test_single_vlan*00:04:37.670* vlan.tag)*00:04:37.671* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 48, in configure_outbound*00:04:37.672* qosOutbound)*00:04:37.672* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 163, in _qdisc_conf_out*00:04:37.674* if not _is_explicit_defined_default_class(dev):*00:04:37.675* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/configurators/qos.py", line 208, in _is_explicit_defined_default_class*00:04:37.677* netinfo = NetInfo(cache_get())*00:04:37.677* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 137, in get*00:04:37.679* return _get(vdsmnets)*00:04:37.679* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 62, in _get*00:04:37.681* devices_info = _devices_report(ipaddrs, routes)*00:04:37.681* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in _devices_report*00:04:37.683* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.684* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/netinfo/cache.py", line 97, in <genexpr>*00:04:37.685* for dev in (link for link in getLinks() if not link.isHidden()):*00:04:37.686* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.688* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.688* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.690* dpdk_devices = _get_dpdk_devices()*00:04:37.690* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.692* value = self.func(*args)*00:04:37.692* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.693* devices = _lshw_command()*00:04:37.694* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.695* return _normalize_lshw_result(out)*00:04:37.696* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.698* return json.loads('[' + result + ']')*00:04:37.698* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.699* return _default_decoder.decode(s)*00:04:37.699* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.700* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.701* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.702* obj, end = self.scan_once(s, idx)*00:04:37.702* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.703* -------------------- >> begin captured logging << --------------------*00:04:37.704* 2018-02-08 11:44:02,101 DEBUG (MainThread) [root] /sbin/ip link add name dummy_ysznk type dummy (cwd None) (cmdutils:150)*00:04:37.705* 2018-02-08 11:44:02,116 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.706* 2018-02-08 11:44:02,117 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ysznk up (cwd None) (cmdutils:150)*00:04:37.708* 2018-02-08 11:44:02,149 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.709* 2018-02-08 11:44:02,151 DEBUG (MainThread) [root] /sbin/ip link add link dummy_ysznk name dummy_ysznk.16 type vlan id 16 (cwd None) (cmdutils:150)*00:04:37.710* 2018-02-08 11:44:02,166 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.712* 2018-02-08 11:44:02,167 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ysznk.16 up (cwd None) (cmdutils:150)*00:04:37.713* 2018-02-08 11:44:02,190 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.714* 2018-02-08 11:44:02,191 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ysznk (cwd None) (cmdutils:150)*00:04:37.715* 2018-02-08 11:44:02,205 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.716* 2018-02-08 11:44:02,206 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ysznk root (cwd None) (cmdutils:150)*00:04:37.718* 2018-02-08 11:44:02,219 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:158)*00:04:37.720* 2018-02-08 11:44:02,220 DEBUG (MainThread) [root] /sbin/tc qdisc del dev dummy_ysznk ingress (cwd None) (cmdutils:150)*00:04:37.721* 2018-02-08 11:44:02,233 DEBUG (MainThread) [root] FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 (cmdutils:158)*00:04:37.723* 2018-02-08 11:44:02,235 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ysznk root handle 0x1389: hfsc default 0x1388 (cwd None) (cmdutils:150)*00:04:37.724* 2018-02-08 11:44:02,248 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.725* 2018-02-08 11:44:02,249 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ysznk ingress (cwd None) (cmdutils:150)*00:04:37.727* 2018-02-08 11:44:02,262 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.728* 2018-02-08 11:44:02,264 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_ysznk parent 1389: classid 1389:10 hfsc ul m2 8000000bit ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.730* 2018-02-08 11:44:02,277 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.732* 2018-02-08 11:44:02,278 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_ysznk parent 1389: classid 1389:1388 hfsc ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.734* 2018-02-08 11:44:02,291 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.735* 2018-02-08 11:44:02,292 DEBUG (MainThread) [root] /sbin/tc filter replace dev dummy_ysznk protocol all parent 1389: pref 5000 u32 match u8 0 0 flowid 0x1388 (cwd None) (cmdutils:150)*00:04:37.738* 2018-02-08 11:44:02,306 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.739* 2018-02-08 11:44:02,307 DEBUG (MainThread) [root] /sbin/tc filter replace dev dummy_ysznk protocol all parent 1389: pref 16 basic match 'meta(vlan eq 16)' flowid 1389:10 (cwd None) (cmdutils:150)*00:04:37.742* 2018-02-08 11:44:02,320 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.743* 2018-02-08 11:44:02,321 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ysznk parent 1389:10 handle 10: fq_codel (cwd None) (cmdutils:150)*00:04:37.745* 2018-02-08 11:44:02,333 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.746* 2018-02-08 11:44:02,334 DEBUG (MainThread) [root] /sbin/tc qdisc add dev dummy_ysznk parent 1389:1388 handle 1388: fq_codel (cwd None) (cmdutils:150)*00:04:37.747* 2018-02-08 11:44:02,346 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.749* 2018-02-08 11:44:02,347 DEBUG (MainThread) [root] /sbin/tc qdisc show dev dummy_ysznk (cwd None) (cmdutils:150)*00:04:37.751* 2018-02-08 11:44:02,359 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.752* 2018-02-08 11:44:02,360 DEBUG (MainThread) [root] /sbin/tc filter show dev dummy_ysznk parent 1389: (cwd None) (cmdutils:150)*00:04:37.754* 2018-02-08 11:44:02,372 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.755* 2018-02-08 11:44:02,374 DEBUG (MainThread) [root] /sbin/tc filter del dev dummy_ysznk pref 16 parent 1389: (cwd None) (cmdutils:150)*00:04:37.757* 2018-02-08 11:44:02,386 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.758* 2018-02-08 11:44:02,387 DEBUG (MainThread) [root] /sbin/tc class del dev dummy_ysznk classid 1389:10 (cwd None) (cmdutils:150)*00:04:37.760* 2018-02-08 11:44:02,398 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.761* 2018-02-08 11:44:02,400 DEBUG (MainThread) [root] /sbin/tc class add dev dummy_ysznk parent 1389: classid 1389:10 hfsc ul m2 8000000bit ls m1 32000000bit d 800000us m2 24000000bit (cwd None) (cmdutils:150)*00:04:37.764* 2018-02-08 11:44:02,412 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.765* 2018-02-08 11:44:02,421 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.768* 2018-02-08 11:44:02,584 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.769* 2018-02-08 11:44:02,587 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140584949905152)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb6436410>>, args=(), kwargs={}) (concurrent:192)*00:04:37.773* 2018-02-08 11:44:02,588 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_ysznk.16 down (cwd None) (cmdutils:150)*00:04:37.775* 2018-02-08 11:44:02,609 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.776* 2018-02-08 11:44:02,610 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140584949905152)> (concurrent:195)*00:04:37.778* 2018-02-08 11:44:02,611 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_ysznk.16 (cwd None) (cmdutils:150)*00:04:37.780* 2018-02-08 11:44:02,636 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.781* --------------------- >> end captured logging << ---------------------*00:04:37.782* *00:04:37.782* ======================================================================*00:04:37.783* ERROR: testHostSampleHandlesDisappearingVlanInterfaces (virttests.sampling_test.InterfaceSampleTests)*00:04:37.784* ----------------------------------------------------------------------*00:04:37.785* Traceback (most recent call last):*00:04:37.785* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:04:37.787* return f(*args, **kwargs)*00:04:37.787* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/virttests/sampling_test.py", line 346, in testHostSampleHandlesDisappearingVlanInterfaces*00:04:37.789* interfaces_and_samples = sampling._get_interfaces_and_samples()*00:04:37.790* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/virt/sampling.py", line 200, in _get_interfaces_and_samples*00:04:37.791* for link in ipwrapper.getLinks():*00:04:37.792* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/virttests/sampling_test.py", line 339, in faultyGetLinks*00:04:37.793* all_links = list(original_getLinks())*00:04:37.794* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.795* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.796* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.798* dpdk_devices = _get_dpdk_devices()*00:04:37.798* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.799* value = self.func(*args)*00:04:37.800* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.801* devices = _lshw_command()*00:04:37.802* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:37.803* return _normalize_lshw_result(out)*00:04:37.804* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:37.805* return json.loads('[' + result + ']')*00:04:37.806* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:37.807* return _default_decoder.decode(s)*00:04:37.807* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:37.808* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:37.809* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:37.810* obj, end = self.scan_once(s, idx)*00:04:37.810* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:37.811* -------------------- >> begin captured logging << --------------------*00:04:37.812* 2018-02-08 11:44:17,378 DEBUG (MainThread) [root] /sbin/ip link add name dummy_8gyyz type dummy (cwd None) (cmdutils:150)*00:04:37.813* 2018-02-08 11:44:17,406 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.814* 2018-02-08 11:44:17,421 DEBUG (netlink/events) [root] START thread <Thread(netlink/events, started daemon 140585068492544)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fdcb6829e10>>, args=(), kwargs={}) (concurrent:192)*00:04:37.817* 2018-02-08 11:44:17,423 DEBUG (MainThread) [root] /sbin/ip link set dev dummy_8gyyz up (cwd None) (cmdutils:150)*00:04:37.819* 2018-02-08 11:44:17,450 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.820* 2018-02-08 11:44:17,454 DEBUG (netlink/events) [root] FINISH thread <Thread(netlink/events, started daemon 140585068492544)> (concurrent:195)*00:04:37.821* 2018-02-08 11:44:17,455 DEBUG (MainThread) [root] /sbin/ip link add link dummy_8gyyz name vlan_669 type vlan id 999 (cwd None) (cmdutils:150)*00:04:37.823* 2018-02-08 11:44:17,477 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.823* 2018-02-08 11:44:17,479 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:37.825* 2018-02-08 11:44:17,657 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.825* 2018-02-08 11:44:17,658 DEBUG (MainThread) [root] /sbin/ip link del dev vlan_669 (cwd None) (cmdutils:150)*00:04:37.826* 2018-02-08 11:44:17,681 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.826* 2018-02-08 11:44:17,683 DEBUG (MainThread) [root] /sbin/ip link del dev dummy_8gyyz (cwd None) (cmdutils:150)*00:04:37.827* 2018-02-08 11:44:17,708 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:37.828* --------------------- >> end captured logging << ---------------------*00:04:37.828* *00:04:37.828* ======================================================================*00:04:37.828* ERROR: testHostSampleReportsNewInterface (virttests.sampling_test.InterfaceSampleTests)*00:04:37.829* ----------------------------------------------------------------------*00:04:37.829* Traceback (most recent call last):*00:04:37.830* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:04:37.830* return f(*args, **kwargs)*00:04:37.831* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/tests/virttests/sampling_test.py", line 326, in testHostSampleReportsNewInterface*00:04:37.831* sampling._get_interfaces_and_samples()))*00:04:37.832* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/virt/sampling.py", line 200, in _get_interfaces_and_samples*00:04:37.833* for link in ipwrapper.getLinks():*00:04:37.833* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 267, in getLinks*00:04:37.834* in six.viewitems(dpdk.get_dpdk_devices()))*00:04:37.834* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 44, in get_dpdk_devices*00:04:37.835* dpdk_devices = _get_dpdk_devices()*00:04:37.835* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/common/cache.py", line 41, in __call__*00:04:37.836* value = self.func(*args)*00:04:37.836* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 111, in _get_dpdk_devices*00:04:37.837* devices = _lshw_command()*00:04:37.837* File "/home/jeCoverage.py warning: Module /home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/vdsm was never imported. (module-not-imported)*00:04:51.224* nkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 127, in _lshw_command*00:04:51.225* return _normalize_lshw_result(out)*00:04:51.226* File "/home/jenkins/workspace/vdsm_master_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/link/dpdk.py", line 182, in _normalize_lshw_result*00:04:51.227* return json.loads('[' + result + ']')*00:04:51.228* File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads*00:04:51.229* return _default_decoder.decode(s)*00:04:51.229* File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode*00:04:51.230* obj, end = self.raw_decode(s, idx=_w(s, 0).end())*00:04:51.231* File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode*00:04:51.231* obj, end = self.scan_once(s, idx)*00:04:51.232* ValueError: Expecting , delimiter: line 11 column 8 (char 242)*00:04:51.233* -------------------- >> begin captured logging << --------------------*00:04:51.234* 2018-02-08 11:44:17,714 DEBUG (MainThread) [root] lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None) (cmdutils:150)*00:04:51.237* 2018-02-08 11:44:17,871 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (cmdutils:158)*00:04:51.238* --------------------- >> end captured logging << ---------------------

3 9

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 07-02-2018 ] [ 004_basic_sanity.vm_run ]
by Dafna Ron 11 Feb '18

11 Feb '18

Hi, We failed test 004_basic_sanity.vm_run in the upgrade from release suite. the reason we failed to run the vm is that the host was filtered from selection for vm. Please also notice other errors in the log relating to network sync failing. *Link and headline of suspected patches: https://gerrit.ovirt.org/#/c/87116/5 <https://gerrit.ovirt.org/#/c/87116/5> - core: Add 4.3 support Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5307/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5307/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5307/art… <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5307/artifact/>(Relevant) error snippet from the log: <error>2018-02-06 16:42:39,431-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4d0244492018-02-06 16:42:39,472-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] Running command: RunVmOnceCommand internal: false. Entities affected : ID: ccfe9852-f2d6-4e18-8213-29d958d3a8ed Type: VMAction group RUN_VM with role type USER, ID: ccfe9852-f2d6-4e18-8213-29d958d3a8ed Type: VMAction group EDIT_ADMIN_VM_PROPERTIES with role type ADMIN2018-02-06 16:42:39,486-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] Candidate host 'lago-upgrade-from-release-suite-master-host0' ('6ec7721d-beae-4b7e-807f-7f3a794fb7c4') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 03cdc0e6-eb94-46a4-9882-77762b2ed787)2018-02-06 16:42:39,486-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] Can't find VDS to run the VM 'ccfe9852-f2d6-4e18-8213-29d958d3a8ed' on, so this VM will not be run.2018-02-06 16:42:39,493-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm0 (User: admin(a)internal-authz).2018-02-06 16:42:39,500-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-5) [03cdc0e6-eb94-46a4-9882-77762b2ed787] Lock freed to object 'EngineLock:{exclusiveLocks='[ccfe9852-f2d6-4e18-8213-29d958d3a8ed=VM]', sharedLocks=''}'2018-02-06 16:42:39,505-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-5) [] Operation Failed: []2018-02-06 16:42:39,512-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-17) [4fe81490] Running command: ProcessDownVmCommand internal: true.(END)</error>*

6 8

OST failures - weekly update [02/02/2018-09/02/2018]
by Dafna Ron 09 Feb '18

09 Feb '18

Hello, I would like to update the OST failures and status for the previous week. before I give the report I would like to say that I have started working on improving our weekly reports and allow better analyses and review of failures so I hope each week the report would become better and more informative. This week was eventful due to the 4.2 branch being stabilized and also starting to merge some 4.3 related patches to master. *We are still failing on the following tests: * - 004_basic_sanity.vm_run - cannot find a acceptable vds ( https://gerrit.ovirt.org/#/c/87181/ and https://gerrit.ovirt.org/#/c/87116/) - 001_upgrade_engine.test_initialize_engine - dwhd.service not starting - 002_bootstrap.get_host_devices - changes to sdk return with no vda1 *Major issues that were resolved during the week: * - 002_bootstrap.add_cluster - https://gerrit.ovirt.org/#/c/87126/ - Infra packaging issue - https://gerrit.ovirt.org/87183 - 002_bootstrap.verify_add_hosts - https://gerrit.ovirt.org/#/c/87386/ I prepared some chart based on this week's alerts and I hope to improve on this in the future. Please note that i issued the reports based on alerts sent by CQ hence these are not separate issues but alerts sent by the system. *Alerts on resolved issues vs alerts on unresolved issues: * Done = alerts on resolved issues Backlog = alerts on unresolved issues - these will go to next week's statistics. [image: Inline image 5] [image: Inline image 6] *the below chart is of OST resolved failures based on cause of failure:* Code= regression of working components/functionalities None= OST did not start to run (this is usually caused by failed artifacts builds) Infra = packaging and other issues related to our infrastructure Other = lago/OST related issues Unknown = anything no falling in those definitions [image: Inline image 2] [image: Inline image 1] *The below is resolved failures based on Suite type: * Basic = Basic Suite Upgrade - Upgrade from release suite None = patches that failed because OST did not run (build-artifacts issues) [image: Inline image 3] [image: Inline image 4] Failures based on Ovirt Version: [image: Inline image 7] [image: Inline image 8] If you have any requests or suggestions to improve the reporting please let me know. Thank you, Dafna

1 0

[vdsm] stable branch ovirt-4.2 created
by Francesco Romani 09 Feb '18

09 Feb '18

Hi all, With the help of Sandro (many thanks @sbonazzo !), we created minutes ago the ovirt-4.2 stable branch: Steps performed: 1. merged https://gerrit.ovirt.org/#/c/87070/ 2. branched out ovirt-4.2 from git master 3. merged https://gerrit.ovirt.org/#/c/87181/ to add support for 4.3 level 4. createed and pushed the tag v4.30.0 from master, to make sure the version number is greater of the stable versions, and to (somehow :)) align with oVirt versioning 5. tested make dist/make rpm on both new branch ovirt-4.2 and master, both looks good and use the right version Maintainers, please check it looks right for you before merging any new patch to master branch. Please let me know about any issue! Happy hacking, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

7 14

[ OST Failure Report ] [ oVirt Master (vdsm) ] [ 08-02-2018 ] [ 002_bootstrap.get_host_numa_nodes+ 004_basic_sanity.vm_run]
by Dafna Ron 08 Feb '18

08 Feb '18

Hi, We have a failure on 002_bootstrap.get_host_numa_nodes in basic suite and 004_basic_sanity.vm_run on upgrade from release suite. Eli, can you please take a look? it seems to be the same reason - host is down or not suitable *Link and headline of suspected patches: Link to Job:Link to all logs:(Relevant) error snippet from the log: <error>* *basic suite *: 2018-02-08 00:14:23,852-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (ServerService Thread Pool -- 41) [] SqlCall for procedure [GetAllFromVdcOption] compiled 2018-02-08 00:14:23,864-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'ConfigDir' 2018-02-08 00:14:23,869-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'DbJustRestored' 2018-02-08 00:14:23,871-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'ConfigDir' 2018-02-08 00:14:23,881-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'DbJustRestored' 2018-02-08 00:14:23,882-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'ConfigDir' 2018-02-08 00:14:23,915-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) [] Could not find enum value for option: 'DbJustRestored' 2018-02-08 00:14:23,919-05 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 41) [] Loading file '/etc/ovirt-engine/osinfo.conf.d/00-defaults.properties' 2018-02-08 00:14:23,971-05 DEBUG [org.ovirt.engine.core.utils.OsRepositoryImpl] (ServerService Thread Pool -- 41) [] Osinfo Repository: backwardCompatibility OtherLinux=5 Windows2008R2x64=17 Windows2003=3 Windows2003x64=10 RHEL3x64=15 Windows8x64=21 Windows8=20 Windows7=11 Windows7x64=12 Windows2008=4 RHEL4=8 RHEL6x64=19 RHEL5x64=13 Windows2012x64=23 WindowsXP=1 RHEL4x64=14 Unassigned=0 Windows2008x64=16 RHEL6=18 RHEL5=7 Other=0 REHL3=9 emptyNode os.debian_7.derivedFrom value=ubuntu_12_04 os.debian_7.id value=1300 os.debian_7.name value=Debian 7 os.freebsd.bus value=32 os.freebsd.derivedFrom value=other os.freebsd.id value=1500 os.freebsd.name value=FreeBSD 9.2 os.freebsdx64.bus value=64 os.freebsdx64.derivedFrom : upgrade suite: 2018-02-08 00:12:07,451-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 51) [] Could not find enum value for option: 'DbJustRestored' 2018-02-08 00:12:07,451-05 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 51) [] Could not find enum value for option: 'ConfigDir' 2018-02-08 00:12:07,452-05 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 51) [] Loading file '/etc/ovirt-engine/osinfo.conf.d/00-defaults.properties' 2018-02-08 00:12:07,505-05 DEBUG [org.ovirt.engine.core.utils.OsRepositoryImpl] (ServerService Thread Pool -- 51) [] Osinfo Repository: backwardCompatibility OtherLinux=5 Windows2008R2x64=17 Windows2003=3 Windows2003x64=10 RHEL3x64=15 Windows8x64=21 Windows8=20 Windows7=11 Windows7x64=12 Windows2008=4 RHEL4=8 RHEL6x64=19 RHEL5x64=13 Windows2012x64=23 WindowsXP=1 RHEL4x64=14 Unassigned=0 Windows2008x64=16 RHEL6=18 RHEL5=7 Other=0 REHL3=9 emptyNode os.debian_7.derivedFrom value=ubuntu_12_04 os.debian_7.id value=1300 os.debian_7.name value=Debian 7 os.freebsd.bus value=32 os.freebsd.derivedFrom value=other os.freebsd.id value=1500 os.freebsd.name value=FreeBSD 9.2 os.freebsdx64.bus value=64 os.freebsdx64.derivedFrom value=freebsd os.freebsdx64.id value=1501 os.freebsdx64.name value=FreeBSD 9.2 x64 os.other.bus value=64 os.other.cpu.hotplugSupport value=true os.other.cpu.hotunplugSupport value=false os.other.cpu.unsupported value= *</error>*

2 5

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 08-02-2018 ] [ 001_initialize_engine.initialize_engine ]
by Dafna Ron 08 Feb '18

08 Feb '18

Hi, We are failing test 001_initialize_engine.initialize_engine It seems that the issue is related to the ovirt-engine-dwhd.service not starting. *Link and headline of suspected patches: https://gerrit.ovirt.org/#/c/86920/ <https://gerrit.ovirt.org/#/c/86920/> - * *core: adding key to bundles/AuditLogMessagesLink to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5365/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5365/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5365/art… <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5365/artifact/>(Relevant) error snippet from the log: <error>Feb 7 22:43:23 lago-basic-suite-master-engine dbus[525]: [system] Reloaded configurationFeb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'beyond-scope' is not supported by the kernel for ipv6.Feb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: beyond-scope: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'failed-policy' is not supported by the kernel for ipv6.Feb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: failed-policy: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'reject-route' is not supported by the kernel for ipv6.Feb 7 22:43:24 lago-basic-suite-master-engine firewalld[565]: WARNING: reject-route: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'beyond-scope' is not supported by the kernel for ipv6.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: beyond-scope: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'failed-policy' is not supported by the kernel for ipv6.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: failed-policy: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: ICMP type 'reject-route' is not supported by the kernel for ipv6.Feb 7 22:43:28 lago-basic-suite-master-engine firewalld[565]: WARNING: reject-route: INVALID_ICMPTYPE: No supported ICMP type., ignoring for run-time.Feb 7 22:43:28 lago-basic-suite-master-engine systemd: Starting oVirt Engine fence_kdump listener...Feb 7 22:43:28 lago-basic-suite-master-engine systemd: Started oVirt Engine fence_kdump listener.Feb 7 22:43:28 lago-basic-suite-master-engine systemd: Reloading.Feb 7 22:43:28 lago-basic-suite-master-engine systemd: Starting oVirt Engine...Feb 7 22:43:28 lago-basic-suite-master-engine journal: 2018-02-07 22:43:28,908-0500 ovirt-engine: INFO _detectJBossVersion:187 Detecting JBoss version. Running: /usr/lib/jvm/jre/bin/java ['ovirt-engine-version', '-server', '-XX:+TieredCompilation', '-Xms1024M', '-Xmx1024M', '-Djava.awt.headless=true', '-Dsun.rmi.dgc.client.gcInterval=3600000', '-Dsun.rmi.dgc.server.gcInterval=3600000', '-Djsse.enableSNIExtension=false', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:HeapDumpPath=/var/log/ovirt-engine/dump', '-Djava.util.logging.manager=org.jboss.logmanager', '-Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties', '-Dorg.jboss.resolver.warning=true', '-Djboss.modules.system.pkgs=org.jboss.byteman', '-Djboss.server.default.config=ovirt-engine', '-Djboss.home.dir=/usr/share/ovirt-engine-wildfly', '-Djboss.server.base.dir=/usr/share/ovirt-engine', '-Djboss.server.data.dir=/var/lib/ovirt-engine', '-Djboss.server.log.dir=/var/log/ovirt-engine', '-Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config', '-Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp', '-Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp', '-jar', '/usr/share/ovirt-engine-wildfly/jboss-modules.jar', '-mp', '/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules', '-jaxpmodule', 'javax.xml.jaxp-provider', 'org.jboss.as.standalone', '-v']Feb 7 22:43:29 lago-basic-suite-master-engine journal: 2018-02-07 22:43:29,481-0500 ovirt-engine: INFO _detectJBossVersion:207 Return code: 0, | stdout: '[u'WildFly Full 11.0.0.Final (WildFly Core 3.0.8.Final)'], | stderr: '[]'Feb 7 22:43:29 lago-basic-suite-master-engine systemd: Started oVirt Engine.Feb 7 22:43:29 lago-basic-suite-master-engine systemd: Reloading.Feb 7 22:43:29 lago-basic-suite-master-engine systemd: Starting oVirt Engine Data Warehouse...Feb 7 22:43:29 lago-basic-suite-master-engine systemd: Started oVirt Engine Data Warehouse.Feb 7 22:43:29 lago-basic-suite-master-engine systemd: Reloading.Feb 7 22:43:30 lago-basic-suite-master-engine systemd: Starting oVirt ImageIO Proxy...Feb 7 22:43:30 lago-basic-suite-master-engine systemd: Started oVirt ImageIO Proxy.Feb 7 22:43:30 lago-basic-suite-master-engine systemd: Reloading.Feb 7 22:43:30 lago-basic-suite-master-engine journal: 2018-02-07 22:43:30,301-0500 ovirt-engine-dwhd: ERROR run:554 Error: process terminated with status code 1Feb 7 22:43:30 lago-basic-suite-master-engine systemd: ovirt-engine-dwhd.service: main process exited, code=exited, status=1/FAILUREFeb 7 22:43:30 lago-basic-suite-master-engine systemd: Unit ovirt-engine-dwhd.service entered failed state.Feb 7 22:43:30 lago-basic-suite-master-engine systemd: ovirt-engine-dwhd.service failed.</error>*

1 0

[OST][HC] HE fails to deploy
by Sahina Bose 07 Feb '18

07 Feb '18

Hi all, I see the HE fails to deploy after task in running ansible playbook create_target_vm : TASK [Wait for the engine to come up on the target VM]", with Error engine state=EngineUnexpectedlyDown Is this a known issue that you are working on? thanks! sahina Full HE setup log at http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_…

2 3

ATTN: oVirt Engine 4.2.z has been branched from master
by Tal Nisan 06 Feb '18

06 Feb '18

As the subject says, we've branched today to ovirt-engine-4.2 branch, every change that is required for 4.2.2 and onward should be backported to this branch. Note that we chose now so there will be no confusion on whether your patches are included, if your patch got merge in master before 21:45 IST (20:45 CET) then no backport is needed. Please note that though we branched out the main focus should still be 4.2.2.

1 0

[vdsm][RFC] reconsidering branching out ovirt-4.2
by Francesco Romani 06 Feb '18

06 Feb '18

Hi all, It is time again to reconsider branching out the 4.2 stable branch. So far we decided to *not* branch out, and we are taking tags for ovirt 4.2 releases from master branch. This means we are merging safe and/or stabilization patches only in master. I think it is time to reconsider this decision and branch out for 4.2, because of two reasons: 1. it sends a clearer signal that 4.2 is going in stabilization mode 2. we have requests from virt team, which wants to start working on the next cycle features. If we decide to branch out, I'd start the new branch on monday, February 5 (1 week from now). The discussion is open, please share your acks/nacks for branching out, and for the branching date. I for myself I'm inclined to branch out, so if noone chimes in (!!) I'll execute the above plan. -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

5 5

[ACTION-REQUIRED] Making accurate CI for oVirt 4.2
by Barak Korren 06 Feb '18

06 Feb '18

Hi, This message is aimed for project maintainers. Other developers may also find it interesting to have a glimpse at the oVirt-wide test and composition processes. TL;DR: To get accurate CI for oVirt 4.2, most projects need to add 4.2 jobs in YAML. Before I can explain what the current issue is and which action is required, I'm need to provide a brief overview into how oVirt CI works. oVirt CI has two major components to it: 1. The STDCI component which is used to build and test individual projects. Most developers interact with this on a daily basis as it is responding on GitHub and Gerrit events. 2. The "change-queue" (CQ) component which is used to automatically compose the whole of oVirt from its sub projects and run system tests (OST) on it. This component is used to gather the information that is used by the infra team to compose the "OST failure report" you can occasionally see being sent to this list. The change queue is also used to automatically maintain the 'tested' and '*-snapshot' (AKA nightly) repositories. The way the CQ composes oVirt is by looking at the post-merge STDCI 'build-artifacts' jobs, and collecting together artifacts built by jobs that target a specific oVirt version into that version's change queue. Essentially the '*_master_build-artifacts-*' jobs go into the 'ovirt-master' change queue, the '*_4.1_build-artifacts-*' jobs go into the 'ovirt-4.1' change queue etc. Over the course of the oVirt 4.2 development, most project used their 'master' branch, which is typically mapped to '*_master_*' jobs, for developing 4.2 code. So the 'ovirt-master' CQ provided good indication of the state of 4.2 code. As projects started addeing 4.2 branches, we have created an 'ovirt-4.2' CQ to gather them. We did so under the assumption that most projects will branch soon after. The assumption turned up to be wrong as most projects did not yet fork and may not do so in the near future. As some projects did fork, the end result is that currently: ___there is no accurate representation of oVirt 4.2 in CI___ 'ovirt-master' CQ no longer represents oVirt 4.2 as some projects already have some 4.3 code in their 'master' branches. 'ovirt-4.2' CQ does not represent oVirt 4.2 as most projects do not push artifacts into it. To get any benefit from CI, we need to have it represent what we are going to release. This means that at this point we need all projects to have '*_4.2_build-artifacts-*' jobs that map to the code that is intended to be included in oVirt 4.2. Projects can either: 1. Create 4.2 branches and map the new jobs to them. 2. Keep 4.2 development in 'master' and create '4.2' jobs that map to it. Taking route #2 means a commitment to not adding any 4.3 code to the 'master' branch. Please keep it, as rolling back "too new" builds from the various repos and caches we have is very difficult. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

7 15

Re: [ovirt-devel] [CQ]: 85992, 1 (ovirt-wgt) failed "ovirt-master" system tests
by Sandro Bonazzola 06 Feb '18

06 Feb '18

This failed because the following builds failed: vdsm_master_check-merged-el7-x86_64 <http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/> #3147 <http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3147/> I think it failed on network/functional/static_ip_test.py ....................+ res=124 But I let VDSM team to review. ovirt-wgt_master_build-artifacts-fc27-x86_64 <http://jenkins.ovirt.org/job/ovirt-wgt_master_build-artifacts-fc27-x86_64/> #2 <http://jenkins.ovirt.org/job/ovirt-wgt_master_build-artifacts-fc27-x86_64/2/> ovirt-wgt_master_build-artifacts-fcraw-x86_64 <http://jenkins.ovirt.org/job/ovirt-wgt_master_build-artifacts-fcraw-x86_64/> #2 <http://jenkins.ovirt.org/job/ovirt-wgt_master_build-artifacts-fcraw-x86_64/…> wgt failed on missing ovirt-guest-agent-windows which was failing due to rich dependency resolution not possible using yum. Fix has been pushed to ovirt-guset-agent: https://gerrit.ovirt.org/#/c/87072/ Once this is merged, guest agent will build on Fedora providing the needed dependency and will fix above 2 jobs. 2018-02-05 22:13 GMT+01:00 oVirt Jenkins <jenkins(a)ovirt.org>: > Change 85992,1 (ovirt-wgt) is probably the reason behind recent system test > failures in the "ovirt-master" change queue and needs to be fixed. > > This change had been removed from the testing queue. Artifacts build from > this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/85992/1 > > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5287/ > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

Re: [ovirt-devel] fcraw Jenkins build fails on import error
by Eyal Edri 05 Feb '18

05 Feb '18

also adding devel, it looks more dev issue than infra. On Mon, Feb 5, 2018 at 2:59 PM, Eyal Edri <eedri(a)redhat.com> wrote: > Adding Sandro, > Are we officially supporting fcraw for engine? > > On Mon, Feb 5, 2018 at 2:26 PM, Tal Nisan <tnisan(a)redhat.com> wrote: > >> >> *http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fcraw-x86_64/29/consoleFull <http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fcraw-x86_64/2…>* >> >> >> >> *10:53:31* ==================================== ERRORS ====================================*10:53:31* ERROR collecting packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py *10:53:31* ImportError while importing test module '/home/jenkins/workspace/ovirt-engine_master_check-patch-fcraw-x86_64/ovirt-engine/packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py'.*10:53:31* Hint: make sure your test modules/packages have valid Python names.*10:53:31* Traceback:*10:53:31* packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py:19: in <module>*10:53:31* import ovirt_engine_setup.engine_common.database as under_test # isort:skip # noqa: E402*10:53:31* packaging/setup/ovirt_engine_setup/engine_common/database.py:30: in <module>*10:53:31* from otopi import base*10:53:31* E ImportError: No module named otopi*10:53:31* !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!*10:53:31* =========================== 1 error in 0.43 seconds ============================ >> >> >> >> >> >> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> >> > > > -- > > Eyal edri > > > MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) > -- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[IOPROCESS] New release for Fedora
by Nir Soffer 04 Feb '18

04 Feb '18

Hi all, I released ioprocess 1.0.0 for Fedora 27 and 28. If you are using Fedora, please install the new version from the updates-testing and test it. Please share your feedback here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2 This version will be available soon from oVirt repositories. Thanks, Nir

1 1