[OST Failure Report][oVirt master][2017-04-11] add_secondary_storage_domains
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303 *OST is failing in the last 50 builds* *00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up. -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine: 2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b ..... 2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689 \00 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/6303
*OST is failing in the last 50 builds*
*00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 On 11 April 2017 at 18:12, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:
2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool. RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool. RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker. PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker. PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core. vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b
.....
2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc. client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689
\00 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc. client.reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma ster/6303
*OST is failing in the last 50 builds*
*00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com> wrote:
This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html
That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307
That's an entirely different game: *2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils* 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_121] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) [bll.jar:] ... Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:]
On 11 April 2017 at 18:12, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:
2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.db broker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core. vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b
.....
2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689
\00 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client .reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Link to Job: http://jenkins.ovirt.org/job/t est-repo_ovirt_experimental_master/6303
*OST is failing in the last 50 builds*
*00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul@redhat.com> wrote: =20 =20 =20 On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com = <mailto:bkorren@redhat.com>> wrote: This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html = <http://lists.ovirt.org/pipermail/devel/2017-April/030139.html> =20 That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307> =20 That's an entirely different game: 2017-04-11 11:24:36,783-04 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: = java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils 2017-04-11 11:24:36,783-04 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: = java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
org/apache/commons/lang/StringUtils at java.util.concurrent.FutureTask.report(FutureTask.java:122) = [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) = [rt.jar:1.8.0_121] at = org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo= mmand.java:69) [vdsbroker.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos= tSetupNetworkPoller.java:56) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet= upNetworkPoller.java:41) [bll.jar:] =20 ... Caused by: java.lang.NoClassDefFoundError: = org/apache/commons/lang/StringUtils at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationI= d(Message.java:75) [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SS= LStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) = [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91= ) [vdsbroker.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundP=
at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:484) [vdsbroker.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:473) [vdsbroker.jar:] =20 =20 =20 =20 =20 On 11 April 2017 at 18:12, Piotr Kliczewski = <piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote: I looks like SD activate failed on the engine side and the spm = connection was gently closed by the engine: =20 2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st = <http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCo= mmand] (default task-29) [2b66a42b] ActivateStorage Domain. After = Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] = (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds = 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to = non-operational: null 2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] = (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds = 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to = non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se = <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] = (default task-29) [2bc83b1c] Running command: = SetNonOperationalVdsCommand internal: true. Entities affected : ID: = e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName =3D = lago-basic-suite-master-host0, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'e1b611f2-67fb-4acb-8261-07133134e0e7', = status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 5c45239 2017-04-11 09:10:24,919-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call = string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure = [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (default task-29) [2bc83b1c] EVENT_ID: = VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: = d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: = -1, Message: Host lago-basic-suite-master-host0 cannot access the = Storage Domain(s) iscsi attached to the Data Center test-dc. Setting = Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG = [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] = (DefaultQuartzScheduler4) [] Rescheduling = DEFAULT.org.ovirt.engine.core.vdsbroker.VdsManager.onTimer#-92233720368547= 75741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se = <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] = (default task-29) [2ffa5981] Running command: = SetNonOperationalVdsCommand internal: true. Entities affected : ID: = 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName =3D = lago-basic-suite-master-host1, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'87a5f691-ea5c-4692-a4ab-52dad0dafe34', = status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 706ffc3b =20 ..... =20 2017-04-11 09:10:25,366-04 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default = task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default = task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689 =20 \00 2017-04-11 09:10:25,367-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] = (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE =20 =20 =20 On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com = <mailto:sbonazzo@redhat.com>> wrote: Link to Job: = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303 = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303> =20 OST is failing in the last 50 builds =20 =20 00:18:22.090 [basic_suit_el7] # add_secondary_storage_domains:=20 00:18:22.090 [basic_suit_el7] Error while running thread 00:18:22.090 [basic_suit_el7] Traceback (most recent call last): 00:18:22.090 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.090 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.090 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in = add_nfs_storage_domain 00:18:22.090 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, = SD_NFS_PATH, nfs_version=3D'v4_2') 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.091 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.091 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.091 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.091 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.092 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.092 [basic_suit_el7] Error while running thread 00:18:22.092 [basic_suit_el7] Traceback (most recent call last): 00:18:22.092 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.092 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.092 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in = add_iso_storage_domain 00:18:22.092 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, = SD_ISO_PATH, sd_format=3D'v1', sd_type=3D'iso', nfs_version=3D'v3') 00:18:22.092 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.092 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.093 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.093 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.093 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.093 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.094 [basic_suit_el7] Error while running thread 00:18:22.094 [basic_suit_el7] Traceback (most recent call last): 00:18:22.094 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.094 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in = add_templates_storage_domain 00:18:22.094 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, = SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format=3D'v1', = sd_type=3D'export', nfs_version=3D'v4_1') 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.094 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.095 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.095 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.095 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.095 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.095 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.095 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.095 [basic_suit_el7] Error while running thread 00:18:22.095 [basic_suit_el7] Traceback (most recent call last): 00:18:22.096 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.096 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in = add_second_nfs_storage_domain 00:18:22.096 [basic_suit_el7] SD_NFS_HOST_NAME, = SD_SECOND_NFS_PATH) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.096 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.096 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.097 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.097 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.097 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.097 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.097 [basic_suit_el7] Error while running thread 00:18:22.097 [basic_suit_el7] Traceback (most recent call last): 00:18:22.097 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.098 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.098 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in = import_template_from_glance 00:18:22.098 [basic_suit_el7] generic_import_from_glance(api, = image_name=3DCIRROS_IMAGE_NAME, image_ext=3D'_glance_template', = as_template=3DTrue) 00:18:22.098 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in = generic_import_from_glance 00:18:22.098 [basic_suit_el7] = target_image.import_image(import_action) 00:18:22.098 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", =
00:18:22.098 [basic_suit_el7] =
00:18:22.098 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", =
00:18:22.098 [basic_suit_el7] =
00:18:22.099 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
00:18:22.099 [basic_suit_el7] persistent_auth) 00:18:22.099 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
this one started in = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/> there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what = seems to be mismatched Piotr/Sandro, are all artifacts published correctly? Thanks, michal oll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] line 26017, in import_image headers=3D{"Correlation-Id":correlation_id} line 122, in request persistent_auth=3Dself.__persistent_auth py", line 79, in do_request py", line 162, in __do_request
00:18:22.099 [basic_suit_el7] raise = errors.RequestError(response_code, response_reason, response_body) 00:18:22.099 [basic_suit_el7] RequestError:=20 00:18:22.099 [basic_suit_el7] status: 400 00:18:22.099 [basic_suit_el7] reason: Bad Request 00:18:22.099 [basic_suit_el7] detail: Cannot import Virtual Disk: = Storage Domain cannot be accessed. 00:18:22.099 [basic_suit_el7] -Please check that at least one Host is = operational and Data Center state is up. =20 --=20 SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat=C2=A0EMEA <https://www.redhat.com/> <https://red.ht/sig>=09 TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 =20 --=20 Barak Korren bkorren@redhat.com <mailto:bkorren@redhat.com> RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/> _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2017, at 08:37, Yaniv Kaul <<a = href=3D"mailto:ykaul@redhat.com" class=3D"">ykaul@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Wed, Apr 12, 2017 at 9:24 AM, = Barak Korren <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D""><div class=3D"">This seems to have been the = Secodary SD issue I reported on Sunday:<br class=3D""><a = href=3D"http://lists.ovirt.org/pipermail/devel/2017-April/030139.html" = target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">pipermail/devel/2017-April/<wbr class=3D"">030139.html</a><br = class=3D""><br class=3D""></div>That was up until build #6307 where = AddHost started failing:<br class=3D""><a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 307" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/<wbr = class=3D"">test-repo_ovirt_experimental_<wbr = class=3D"">master/6307</a></div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">That's an entirely different = game:</div><div class=3D""><b class=3D"">2017-04-11 11:24:36,783-04 = ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: = java.lang.NoClassDefFoundError: = org/apache/commons/lang/StringUtils</b></div><div class=3D"">2017-04-11 = 11:24:36,783-04 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: = java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: = </div></div></div></div></div></blockquote><div><br class=3D""></div>this = one started in <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 307/" = class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_maste= r/6307/</a></div><div><br class=3D""></div><div>there are 2 patches = linkded, one is in vdsm-jsonrpc-java, which is what seems to be = mismatched</div><div>Piotr/Sandro, are all artifacts published = correctly?</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><div = class=3D"">org/apache/commons/lang/StringUtils</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at java.util.concurrent.FutureTask.report(FutureTask.java:122) = [rt.jar:1.8.0_121]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at java.util.concurrent.FutureTask.get(FutureTask.java:206) = [rt.jar:1.8.0_121]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo= mmand.java:69) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos= tSetupNetworkPoller.java:56) [bll.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet= upNetworkPoller.java:41) [bll.jar:]</div><div class=3D""><br = class=3D""></div><div class=3D"">...</div><div class=3D"">Caused by: = java.lang.NoClassDefFoundError: = org/apache/commons/lang/StringUtils</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationI= d(Message.java:75) [vdsm-jsonrpc-java-client.jar:]</div><div = class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">= </span>at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SS= LStompClient.java:85) [vdsm-jsonrpc-java-client.jar:]</div><div = class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">= </span>at = org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) = [vdsm-jsonrpc-java-client.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.j= ava:91) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundP= oll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:]</div><div = class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">= </span>at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:484) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> = </span>at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:473) [vdsbroker.jar:]</div><div class=3D""><br = class=3D""></div><div class=3D""> </div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D""><br = class=3D""><br class=3D""><div class=3D""><div class=3D""><div = class=3D"gmail_extra"><div class=3D""><div class=3D"gmail-h5"><br = class=3D""><div class=3D"gmail_quote">On 11 April 2017 at 18:12, Piotr = Kliczewski <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:piotr.kliczewski@gmail.com" target=3D"_blank" = class=3D"">piotr.kliczewski@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D"">I looks like SD activate failed on the engine = side and the spm connection was gently closed by the engine:<div = class=3D""><br class=3D""></div><div class=3D""><div class=3D"">2017-04-11= 09:10:24,807-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.st/" = class=3D"">org.ovirt.engine.core.bll.st</a><wbr = class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand] = (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all = hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017</div><div = class=3D"">2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.sto<wbr = class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] = (org.ovirt.thread.pool-7-threa<wbr class=3D"">d-29) [2b66a42b] Could not = connect vds 'lago-basic-suite-master-host0<wbr class=3D"">' to pool = 'test-dc' - moving host to non-operational: null</div><div = class=3D"">2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.sto<wbr = class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] = (org.ovirt.thread.pool-7-threa<wbr class=3D"">d-28) [2b66a42b] Could not = connect vds 'lago-basic-suite-master-host1<wbr class=3D"">' to pool = 'test-dc' - moving host to non-operational: null</div><div = class=3D"">2017-04-11 09:10:24,904-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.se/" = class=3D"">org.ovirt.engine.core.bll.Se</a><wbr = class=3D"">tNonOperationalVdsCommand] (default task-29) [2bc83b1c] = Running command: SetNonOperationalVdsCommand internal: true. Entities = affected : ID: e1b611f2-67fb-4acb-8261-071331<wbr class=3D"">34e0e7 = Type: VDS</div><div class=3D"">2017-04-11 09:10:24,909-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] = START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParamete<wbr = class=3D"">rs:{runAsync=3D'true', hostId=3D'e1b611f2-67fb-4acb-826<wbr = class=3D"">1-07133134e0e7', status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 5c45239</div><div class=3D"">2017-04-11 09:10:24,919-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] = FINISH, SetVdsStatusVDSCommand, log id: 5c45239</div><div = class=3D"">2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbb<wbr = class=3D"">roker.PostgresDbEngineDialect$<wbr = class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled = stored procedure. Call string is [{call = getvmsrunningonvds(?)}]</div><div class=3D"">2017-04-11 09:10:24,922-04 = DEBUG [org.ovirt.engine.core.dal.dbb<wbr = class=3D"">roker.PostgresDbEngineDialect$<wbr = class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall = for procedure [GetVmsRunningOnVds] compiled</div><div = class=3D"">2017-04-11 09:10:24,985-04 WARN = [org.ovirt.engine.core.dal.db<wbr = class=3D"">broker.auditloghandling.AuditL<wbr class=3D"">ogDirector] = (default task-29) [2bc83b1c] EVENT_ID: = VDS_SET_NONOPERATIONAL_DOMAIN(<wbr class=3D"">522), Correlation ID: = 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e98269<wbr class=3D"">44b9a0, = Call Stack: null, Custom Event ID: -1, Message: Host = lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi = attached to the Data Center test-dc. Setting Host state to = Non-Operational.</div><div class=3D"">2017-04-11 09:10:24,985-04 DEBUG = [org.ovirt.engine.core.utils.t<wbr class=3D"">imer.FixedDelayJobListener] = (DefaultQuartzScheduler4) [] Rescheduling <a href=3D"http://DEFAULT.org" = class=3D"">DEFAULT.org</a>.ovirt.engine.core.<wbr = class=3D"">vdsbroker.VdsManager.onTimer#-<wbr = class=3D"">9223372036854775741 as there is no unfired trigger.</div><div = class=3D"">2017-04-11 09:10:25,036-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.se/" = class=3D"">org.ovirt.engine.core.bll.Se</a><wbr = class=3D"">tNonOperationalVdsCommand] (default task-29) [2ffa5981] = Running command: SetNonOperationalVdsCommand internal: true. Entities = affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0<wbr class=3D"">dafe34 = Type: VDS</div><div class=3D"">2017-04-11 09:10:25,039-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] = START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParamete<wbr = class=3D"">rs:{runAsync=3D'true', hostId=3D'87a5f691-ea5c-4692-a4a<wbr = class=3D"">b-52dad0dafe34', status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 706ffc3b</div></div><div class=3D""><br class=3D""></div><div = class=3D"">.....</div><div class=3D""><br class=3D""></div><div = class=3D""><div class=3D"">2017-04-11 09:10:25,366-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.vdsbroker.SpmStopVDSComman<wbr class=3D"">d] (default = task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: = 62f82cfb</div><div class=3D"">2017-04-11 09:10:25,366-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.reactors.stomp.impl.Message] (default task-29) [2ffa5981] = UNSUBSCRIBE</div><div class=3D"">id:4cd59c67-1b57-4456-9a16-52a<wbr = class=3D"">72ac6f689</div><div class=3D""><br class=3D""></div><div = class=3D"">\00</div><div class=3D"">2017-04-11 09:10:25,367-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.reactors.stomp.StompCommonCli<wbr class=3D"">ent] (default = task-29) [2ffa5981] Message sent: UNSUBSCRIBE</div></div><div = class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote"><div class=3D""><div = class=3D"gmail-m_5006786018485841575gmail-h5">On Tue, Apr 11, 2017 at = 4:49 PM, Sandro Bonazzola <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:sbonazzo@redhat.com" target=3D"_blank" = class=3D"">sbonazzo@redhat.com</a>></span> wrote:<br = class=3D""></div></div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div class=3D""><div = class=3D"gmail-m_5006786018485841575gmail-h5"><div dir=3D"ltr" = class=3D"">Link to Job: <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 303" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/t<wbr = class=3D"">est-repo_ovirt_experimental_ma<wbr class=3D"">ster/6303</a><pre= = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-console-output" = style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-= bottom:0px;color:rgb(51,51,51);font-size:14px"><span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D""><br = class=3D""></b></span></pre><pre = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-console-output" = style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-= bottom:0px;color:rgb(51,51,51);font-size:14px"><span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">OST is failing in the last 50 = builds</b></span></pre><pre = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-console-output" = style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-= bottom:0px;color:rgb(51,51,51);font-size:14px"><span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D""><br = class=3D""></b></span></pre><pre = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-console-output" = style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-= bottom:0px;color:rgb(51,51,51);font-size:14px"><br class=3D""></pre><pre = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-console-output" = style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-= bottom:0px;color:rgb(51,51,51);font-size:14px"><span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(0,205,205)" class=3D""> # = add_secondary_storage_domains: </span> <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error = while running thread <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] Traceback (most recent call last): <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] queue.put({'return': func()}) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 427, in add_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, = nfs_version=3D'v4_2') <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, = sd_type, nfs_version) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] host=3D_random_host_from_dc_4(ap<wbr = class=3D"">i, DC_NAME), <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr = class=3D"">pi, dc_name)) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> = </span>[basic_suit_el7] raise RuntimeError('Could not find hosts = that are up in DC %s' % dc_name) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up = in DC test-dc</span> <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error = while running thread <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] Traceback (most recent call last): <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] queue.put({'return': func()}) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 583, in add_iso_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, = sd_format=3D'v1', sd_type=3D'iso', nfs_version=3D'v3') <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, = sd_type, nfs_version) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] host=3D_random_host_from_dc_4(ap<wbr = class=3D"">i, DC_NAME), <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr = class=3D"">pi, dc_name)) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] raise RuntimeError('Could not find hosts = that are up in DC %s' % dc_name) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> = </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up = in DC test-dc</span> <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error = while running thread <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] Traceback (most recent call last): <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] queue.put({'return': func()}) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 587, in add_templates_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, = SD_TEMPLATES_PATH, sd_format=3D'v1', sd_type=3D'export', = nfs_version=3D'v4_1') <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, = sd_type, nfs_version) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] host=3D_random_host_from_dc_4(ap<wbr = class=3D"">i, DC_NAME), <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr = class=3D"">pi, dc_name)) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] raise RuntimeError('Could not find hosts = that are up in DC %s' % dc_name) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up = in DC test-dc</span> <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error = while running thread <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> = </span>[basic_suit_el7] Traceback (most recent call last): <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] queue.put({'return': func()}) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 436, in add_second_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr = class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, = sd_type, nfs_version) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] host=3D_random_host_from_dc_4(ap<wbr = class=3D"">i, DC_NAME), <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr = class=3D"">pi, dc_name)) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] raise RuntimeError('Could not find hosts = that are up in DC %s' % dc_name) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up = in DC test-dc</span> <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] <span = style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error = while running thread <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] Traceback (most recent call last): <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] queue.put({'return': func()}) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 803, in import_template_from_glance <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] generic_import_from_glance(api<wbr = class=3D"">, image_name=3DCIRROS_IMAGE_NAME, = image_ext=3D'_glance_template', as_template=3DTrue) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", = line 637, in generic_import_from_glance <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] target_image.import_image(impo<wbr = class=3D"">rt_action) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/br<wbr class=3D"">okers.py", line = 26017, in import_image <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] headers=3D{"Correlation-Id":corr<wbr = class=3D"">elation_id} <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/pr<wbr class=3D"">oxy.py", line = 122, in request <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> = </span>[basic_suit_el7] persistent_auth=3Dself.__persist<wbr = class=3D"">ent_auth <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/co<wbr = class=3D"">nnectionspool.py", line 79, in do_request <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] persistent_auth) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/co<wbr = class=3D"">nnectionspool.py", line 162, in __do_request <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] raise errors.RequestError(response_c<wbr = class=3D"">ode, response_reason, response_body) <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] RequestError:=20 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] status: 400 <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] reason: Bad Request <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] detail: Cannot import Virtual Disk: Storage = Domain cannot be accessed. <span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b = style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> = </span>[basic_suit_el7] -Please check that at least one Host is = operational and Data Center state is up.</span></pre><span = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735HOEnZb"><fo= nt color=3D"#888888" class=3D""><div class=3D""><br class=3D""></div>-- = <br class=3D""><div = class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228= 0271388056gmail_signature"><div dir=3D"ltr" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div = dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div = style=3D"font-family: overpass, sans-serif; font-weight: bold; margin: = 0px; padding: 0px; font-size: 14px; text-transform: uppercase;" = class=3D""><span class=3D"">SANDRO</span> <span = class=3D"">BONAZZOLA</span></div><p style=3D"font-family: overpass, = sans-serif; font-size: 10px; margin: 0px 0px 4px; text-transform: = uppercase;" class=3D""><span class=3D"">ASSOCIATE MANAGER, SOFTWARE = ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><div = style=3D"font-family: overpass, sans-serif; margin: 0px; font-size: = 10px; color: rgb(153, 153, 153);" class=3D""><a = href=3D"https://www.redhat.com/" style=3D"color:rgb(0,136,206);margin:0px"= target=3D"_blank" class=3D"">Red Hat <span = class=3D"">EMEA</span></a></div><table style=3D"font-family: overpass, = sans-serif; font-size: inherit;" border=3D"0" class=3D""><tbody = class=3D""><tr class=3D""><td width=3D"100px" class=3D""><a = href=3D"https://red.ht/sig" target=3D"_blank" class=3D""><img = src=3D"https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red= -hat-black.png" width=3D"90" height=3D"auto" class=3D""></a></td><td = style=3D"font-size:10px" class=3D""><div class=3D""><a = href=3D"https://redhat.com/trusted" = style=3D"color:rgb(204,0,0);font-weight:bold" target=3D"_blank" = class=3D"">TRIED. TESTED. = TRUSTED.</a></div></td></tr></tbody></table></div></div></div></div></div>= </div></div> </font></span></div> <br class=3D""></div></div>______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br = class=3D""></div> <br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br = class=3D""><br clear=3D"all" class=3D""><br class=3D""></div></div><span = class=3D"gmail-HOEnZb"><font color=3D"#888888" class=3D"">-- <br = class=3D""><div class=3D"gmail-m_5006786018485841575gmail_signature">Barak= Korren<br class=3D""><a href=3D"mailto:bkorren@redhat.com" = target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">RHCE, = RHCi, RHV-DevOps Team<br class=3D""><a = href=3D"https://ifireball.wordpress.com/" target=3D"_blank" = class=3D"">https://ifireball.wordpress.<wbr class=3D"">com/</a></div> </font></span></div></div></div></div> <br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br= class=3D""></div></div> _______________________________________________<br class=3D"">Devel = mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote=
</div><br class=3D""></body></html>=
--Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032--
I am working on providing correlation id between the engine and vdsm. The engine patch which bumps jsonrpc version (already merged) is still open. It contains update [1] for the engine. I thought that once we base our artifacts using version (tag) we are not using head. I need to have a way to introduce new features. [1] https://gerrit.ovirt.org/#/c/75333/2/backend/manager/dependencies/common/src... On Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul@redhat.com> wrote:
On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com> wrote:
This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html
That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307
That's an entirely different game: *2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils* 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
this one started in http://jenkins.ovirt.org/job/test-repo_ovirt_ experimental_master/6307/
there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what seems to be mismatched Piotr/Sandro, are all artifacts published correctly?
Thanks, michal
org/apache/commons/lang/StringUtils at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_121] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue( HostSetupNetworkPoller.java:56) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll( HostSetupNetworkPoller.java:41) [bll.jar:]
... Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/ StringUtils at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message. withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient. sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$ timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:]
On 11 April 2017 at 18:12, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:
2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.db broker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core. vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b
.....
2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689
\00 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client .reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Link to Job: http://jenkins.ovirt.org/job/t est-repo_ovirt_experimental_master/6303
*OST is failing in the last 50 builds*
*00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
-- SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul@redhat.com> wrote:
On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com> wrote:
This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html
That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307
That's an entirely different game: *2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils* 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
this one started in http://jenkins.ovirt.org/job/test-repo_ovirt_ experimental_master/6307/
there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what seems to be mismatched Piotr/Sandro, are all artifacts published correctly?
Please define published correctly. Since the publisher job is not used anymore to publish artifacts I've no list of what gets published. I can only check repository content: cockpit-ovirt-dashboard-0:0.11.0-0.0.2.el7.centos.noarch cockpit-ovirt-uiplugin-0:0.11.0-0.0.2.el7.centos.noarch imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch ioprocess-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64 ioprocess-debuginfo-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64 java-ovirt-engine-sdk4-0:4.1.0-0.1.20170331gitdcf3f4d.el7.centos.noarch libcacard-devel-ev-10:2.3.0-31.el7_2.21.1.x86_64 libcacard-ev-10:2.3.0-31.el7_2.21.1.x86_64 libcacard-tools-ev-10:2.3.0-31.el7_2.21.1.x86_64 mom-0:0.5.9-0.0.master.el7.centos.noarch nodectl-0:4.1-999.20160902.0.el7.noarch otopi-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch otopi-debug-plugins-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch otopi-devtools-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch otopi-java-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch otopi-javadoc-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-appliance-0:4.2-20170408.1.el7.centos.noarch ovirt-engine-backend-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-cli-0:3.6.9.3-0.1.20170116.gitad0561a.el7.centos.noarch ovirt-engine-dashboard-0:1.2.0-0.5.20170331git8568fc5.el7.centos.noarch ovirt-engine-dbscripts-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-dwh-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch ovirt-engine-dwh-setup-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch ovirt-engine-extension-aaa-jdbc-0:1.1.5-0.0.master.git2a03d42.el7.centos.noarch ovirt-engine-extension-aaa-ldap-0:1.3.2-0.0.master.gitabe7725.el7.centos.noarch ovirt-engine-extension-aaa-ldap-setup-0:1.3.2-0.0.master.gitabe7725.el7.centos.noarch ovirt-engine-extension-aaa-misc-0:1.0.2-0.0.master.git9430be4.el7.centos.noarch ovirt-engine-extension-logger-log4j-0:1.0.3-0.0.master.el7.centos.noarch ovirt-engine-extensions-api-impl-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-extensions-api-impl-javadoc-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-hosts-ansible-inventory-0:4.2.0-0.0.master.20170302072348.git4674298.el7.centos.noarch ovirt-engine-lib-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-metrics-0:1.0.3-0.1.master.20170406102316.el7.centos.noarch ovirt-engine-nodejs-0:6.9.4-5.el7.centos.x86_64 ovirt-engine-nodejs-modules-0:1.0.4-5.el7.centos.x86_64 ovirt-engine-nodejs-modules-0:1.0.7-1.el7.centos.noarch ovirt-engine-phantomjs-0:2.1.1-1.el7.centos.x86_64 ovirt-engine-phantomjs-debuginfo-0:2.1.1-1.el7.centos.x86_64 ovirt-engine-restapi-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-sdk-java-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.noarch ovirt-engine-sdk-java-javadoc-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.noarch ovirt-engine-sdk-python-0:3.6.9.2-0.1.20161204.gite99bbd1.el7.centos.noarch ovirt-engine-setup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-base-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-plugin-dockerc-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-plugin-live-0:4.2.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-common-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-setup-plugin-websocket-proxy-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-tools-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-tools-backup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-userportal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-userportal-debuginfo-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-webadmin-portal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-webadmin-portal-debuginfo-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-websocket-proxy-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch ovirt-engine-wildfly-0:10.1.0-1.el7.x86_64 ovirt-engine-wildfly-overlay-0:10.0.0-1.el7.noarch ovirt-engine-yarn-0:0.19.1-2.el7.centos.x86_64 ovirt-engine-yarn-0:0.19.1-3.el7.centos.noarch ovirt-guest-agent-common-0:1.0.13-1.el7.centos.noarch ovirt-guest-agent-debuginfo-0:1.0.13-1.el7.centos.x86_64 ovirt-guest-agent-gdm-plugin-0:1.0.13-1.el7.centos.noarch ovirt-guest-agent-pam-module-0:1.0.13-1.el7.centos.x86_64 ovirt-host-deploy-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch ovirt-host-deploy-java-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch ovirt-host-deploy-javadoc-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch ovirt-hosted-engine-ha-0:2.2.0-0.0.master.20170404202238.20170404202234.git68258f1.el7.centos.noarch ovirt-hosted-engine-setup-0:2.2.0-0.0.master.20170330072130.git8a05e04.el7.centos.noarch ovirt-image-uploader-0:4.1.0-0.0.master.20160830110830.gitad0be34.el7.centos.noarch ovirt-imageio-common-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch ovirt-imageio-daemon-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch ovirt-imageio-proxy-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch ovirt-imageio-proxy-setup-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch ovirt-iso-uploader-0:4.1.0-0.0.master.20160909154152.git14502bd.el7.centos.noarch ovirt-jboss-modules-maven-plugin-0:2.0.2-0.1.20160715.gitec01507.el7.centos.noarch ovirt-jboss-modules-maven-plugin-javadoc-0:2.0.2-0.1.20160715.gitec01507.el7.centos.noarch ovirt-js-dependencies-0:0.0.3-2.el7.centos.x86_64 ovirt-js-dependencies-0:0.0.3-4.el7.centos.noarch ovirt-live-artwork-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch ovirt-live-artwork-gnome-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch ovirt-live-artwork-mate-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch ovirt-log-collector-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.el7.centos.noarch ovirt-log-collector-analyzer-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.el7.centos.noarch ovirt-node-ng-docs-0:4.1.0-0.20170408.0.el7.noarch ovirt-node-ng-image-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el7.centos.noarch ovirt-node-ng-image-update-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el7.centos.noarch ovirt-node-ng-image-update-placeholder-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch ovirt-node-ng-nodectl-0:4.1.0-0.20170408.0.el7.noarch ovirt-node-ng-tools-0:4.1.0-0.20170408.0.el7.noarch ovirt-optimizer-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-optimizer-dependencies-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-optimizer-jboss-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-optimizer-ui-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-provider-ovn-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noarch ovirt-provider-ovn-driver-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noarch ovirt-release-host-node-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch ovirt-release-master-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch ovirt-release-master-experimental-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch ovirt-setup-lib-0:1.1.0-1.20161219151842.git402fec4.el7.centos.noarch ovirt-vmconsole-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch ovirt-vmconsole-host-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch ovirt-vmconsole-proxy-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch ovirt-web-ui-0:0.1.2-1.el7.centos.x86_64 python-imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch python-ioprocess-0:0.17.0-1.201702282324.gite82786d.el7.centos.noarch python-ovirt-engine-sdk4-0:4.2.0-1.a0.20170331gitd6b662b.el7.centos.x86_64 python-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.a0.20170331gitd6b662b.el7.centos.x86_64 qemu-img-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-common-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-ev-debuginfo-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-tools-ev-10:2.6.0-28.el7_3.6.1.x86_64 rubygem-ovirt-engine-sdk4-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64 rubygem-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64 rubygem-ovirt-engine-sdk4-doc-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64 unboundid-ldapsdk-0:3.2.0-1.el7.noarch unboundid-ldapsdk-javadoc-0:3.2.0-1.el7.noarch vdsm-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-api-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-client-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-gluster-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-allocate_net-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-checkimages-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-checkips-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-diskunmap-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-ethtool-options-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-extnet-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-extra-ipv4-addrs-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-fakesriov-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-fakevmstats-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-faqemu-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-fcoe-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-fileinject-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-floppy-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-hostusb-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-httpsisoboot-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-hugepages-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-ipv6-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-isolatedprivatevlan-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-localdisk-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-macbind-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-macspoof-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-nestedvt-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-noipspoof-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-numa-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-openstacknet-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-pincpu-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-promisc-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-qemucmdline-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-qos-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-scratchpad-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-smbios-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-spiceoptions-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vhostmd-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmdisk-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmfex-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmfex-dev-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-infra-0:4.18.999-212.git2556c05.el7.centos.noarch vdsm-jsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-jsonrpc-java-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.noarch vdsm-jsonrpc-java-javadoc-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.noarch vdsm-python-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-tests-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-xmlrpc-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-yajsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch
Thanks, michal
org/apache/commons/lang/StringUtils at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_121] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue( HostSetupNetworkPoller.java:56) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll( HostSetupNetworkPoller.java:41) [bll.jar:]
... Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/ StringUtils at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message. withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient. sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$ timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:]
On 11 April 2017 at 18:12, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:
2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto rage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.db broker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core. vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b
.....
2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689
\00 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client .reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Link to Job: http://jenkins.ovirt.org/job/t est-repo_ovirt_experimental_master/6303
*OST is failing in the last 50 builds*
*00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.091* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call last):*00:18:22.092* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': func()})*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')*00:18:22.092* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.092* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call last):*00:18:22.094* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': func()})*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.094* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call last):*00:18:22.096* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': func()})*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain*00:18:22.096* [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),*00:18:22.096* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call last):*00:18:22.097* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': func()})*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)*00:18:22.098* [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image*00:18:22.098* [basic_suit_el7] headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request*00:18:22.098* [basic_suit_el7] persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request*00:18:22.099* [basic_suit_el7] persistent_auth)*00:18:22.099* [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
-- SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
--Apple-Mail=_9B8B03C8-0839-4F3A-94B7-1252B9DCF326 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 12 Apr 2017, at 09:17, Sandro Bonazzola <sbonazzo@redhat.com> = wrote: =20 =20 =20 On Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek = <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: =20
On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul@redhat.com = <mailto:ykaul@redhat.com>> wrote: =20 =20 =20 On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com = <mailto:bkorren@redhat.com>> wrote: This seems to have been the Secodary SD issue I reported on Sunday: http://lists.ovirt.org/pipermail/devel/2017-April/030139.html = <http://lists.ovirt.org/pipermail/devel/2017-April/030139.html> =20 That was up until build #6307 where AddHost started failing: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307> =20 That's an entirely different game: 2017-04-11 11:24:36,783-04 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: = java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils 2017-04-11 11:24:36,783-04 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] = (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: = java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:=20=
=20 this one started in = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/> =20 there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is = what seems to be mismatched Piotr/Sandro, are all artifacts published correctly? =20 Please define published correctly. Since the publisher job is not used anymore to publish artifacts I've = no list of what gets published.
Well, Im not sure who=E2=80=99s handling that exactly, but = vdsm-jsonrpc-java is getting into OST runs despite the OST failing since = Saturday. That should not happen btw, in the meantime, me and Tomas are working on narrowing down the = original failure. Thanks, michal=20
I can only check repository content: =20 cockpit-ovirt-dashboard-0:0.11.0-0.0.2.el7.centos.noarch cockpit-ovirt-uiplugin-0:0.11.0-0.0.2.el7.centos.noarch imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch ioprocess-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64 = ioprocess-debuginfo-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64 = java-ovirt-engine-sdk4-0:4.1.0-0.1.20170331gitdcf3f4d.el7.centos.noarch libcacard-devel-ev-10:2.3.0-31.el7_2.21.1.x86_64 libcacard-ev-10:2.3.0-31.el7_2.21.1.x86_64 libcacard-tools-ev-10:2.3.0-31.el7_2.21.1.x86_64 mom-0:0.5.9-0.0.master.el7.centos.noarch nodectl-0:4.1-999.20160902.0.el7.noarch otopi-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch = otopi-debug-plugins-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.cento= s.noarch = otopi-devtools-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noa= rch = otopi-java-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch = otopi-javadoc-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noar= ch = ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarc= h ovirt-engine-appliance-0:4.2-20170408.1.el7.centos.noarch = ovirt-engine-backend-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.cent= os.noarch ovirt-engine-cli-0:3.6.9.3-0.1.20170116.gitad0561a.el7.centos.noarch = ovirt-engine-dashboard-0:1.2.0-0.5.20170331git8568fc5.el7.centos.noarch = ovirt-engine-dbscripts-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.ce= ntos.noarch ovirt-engine-dwh-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch = ovirt-engine-dwh-setup-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch=
= ovirt-engine-extension-aaa-jdbc-0:1.1.5-0.0.master.git2a03d42.el7.centos.n= oarch = ovirt-engine-extension-aaa-ldap-0:1.3.2-0.0.master.gitabe7725.el7.centos.n= oarch = ovirt-engine-extension-aaa-ldap-setup-0:1.3.2-0.0.master.gitabe7725.el7.ce= ntos.noarch = ovirt-engine-extension-aaa-misc-0:1.0.2-0.0.master.git9430be4.el7.centos.n= oarch = ovirt-engine-extension-logger-log4j-0:1.0.3-0.0.master.el7.centos.noarch = ovirt-engine-extensions-api-impl-0:4.2.0-0.0.master.20170408115943.gitea6b= d62.el7.centos.noarch = ovirt-engine-extensions-api-impl-javadoc-0:4.2.0-0.0.master.20170408115943= .gitea6bd62.el7.centos.noarch = ovirt-engine-hosts-ansible-inventory-0:4.2.0-0.0.master.20170302072348.git= 4674298.el7.centos.noarch = ovirt-engine-lib-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.n= oarch = ovirt-engine-metrics-0:1.0.3-0.1.master.20170406102316.el7.centos.noarch ovirt-engine-nodejs-0:6.9.4-5.el7.centos.x86_64 ovirt-engine-nodejs-modules-0:1.0.4-5.el7.centos.x86_64 ovirt-engine-nodejs-modules-0:1.0.7-1.el7.centos.noarch ovirt-engine-phantomjs-0:2.1.1-1.el7.centos.x86_64 ovirt-engine-phantomjs-debuginfo-0:2.1.1-1.el7.centos.x86_64 = ovirt-engine-restapi-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.cent= os.noarch = ovirt-engine-sdk-java-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.noarch = ovirt-engine-sdk-java-javadoc-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.n= oarch = ovirt-engine-sdk-python-0:3.6.9.2-0.1.20161204.gite99bbd1.el7.centos.noarc= h = ovirt-engine-setup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos= .noarch = ovirt-engine-setup-base-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.c= entos.noarch = ovirt-engine-setup-plugin-dockerc-0:4.2.0-0.0.master.20170408115943.gitea6= bd62.el7.centos.noarch = ovirt-engine-setup-plugin-live-0:4.2.0-0.0.master.20170331075633.git8b425a= 3.el7.centos.noarch = ovirt-engine-setup-plugin-ovirt-engine-0:4.2.0-0.0.master.20170408115943.g= itea6bd62.el7.centos.noarch = ovirt-engine-setup-plugin-ovirt-engine-common-0:4.2.0-0.0.master.201704081= 15943.gitea6bd62.el7.centos.noarch = ovirt-engine-setup-plugin-vmconsole-proxy-helper-0:4.2.0-0.0.master.201704= 08115943.gitea6bd62.el7.centos.noarch = ovirt-engine-setup-plugin-websocket-proxy-0:4.2.0-0.0.master.2017040811594= 3.gitea6bd62.el7.centos.noarch = ovirt-engine-tools-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos= .noarch = ovirt-engine-tools-backup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7= .centos.noarch = ovirt-engine-userportal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.c= entos.noarch = ovirt-engine-userportal-debuginfo-0:4.2.0-0.0.master.20170408115943.gitea6= bd62.el7.centos.noarch = ovirt-engine-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408115943.gite= a6bd62.el7.centos.noarch = ovirt-engine-webadmin-portal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.= el7.centos.noarch = ovirt-engine-webadmin-portal-debuginfo-0:4.2.0-0.0.master.20170408115943.g= itea6bd62.el7.centos.noarch = ovirt-engine-websocket-proxy-0:4.2.0-0.0.master.20170408115943.gitea6bd62.= el7.centos.noarch ovirt-engine-wildfly-0:10.1.0-1.el7.x86_64 ovirt-engine-wildfly-overlay-0:10.0.0-1.el7.noarch ovirt-engine-yarn-0:0.19.1-2.el7.centos.x86_64 ovirt-engine-yarn-0:0.19.1-3.el7.centos.noarch ovirt-guest-agent-common-0:1.0.13-1.el7.centos.noarch ovirt-guest-agent-debuginfo-0:1.0.13-1.el7.centos.x86_64 ovirt-guest-agent-gdm-plugin-0:1.0.13-1.el7.centos.noarch ovirt-guest-agent-pam-module-0:1.0.13-1.el7.centos.x86_64 = ovirt-host-deploy-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.= noarch = ovirt-host-deploy-java-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.ce= ntos.noarch = ovirt-host-deploy-javadoc-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7= .centos.noarch = ovirt-hosted-engine-ha-0:2.2.0-0.0.master.20170404202238.20170404202234.gi= t68258f1.el7.centos.noarch = ovirt-hosted-engine-setup-0:2.2.0-0.0.master.20170330072130.git8a05e04.el7= .centos.noarch = ovirt-image-uploader-0:4.1.0-0.0.master.20160830110830.gitad0be34.el7.cent= os.noarch = ovirt-imageio-common-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch = ovirt-imageio-daemon-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch = ovirt-imageio-proxy-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch = ovirt-imageio-proxy-setup-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noa= rch = ovirt-iso-uploader-0:4.1.0-0.0.master.20160909154152.git14502bd.el7.centos= .noarch = ovirt-jboss-modules-maven-plugin-0:2.0.2-0.1.20160715.gitec01507.el7.cento= s.noarch = ovirt-jboss-modules-maven-plugin-javadoc-0:2.0.2-0.1.20160715.gitec01507.e= l7.centos.noarch ovirt-js-dependencies-0:0.0.3-2.el7.centos.x86_64 ovirt-js-dependencies-0:0.0.3-4.el7.centos.noarch = ovirt-live-artwork-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos= .noarch = ovirt-live-artwork-gnome-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.= centos.noarch = ovirt-live-artwork-mate-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.c= entos.noarch = ovirt-log-collector-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.el7.cento= s.noarch = ovirt-log-collector-analyzer-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.= el7.centos.noarch ovirt-node-ng-docs-0:4.1.0-0.20170408.0.el7.noarch = ovirt-node-ng-image-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el7.cento= s.noarch = ovirt-node-ng-image-update-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el= 7.centos.noarch = ovirt-node-ng-image-update-placeholder-0:4.2.0-0.3.master.20170408002716.g= it2f43a3c.el7.centos.noarch ovirt-node-ng-nodectl-0:4.1.0-0.20170408.0.el7.noarch ovirt-node-ng-tools-0:4.1.0-0.20170408.0.el7.noarch ovirt-optimizer-0:0.15-0.176.201702021258.el7.centos.noarch = ovirt-optimizer-dependencies-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-optimizer-jboss-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-optimizer-ui-0:0.15-0.176.201702021258.el7.centos.noarch ovirt-provider-ovn-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noarch = ovirt-provider-ovn-driver-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noa= rch = ovirt-release-host-node-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.c= entos.noarch = ovirt-release-master-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.cent= os.noarch = ovirt-release-master-experimental-0:4.2.0-0.3.master.20170408002716.git2f4= 3a3c.el7.centos.noarch ovirt-setup-lib-0:1.1.0-1.20161219151842.git402fec4.el7.centos.noarch = ovirt-vmconsole-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.no= arch = ovirt-vmconsole-host-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.cent= os.noarch = ovirt-vmconsole-proxy-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.cen= tos.noarch ovirt-web-ui-0:0.1.2-1.el7.centos.x86_64 python-imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch python-ioprocess-0:0.17.0-1.201702282324.gite82786d.el7.centos.noarch = python-ovirt-engine-sdk4-0:4.2.0-1.a0.20170331gitd6b662b.el7.centos.x86_64=
org/apache/commons/lang/StringUtils at java.util.concurrent.FutureTask.report(FutureTask.java:122) = [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) = [rt.jar:1.8.0_121] at = org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo= mmand.java:69) [vdsbroker.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos= tSetupNetworkPoller.java:56) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet= upNetworkPoller.java:41) [bll.jar:] =20 ... Caused by: java.lang.NoClassDefFoundError: = org/apache/commons/lang/StringUtils at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationI= d(Message.java:75) [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SS= LStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) = [vdsm-jsonrpc-java-client.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91= ) [vdsbroker.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundP=
at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:484) [vdsbroker.jar:] at = org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca= ll(JsonRpcVdsServer.java:473) [vdsbroker.jar:] =20 =20 =20 =20 =20 On 11 April 2017 at 18:12, Piotr Kliczewski = <piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote: I looks like SD activate failed on the engine side and the spm = connection was gently closed by the engine: =20 2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st = <http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCo= mmand] (default task-29) [2b66a42b] ActivateStorage Domain. After = Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017 2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] = (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds = 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to = non-operational: null 2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] = (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds = 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to = non-operational: null 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se = <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] = (default task-29) [2bc83b1c] Running command: = SetNonOperationalVdsCommand internal: true. Entities affected : ID: = e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS 2017-04-11 09:10:24,909-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName =3D = lago-basic-suite-master-host0, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'e1b611f2-67fb-4acb-8261-07133134e0e7', = status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 5c45239 2017-04-11 09:10:24,919-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call = string is [{call getvmsrunningonvds(?)}] 2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure = [GetVmsRunningOnVds] compiled 2017-04-11 09:10:24,985-04 WARN = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (default task-29) [2bc83b1c] EVENT_ID: = VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: = d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: = -1, Message: Host lago-basic-suite-master-host0 cannot access the = Storage Domain(s) iscsi attached to the Data Center test-dc. Setting = Host state to Non-Operational. 2017-04-11 09:10:24,985-04 DEBUG = [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] = (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org = <http://default.org/>.ovirt.engine.core.vdsbroker.VdsManager.onTimer#-9223= 372036854775741 as there is no unfired trigger. 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se = <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] = (default task-29) [2ffa5981] Running command: = SetNonOperationalVdsCommand internal: true. Entities affected : ID: = 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS 2017-04-11 09:10:25,039-04 INFO = [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default = task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName =3D = lago-basic-suite-master-host1, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'87a5f691-ea5c-4692-a4ab-52dad0dafe34', = status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 706ffc3b =20 ..... =20 2017-04-11 09:10:25,366-04 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default = task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb 2017-04-11 09:10:25,366-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default = task-29) [2ffa5981] UNSUBSCRIBE id:4cd59c67-1b57-4456-9a16-52a72ac6f689 =20 \00 2017-04-11 09:10:25,367-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] = (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE =20 =20 =20 On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola = <sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>> wrote: Link to Job: = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303 = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303> =20 OST is failing in the last 50 builds =20 =20 00:18:22.090 [basic_suit_el7] # add_secondary_storage_domains:=20 00:18:22.090 [basic_suit_el7] Error while running thread 00:18:22.090 [basic_suit_el7] Traceback (most recent call last): 00:18:22.090 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.090 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.090 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in = add_nfs_storage_domain 00:18:22.090 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, = SD_NFS_PATH, nfs_version=3D'v4_2') 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.091 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.091 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.091 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.091 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.091 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.092 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.092 [basic_suit_el7] Error while running thread 00:18:22.092 [basic_suit_el7] Traceback (most recent call last): 00:18:22.092 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.092 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.092 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in = add_iso_storage_domain 00:18:22.092 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, = SD_ISO_PATH, sd_format=3D'v1', sd_type=3D'iso', nfs_version=3D'v3') 00:18:22.092 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.092 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.093 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.093 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.093 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.093 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.093 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.094 [basic_suit_el7] Error while running thread 00:18:22.094 [basic_suit_el7] Traceback (most recent call last): 00:18:22.094 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.094 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in = add_templates_storage_domain 00:18:22.094 [basic_suit_el7] = add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, = SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format=3D'v1', = sd_type=3D'export', nfs_version=3D'v4_1') 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.094 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.094 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.095 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.095 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.095 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.095 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.095 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.095 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.095 [basic_suit_el7] Error while running thread 00:18:22.095 [basic_suit_el7] Traceback (most recent call last): 00:18:22.096 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.096 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in = add_second_nfs_storage_domain 00:18:22.096 [basic_suit_el7] SD_NFS_HOST_NAME, = SD_SECOND_NFS_PATH) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in = add_generic_nfs_storage_domain 00:18:22.096 [basic_suit_el7] = add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, = mount_path, sd_format, sd_type, nfs_version) 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in = add_generic_nfs_storage_domain_4 00:18:22.096 [basic_suit_el7] host=3D_random_host_from_dc_4(api, = DC_NAME), 00:18:22.096 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in = _random_host_from_dc_4 00:18:22.097 [basic_suit_el7] return = random.choice(_hosts_in_dc_4(api, dc_name)) 00:18:22.097 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in = _hosts_in_dc_4 00:18:22.097 [basic_suit_el7] raise RuntimeError('Could not find = hosts that are up in DC %s' % dc_name) 00:18:22.097 [basic_suit_el7] RuntimeError: Could not find hosts that = are up in DC test-dc 00:18:22.097 [basic_suit_el7] Error while running thread 00:18:22.097 [basic_suit_el7] Traceback (most recent call last): 00:18:22.097 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in = _ret_via_queue 00:18:22.098 [basic_suit_el7] queue.put({'return': func()}) 00:18:22.098 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in = import_template_from_glance 00:18:22.098 [basic_suit_el7] generic_import_from_glance(api, = image_name=3DCIRROS_IMAGE_NAME, image_ext=3D'_glance_template', = as_template=3DTrue) 00:18:22.098 [basic_suit_el7] File = "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in = generic_import_from_glance 00:18:22.098 [basic_suit_el7] = target_image.import_image(import_action) 00:18:22.098 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", =
00:18:22.098 [basic_suit_el7] =
00:18:22.098 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", =
00:18:22.098 [basic_suit_el7] =
00:18:22.099 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
00:18:22.099 [basic_suit_el7] persistent_auth) 00:18:22.099 [basic_suit_el7] File = "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
= python-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.a0.20170331gitd6b662b.el7.cen= tos.x86_64 qemu-img-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-common-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-ev-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-ev-debuginfo-10:2.6.0-28.el7_3.6.1.x86_64 qemu-kvm-tools-ev-10:2.6.0-28.el7_3.6.1.x86_64 = rubygem-ovirt-engine-sdk4-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x= 86_64 = rubygem-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.alpha0.20170407gite402de7.el= 7.centos.x86_64 = rubygem-ovirt-engine-sdk4-doc-0:4.2.0-1.alpha0.20170407gite402de7.el7.cent= os.x86_64 unboundid-ldapsdk-0:3.2.0-1.el7.noarch unboundid-ldapsdk-javadoc-0:3.2.0-1.el7.noarch vdsm-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-api-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-client-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-gluster-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-allocate_net-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-checkimages-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-checkips-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-diskunmap-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-ethtool-options-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-extnet-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-extra-ipv4-addrs-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-fakesriov-0:4.20.0-612.git378d837.el7.centos.x86_64 vdsm-hook-fakevmstats-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-faqemu-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-fcoe-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-fileinject-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-floppy-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-hostusb-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-httpsisoboot-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-hugepages-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-ipv6-0:4.20.0-612.git378d837.el7.centos.noarch = vdsm-hook-isolatedprivatevlan-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-localdisk-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-macbind-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-macspoof-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-nestedvt-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-noipspoof-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-numa-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-openstacknet-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-pincpu-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-promisc-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-qemucmdline-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-qos-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-scratchpad-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-smbios-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-spiceoptions-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vhostmd-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmdisk-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmfex-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-hook-vmfex-dev-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-infra-0:4.18.999-212.git2556c05.el7.centos.noarch vdsm-jsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch = vdsm-jsonrpc-java-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.noarch = vdsm-jsonrpc-java-javadoc-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.n= oarch vdsm-python-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-tests-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-xmlrpc-0:4.20.0-612.git378d837.el7.centos.noarch vdsm-yajsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch =20 =20 =20 =20 =20 =20 Thanks, michal =20 oll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] line 26017, in import_image headers=3D{"Correlation-Id":correlation_id} line 122, in request persistent_auth=3Dself.__persistent_auth py", line 79, in do_request py", line 162, in __do_request
00:18:22.099 [basic_suit_el7] raise = errors.RequestError(response_code, response_reason, response_body) 00:18:22.099 [basic_suit_el7] RequestError:=20 00:18:22.099 [basic_suit_el7] status: 400 00:18:22.099 [basic_suit_el7] reason: Bad Request 00:18:22.099 [basic_suit_el7] detail: Cannot import Virtual Disk: = Storage Domain cannot be accessed. 00:18:22.099 [basic_suit_el7] -Please check that at least one Host is = operational and Data Center state is up. =20 --=20 SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat=C2=A0EMEA <https://www.redhat.com/> <https://red.ht/sig>=09 TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 =20 --=20 Barak Korren bkorren@redhat.com <mailto:bkorren@redhat.com> RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/> _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 =20 --=20 SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat=C2=A0EMEA <https://www.redhat.com/> <https://red.ht/sig>=09 TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
</blockquote><div class=3D""><br class=3D""></div></span>this one = started in <a =
<div = class=3D"">vdsm-hook-extra-ipv4-addrs-0:4.20.0-612.git378d837.el7.centos.x= 86_64</div><div = class=3D"">vdsm-hook-fakesriov-0:4.20.0-612.git378d837.el7.centos.x86_64</=
<div = class=3D"">vdsm-hook-fcoe-0:4.20.0-612.git378d837.el7.centos.noarch</div><=
<div = class=3D"">vdsm-hook-hostusb-0:4.20.0-612.git378d837.el7.centos.noarch</di= v><div = class=3D"">vdsm-hook-httpsisoboot-0:4.20.0-612.git378d837.el7.centos.noarc= h</div><div = class=3D"">vdsm-hook-hugepages-0:4.20.0-612.git378d837.el7.centos.noarch</=
<div = class=3D"">vdsm-hook-promisc-0:4.20.0-612.git378d837.el7.centos.noarch</di= v><div = class=3D"">vdsm-hook-qemucmdline-0:4.20.0-612.git378d837.el7.centos.noarch= </div><div = class=3D"">vdsm-hook-qos-0:4.20.0-612.git378d837.el7.centos.noarch</div><d= iv = class=3D"">vdsm-hook-scratchpad-0:4.20.0-612.git378d837.el7.centos.noarch<= /div><div = class=3D"">vdsm-hook-smbios-0:4.20.0-612.git378d837.el7.centos.noarch</div= <div = class=3D"">vdsm-hook-spiceoptions-0:4.20.0-612.git378d837.el7.centos.noarc= h</div><div = class=3D"">vdsm-hook-vhostmd-0:4.20.0-612.git378d837.el7.centos.noarch</di= v><div = class=3D"">vdsm-hook-vmdisk-0:4.20.0-612.git378d837.el7.centos.noarch</div= <div = class=3D"">vdsm-hook-vmfex-0:4.20.0-612.git378d837.el7.centos.noarch</div>= <div = class=3D"">vdsm-hook-vmfex-dev-0:4.20.0-612.git378d837.el7.centos.noarch</=
--Apple-Mail=_9B8B03C8-0839-4F3A-94B7-1252B9DCF326 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2017, at 09:17, Sandro Bonazzola <<a = href=3D"mailto:sbonazzo@redhat.com" class=3D"">sbonazzo@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><br = class=3D"Apple-interchange-newline"><br style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><div class=3D"gmail_quote" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On = Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek<span = class=3D"Apple-converted-space"> </span><span dir=3D"ltr" = class=3D""><<a href=3D"mailto:michal.skrivanek@redhat.com" = target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = style=3D"word-wrap: break-word;" class=3D""><br class=3D""><div = class=3D""><span class=3D"gmail-"><blockquote type=3D"cite" = class=3D""><div class=3D"">On 12 Apr 2017, at 08:37, Yaniv Kaul <<a = href=3D"mailto:ykaul@redhat.com" target=3D"_blank" = class=3D"">ykaul@redhat.com</a>> wrote:</div><br = class=3D"gmail-m_-7272531751306879750Apple-interchange-newline"><div = class=3D""><div dir=3D"ltr" class=3D""><br class=3D""><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Wed, = Apr 12, 2017 at 9:24 AM, Barak Korren<span = class=3D"Apple-converted-space"> </span><span dir=3D"ltr" = class=3D""><<a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = dir=3D"ltr" class=3D""><div class=3D"">This seems to have been the = Secodary SD issue I reported on Sunday:<br class=3D""><a = href=3D"http://lists.ovirt.org/pipermail/devel/2017-April/030139.html" = target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr = class=3D"">il/devel/2017-April/030139.<wbr class=3D"">html</a><br = class=3D""><br class=3D""></div>That was up until build #6307 where = AddHost started failing:<br class=3D""><a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 307" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/t<wbr = class=3D"">est-repo_ovirt_experimental_ma<wbr = class=3D"">ster/6307</a></div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">That's an entirely different = game:</div><div class=3D""><b class=3D"">2017-04-11 11:24:36,783-04 = ERROR [org.ovirt.engine.core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr = class=3D"">PollVDSCommand] (org.ovirt.thread.pool-7-<wbr = class=3D"">thread-1) [3497aa05] Error: java.lang.<wbr = class=3D"">NoClassDefFoundError: org/apache/commons/lang/<wbr = class=3D"">StringUtils</b></div><div class=3D"">2017-04-11 = 11:24:36,783-04 ERROR [org.ovirt.engine.core.<wbr = class=3D"">vdsbroker.vdsbroker.<wbr class=3D"">PollVDSCommand] = (org.ovirt.thread.pool-7-<wbr class=3D"">thread-1) [3497aa05] Exception: = java.util.concurrent.<wbr class=3D"">ExecutionException: java.lang.<wbr = class=3D"">NoClassDefFoundError:<span = class=3D"Apple-converted-space"> </span></div></div></div></div></div= href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 307/" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/<wbr = class=3D"">job/test-repo_ovirt_<wbr = class=3D"">experimental_master/6307/</a></div><div class=3D""><br = class=3D""></div><div class=3D"">there are 2 patches linkded, one is in = vdsm-jsonrpc-java, which is what seems to be mismatched</div><div = class=3D"">Piotr/Sandro, are all artifacts published = correctly?</div></div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">Please define published = correctly.</div><div class=3D"">Since the publisher job is not used = anymore to publish artifacts I've no list of what gets = published.</div></div></div></blockquote><div><br class=3D""></div>Well, = Im not sure who=E2=80=99s handling that exactly, but vdsm-jsonrpc-java = is getting into OST runs despite the OST failing since Saturday. That = should not happen</div><div><br class=3D""></div><div>btw, in the = meantime, me and Tomas are working on narrowing down the original = failure.</div><div><br = class=3D""></div><div>Thanks,</div><div>michal </div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D"gmail_quote" style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=3D"">I = can only check repository content:</div><div class=3D""><br = class=3D""></div><div class=3D""><div = class=3D"">cockpit-ovirt-dashboard-0:0.11.0-0.0.2.el7.centos.noarch</div><= div = class=3D"">cockpit-ovirt-uiplugin-0:0.11.0-0.0.2.el7.centos.noarch</div><d= iv = class=3D"">imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch</= div><div = class=3D"">ioprocess-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64<= /div><div = class=3D"">ioprocess-debuginfo-0:0.17.0-1.201702282323.gite82786d.el7.cent= os.x86_64</div><div = class=3D"">java-ovirt-engine-sdk4-0:4.1.0-0.1.20170331gitdcf3f4d.el7.cento= s.noarch</div><div = class=3D"">libcacard-devel-ev-10:2.3.0-31.el7_2.21.1.x86_64</div><div = class=3D"">libcacard-ev-10:2.3.0-31.el7_2.21.1.x86_64</div><div = class=3D"">libcacard-tools-ev-10:2.3.0-31.el7_2.21.1.x86_64</div><div = class=3D"">mom-0:0.5.9-0.0.master.el7.centos.noarch</div><div = class=3D"">nodectl-0:4.1-999.20160902.0.el7.noarch</div><div = class=3D"">otopi-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.n= oarch</div><div = class=3D"">otopi-debug-plugins-0:1.7.0-0.0.master.20170405141623.git0166f1= 4.el7.centos.noarch</div><div = class=3D"">otopi-devtools-0:1.7.0-0.0.master.20170405141623.git0166f14.el7= .centos.noarch</div><div = class=3D"">otopi-java-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.cen= tos.noarch</div><div = class=3D"">otopi-javadoc-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.= centos.noarch</div><div = class=3D"">ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.c= entos.noarch</div><div = class=3D"">ovirt-engine-appliance-0:4.2-20170408.1.el7.centos.noarch</div>= <div = class=3D"">ovirt-engine-backend-0:4.2.0-0.0.master.20170408115943.gitea6bd= 62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-cli-0:3.6.9.3-0.1.20170116.gitad0561a.el7.centos.n= oarch</div><div = class=3D"">ovirt-engine-dashboard-0:1.2.0-0.5.20170331git8568fc5.el7.cento= s.noarch</div><div = class=3D"">ovirt-engine-dbscripts-0:4.2.0-0.0.master.20170408115943.gitea6= bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-dwh-0:4.2.0-0.0.master.20170403125608.el7.centos.n= oarch</div><div = class=3D"">ovirt-engine-dwh-setup-0:4.2.0-0.0.master.20170403125608.el7.ce= ntos.noarch</div><div = class=3D"">ovirt-engine-extension-aaa-jdbc-0:1.1.5-0.0.master.git2a03d42.e= l7.centos.noarch</div><div = class=3D"">ovirt-engine-extension-aaa-ldap-0:1.3.2-0.0.master.gitabe7725.e= l7.centos.noarch</div><div = class=3D"">ovirt-engine-extension-aaa-ldap-setup-0:1.3.2-0.0.master.gitabe= 7725.el7.centos.noarch</div><div = class=3D"">ovirt-engine-extension-aaa-misc-0:1.0.2-0.0.master.git9430be4.e= l7.centos.noarch</div><div = class=3D"">ovirt-engine-extension-logger-log4j-0:1.0.3-0.0.master.el7.cent= os.noarch</div><div = class=3D"">ovirt-engine-extensions-api-impl-0:4.2.0-0.0.master.20170408115= 943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-extensions-api-impl-javadoc-0:4.2.0-0.0.master.201= 70408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-hosts-ansible-inventory-0:4.2.0-0.0.master.2017030= 2072348.git4674298.el7.centos.noarch</div><div = class=3D"">ovirt-engine-lib-0:4.2.0-0.0.master.20170408115943.gitea6bd62.e= l7.centos.noarch</div><div = class=3D"">ovirt-engine-metrics-0:1.0.3-0.1.master.20170406102316.el7.cent= os.noarch</div><div = class=3D"">ovirt-engine-nodejs-0:6.9.4-5.el7.centos.x86_64</div><div = class=3D"">ovirt-engine-nodejs-modules-0:1.0.4-5.el7.centos.x86_64</div><d= iv = class=3D"">ovirt-engine-nodejs-modules-0:1.0.7-1.el7.centos.noarch</div><d= iv class=3D"">ovirt-engine-phantomjs-0:2.1.1-1.el7.centos.x86_64</div><div= = class=3D"">ovirt-engine-phantomjs-debuginfo-0:2.1.1-1.el7.centos.x86_64</d= iv><div = class=3D"">ovirt-engine-restapi-0:4.2.0-0.0.master.20170408115943.gitea6bd= 62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-sdk-java-0:3.6.10.0-1.20170215git67dcbcd.el7.cento= s.noarch</div><div = class=3D"">ovirt-engine-sdk-java-javadoc-0:3.6.10.0-1.20170215git67dcbcd.e= l7.centos.noarch</div><div = class=3D"">ovirt-engine-sdk-python-0:3.6.9.2-0.1.20161204.gite99bbd1.el7.c= entos.noarch</div><div = class=3D"">ovirt-engine-setup-0:4.2.0-0.0.master.20170408115943.gitea6bd62= .el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-base-0:4.2.0-0.0.master.20170408115943.gitea= 6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-dockerc-0:4.2.0-0.0.master.2017040811= 5943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-live-0:4.2.0-0.0.master.2017033107563= 3.git8b425a3.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-ovirt-engine-0:4.2.0-0.0.master.20170= 408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-ovirt-engine-common-0:4.2.0-0.0.maste= r.20170408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-vmconsole-proxy-helper-0:4.2.0-0.0.ma= ster.20170408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-setup-plugin-websocket-proxy-0:4.2.0-0.0.master.20= 170408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-tools-0:4.2.0-0.0.master.20170408115943.gitea6bd62= .el7.centos.noarch</div><div = class=3D"">ovirt-engine-tools-backup-0:4.2.0-0.0.master.20170408115943.git= ea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-userportal-0:4.2.0-0.0.master.20170408115943.gitea= 6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-userportal-debuginfo-0:4.2.0-0.0.master.2017040811= 5943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408= 115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-webadmin-portal-0:4.2.0-0.0.master.20170408115943.= gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-webadmin-portal-debuginfo-0:4.2.0-0.0.master.20170= 408115943.gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-websocket-proxy-0:4.2.0-0.0.master.20170408115943.= gitea6bd62.el7.centos.noarch</div><div = class=3D"">ovirt-engine-wildfly-0:10.1.0-1.el7.x86_64</div><div = class=3D"">ovirt-engine-wildfly-overlay-0:10.0.0-1.el7.noarch</div><div = class=3D"">ovirt-engine-yarn-0:0.19.1-2.el7.centos.x86_64</div><div = class=3D"">ovirt-engine-yarn-0:0.19.1-3.el7.centos.noarch</div><div = class=3D"">ovirt-guest-agent-common-0:1.0.13-1.el7.centos.noarch</div><div= = class=3D"">ovirt-guest-agent-debuginfo-0:1.0.13-1.el7.centos.x86_64</div><= div = class=3D"">ovirt-guest-agent-gdm-plugin-0:1.0.13-1.el7.centos.noarch</div>= <div = class=3D"">ovirt-guest-agent-pam-module-0:1.0.13-1.el7.centos.x86_64</div>= <div = class=3D"">ovirt-host-deploy-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.= el7.centos.noarch</div><div = class=3D"">ovirt-host-deploy-java-0:1.7.0-0.0.master.20170407114427.gitb41= f5b9.el7.centos.noarch</div><div = class=3D"">ovirt-host-deploy-javadoc-0:1.7.0-0.0.master.20170407114427.git= b41f5b9.el7.centos.noarch</div><div = class=3D"">ovirt-hosted-engine-ha-0:2.2.0-0.0.master.20170404202238.201704= 04202234.git68258f1.el7.centos.noarch</div><div = class=3D"">ovirt-hosted-engine-setup-0:2.2.0-0.0.master.20170330072130.git= 8a05e04.el7.centos.noarch</div><div = class=3D"">ovirt-image-uploader-0:4.1.0-0.0.master.20160830110830.gitad0be= 34.el7.centos.noarch</div><div = class=3D"">ovirt-imageio-common-0:1.0.0-0.201703211442.gitfa6a59d.el7.cent= os.noarch</div><div = class=3D"">ovirt-imageio-daemon-0:1.0.0-0.201703211442.gitfa6a59d.el7.cent= os.noarch</div><div = class=3D"">ovirt-imageio-proxy-0:1.0.0-0.201703211442.gitfa6a59d.el7.cento= s.noarch</div><div = class=3D"">ovirt-imageio-proxy-setup-0:1.0.0-0.201703211442.gitfa6a59d.el7= .centos.noarch</div><div = class=3D"">ovirt-iso-uploader-0:4.1.0-0.0.master.20160909154152.git14502bd= .el7.centos.noarch</div><div = class=3D"">ovirt-jboss-modules-maven-plugin-0:2.0.2-0.1.20160715.gitec0150= 7.el7.centos.noarch</div><div = class=3D"">ovirt-jboss-modules-maven-plugin-javadoc-0:2.0.2-0.1.20160715.g= itec01507.el7.centos.noarch</div><div = class=3D"">ovirt-js-dependencies-0:0.0.3-2.el7.centos.x86_64</div><div = class=3D"">ovirt-js-dependencies-0:0.0.3-4.el7.centos.noarch</div><div = class=3D"">ovirt-live-artwork-0:1.1.0-0.0.master.20170331075633.git8b425a3= .el7.centos.noarch</div><div = class=3D"">ovirt-live-artwork-gnome-0:1.1.0-0.0.master.20170331075633.git8= b425a3.el7.centos.noarch</div><div = class=3D"">ovirt-live-artwork-mate-0:1.1.0-0.0.master.20170331075633.git8b= 425a3.el7.centos.noarch</div><div = class=3D"">ovirt-log-collector-0:4.1.0-0.0.master.20170405134355.gitd0eaad= 4.el7.centos.noarch</div><div = class=3D"">ovirt-log-collector-analyzer-0:4.1.0-0.0.master.20170405134355.= gitd0eaad4.el7.centos.noarch</div><div = class=3D"">ovirt-node-ng-docs-0:4.1.0-0.20170408.0.el7.noarch</div><div = class=3D"">ovirt-node-ng-image-0:4.2.0-0.3.master.20170406003036.git2f43a3= c.el7.centos.noarch</div><div = class=3D"">ovirt-node-ng-image-update-0:4.2.0-0.3.master.20170406003036.gi= t2f43a3c.el7.centos.noarch</div><div = class=3D"">ovirt-node-ng-image-update-placeholder-0:4.2.0-0.3.master.20170= 408002716.git2f43a3c.el7.centos.noarch</div><div = class=3D"">ovirt-node-ng-nodectl-0:4.1.0-0.20170408.0.el7.noarch</div><div= class=3D"">ovirt-node-ng-tools-0:4.1.0-0.20170408.0.el7.noarch</div><div = class=3D"">ovirt-optimizer-0:0.15-0.176.201702021258.el7.centos.noarch</di= v><div = class=3D"">ovirt-optimizer-dependencies-0:0.15-0.176.201702021258.el7.cent= os.noarch</div><div = class=3D"">ovirt-optimizer-jboss-0:0.15-0.176.201702021258.el7.centos.noar= ch</div><div = class=3D"">ovirt-optimizer-ui-0:0.15-0.176.201702021258.el7.centos.noarch<= /div><div = class=3D"">ovirt-provider-ovn-0:1.1-2.20170407114314.git8f0d7ee.el7.centos= .noarch</div><div = class=3D"">ovirt-provider-ovn-driver-0:1.1-2.20170407114314.git8f0d7ee.el7= .centos.noarch</div><div = class=3D"">ovirt-release-host-node-0:4.2.0-0.3.master.20170408002716.git2f= 43a3c.el7.centos.noarch</div><div = class=3D"">ovirt-release-master-0:4.2.0-0.3.master.20170408002716.git2f43a= 3c.el7.centos.noarch</div><div = class=3D"">ovirt-release-master-experimental-0:4.2.0-0.3.master.2017040800= 2716.git2f43a3c.el7.centos.noarch</div><div = class=3D"">ovirt-setup-lib-0:1.1.0-1.20161219151842.git402fec4.el7.centos.= noarch</div><div = class=3D"">ovirt-vmconsole-0:1.0.4-0.0.master.20170301155209.git50f0014.el= 7.centos.noarch</div><div = class=3D"">ovirt-vmconsole-host-0:1.0.4-0.0.master.20170301155209.git50f00= 14.el7.centos.noarch</div><div = class=3D"">ovirt-vmconsole-proxy-0:1.0.4-0.0.master.20170301155209.git50f0= 014.el7.centos.noarch</div><div = class=3D"">ovirt-web-ui-0:0.1.2-1.el7.centos.x86_64</div><div = class=3D"">python-imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.n= oarch</div><div = class=3D"">python-ioprocess-0:0.17.0-1.201702282324.gite82786d.el7.centos.= noarch</div><div = class=3D"">python-ovirt-engine-sdk4-0:4.2.0-1.a0.20170331gitd6b662b.el7.ce= ntos.x86_64</div><div = class=3D"">python-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.a0.20170331gitd6b6= 62b.el7.centos.x86_64</div><div = class=3D"">qemu-img-ev-10:2.6.0-28.el7_3.6.1.x86_64</div><div = class=3D"">qemu-kvm-common-ev-10:2.6.0-28.el7_3.6.1.x86_64</div><div = class=3D"">qemu-kvm-ev-10:2.6.0-28.el7_3.6.1.x86_64</div><div = class=3D"">qemu-kvm-ev-debuginfo-10:2.6.0-28.el7_3.6.1.x86_64</div><div = class=3D"">qemu-kvm-tools-ev-10:2.6.0-28.el7_3.6.1.x86_64</div><div = class=3D"">rubygem-ovirt-engine-sdk4-0:4.2.0-1.alpha0.20170407gite402de7.e= l7.centos.x86_64</div><div = class=3D"">rubygem-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.alpha0.20170407gi= te402de7.el7.centos.x86_64</div><div = class=3D"">rubygem-ovirt-engine-sdk4-doc-0:4.2.0-1.alpha0.20170407gite402d= e7.el7.centos.x86_64</div><div = class=3D"">unboundid-ldapsdk-0:3.2.0-1.el7.noarch</div><div = class=3D"">unboundid-ldapsdk-javadoc-0:3.2.0-1.el7.noarch</div><div = class=3D"">vdsm-0:4.20.0-612.git378d837.el7.centos.x86_64</div><div = class=3D"">vdsm-api-0:4.20.0-612.git378d837.el7.centos.noarch</div><div = class=3D"">vdsm-client-0:4.20.0-612.git378d837.el7.centos.noarch</div><div= = class=3D"">vdsm-gluster-0:4.20.0-612.git378d837.el7.centos.noarch</div><di= v = class=3D"">vdsm-hook-allocate_net-0:4.20.0-612.git378d837.el7.centos.noarc= h</div><div = class=3D"">vdsm-hook-checkimages-0:4.20.0-612.git378d837.el7.centos.noarch= </div><div = class=3D"">vdsm-hook-checkips-0:4.20.0-612.git378d837.el7.centos.x86_64</d= iv><div = class=3D"">vdsm-hook-diskunmap-0:4.20.0-612.git378d837.el7.centos.noarch</= div><div = class=3D"">vdsm-hook-ethtool-options-0:4.20.0-612.git378d837.el7.centos.no= arch</div><div = class=3D"">vdsm-hook-extnet-0:4.20.0-612.git378d837.el7.centos.noarch</div= div><div = class=3D"">vdsm-hook-fakevmstats-0:4.20.0-612.git378d837.el7.centos.noarch= </div><div = class=3D"">vdsm-hook-faqemu-0:4.20.0-612.git378d837.el7.centos.noarch</div= div = class=3D"">vdsm-hook-fileinject-0:4.20.0-612.git378d837.el7.centos.noarch<= /div><div = class=3D"">vdsm-hook-floppy-0:4.20.0-612.git378d837.el7.centos.noarch</div= div><div = class=3D"">vdsm-hook-ipv6-0:4.20.0-612.git378d837.el7.centos.noarch</div><= div = class=3D"">vdsm-hook-isolatedprivatevlan-0:4.20.0-612.git378d837.el7.cento= s.noarch</div><div = class=3D"">vdsm-hook-localdisk-0:4.20.0-612.git378d837.el7.centos.noarch</= div><div = class=3D"">vdsm-hook-macbind-0:4.20.0-612.git378d837.el7.centos.noarch</di= v><div = class=3D"">vdsm-hook-macspoof-0:4.20.0-612.git378d837.el7.centos.noarch</d= iv><div = class=3D"">vdsm-hook-nestedvt-0:4.20.0-612.git378d837.el7.centos.noarch</d= iv><div = class=3D"">vdsm-hook-noipspoof-0:4.20.0-612.git378d837.el7.centos.noarch</= div><div = class=3D"">vdsm-hook-numa-0:4.20.0-612.git378d837.el7.centos.noarch</div><= div = class=3D"">vdsm-hook-openstacknet-0:4.20.0-612.git378d837.el7.centos.noarc= h</div><div = class=3D"">vdsm-hook-pincpu-0:4.20.0-612.git378d837.el7.centos.noarch</div= div><div = class=3D"">vdsm-infra-0:4.18.999-212.git2556c05.el7.centos.noarch</div><di= v = class=3D"">vdsm-jsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch</div><di= v = class=3D"">vdsm-jsonrpc-java-0:1.4.5-1.20170330074257.gitbef06c7.el7.cento= s.noarch</div><div = class=3D"">vdsm-jsonrpc-java-javadoc-0:1.4.5-1.20170330074257.gitbef06c7.e= l7.centos.noarch</div><div = class=3D"">vdsm-python-0:4.20.0-612.git378d837.el7.centos.noarch</div><div= class=3D"">vdsm-tests-0:4.20.0-612.git378d837.el7.centos.noarch</div><div= = class=3D"">vdsm-xmlrpc-0:4.20.0-612.git378d837.el7.centos.noarch</div><div= = class=3D"">vdsm-yajsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch</div><= /div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D""> </div><blockquote = class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; = border-left-width: 1px; border-left-style: solid; border-left-color: = rgb(204, 204, 204); padding-left: 1ex;"><div style=3D"word-wrap: = break-word;" class=3D""><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">michal</div><div class=3D""><div = class=3D"gmail-h5"><div class=3D""><br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><div = class=3D"">org/apache/commons/lang/<wbr class=3D"">StringUtils</div><div = class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = java.util.concurrent.<wbr class=3D"">FutureTask.report(FutureTask.<wbr = class=3D"">java:122) [rt.jar:1.8.0_121]</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = java.util.concurrent.<wbr class=3D"">FutureTask.get(FutureTask.<wbr = class=3D"">java:206) [rt.jar:1.8.0_121]</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr = class=3D"">FutureVDSCommand.get(<wbr class=3D"">FutureVDSCommand.java:69) = [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.bll.<wbr class=3D"">network.host.<wbr = class=3D"">HostSetupNetworkPoller.<wbr class=3D"">getValue(<wbr = class=3D"">HostSetupNetworkPoller.java:<wbr class=3D"">56) = [bll.jar:]</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.bll.<wbr class=3D"">network.host.<wbr = class=3D"">HostSetupNetworkPoller.poll(<wbr = class=3D"">HostSetupNetworkPoller.java:<wbr class=3D"">41) = [bll.jar:]</div><div class=3D""><br class=3D""></div><div = class=3D"">...</div><div class=3D"">Caused by: java.lang.<wbr = class=3D"">NoClassDefFoundError: org/apache/commons/lang/<wbr = class=3D"">StringUtils</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.vdsm.jsonrpc.client.<wbr = class=3D"">reactors.stomp.impl.Message.<wbr = class=3D"">withCorrelationId(Message.<wbr class=3D"">java:75) = [vdsm-jsonrpc-java-client.jar:<wbr class=3D"">]</div><div class=3D""><span= class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.vdsm.jsonrpc.client.<wbr = class=3D"">reactors.stomp.SSLStompClient.<wbr = class=3D"">sendMessage(SSLStompClient.<wbr class=3D"">java:85) = [vdsm-jsonrpc-java-client.jar:<wbr class=3D"">]</div><div class=3D""><span= class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.vdsm.jsonrpc.client.<wbr class=3D"">JsonRpcClient.call(<wbr = class=3D"">JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:<wbr = class=3D"">]</div><div class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.jsonrpc.FutureMap.<<wbr= class=3D"">init>(FutureMap.java:91) [vdsbroker.jar:]</div><div = class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.jsonrpc.<wbr = class=3D"">JsonRpcVdsServer.lambda$<wbr class=3D"">timeBoundPoll$1(<wbr = class=3D"">JsonRpcVdsServer.java:965) [vdsbroker.jar:]</div><div = class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.jsonrpc.<wbr = class=3D"">JsonRpcVdsServer$<wbr class=3D"">FutureCallable.call(<wbr = class=3D"">JsonRpcVdsServer.java:484) [vdsbroker.jar:]</div><div = class=3D""><span = class=3D"gmail-m_-7272531751306879750gmail-Apple-tab-span" = style=3D"white-space: pre-wrap;"> </span>at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.jsonrpc.<wbr = class=3D"">JsonRpcVdsServer$<wbr class=3D"">FutureCallable.call(<wbr = class=3D"">JsonRpcVdsServer.java:473) [vdsbroker.jar:]</div><div = class=3D""><br class=3D""></div><div class=3D""> </div><blockquote = class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; = border-left-width: 1px; border-left-style: solid; border-left-color: = rgb(204, 204, 204); padding-left: 1ex;"><div dir=3D"ltr" class=3D""><br = class=3D""><br class=3D""><div class=3D""><div class=3D""><div = class=3D"gmail_extra"><div class=3D""><div = class=3D"gmail-m_-7272531751306879750gmail-h5"><br class=3D""><div = class=3D"gmail_quote">On 11 April 2017 at 18:12, Piotr Kliczewski<span = class=3D"Apple-converted-space"> </span><span dir=3D"ltr" = class=3D""><<a href=3D"mailto:piotr.kliczewski@gmail.com" = target=3D"_blank" = class=3D"">piotr.kliczewski@gmail.com</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = dir=3D"ltr" class=3D"">I looks like SD activate failed on the engine = side and the spm connection was gently closed by the engine:<div = class=3D""><br class=3D""></div><div class=3D""><div class=3D"">2017-04-11= 09:10:24,807-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.st/" target=3D"_blank" = class=3D"">org.ovirt.engine.core.bll.st</a><wbr = class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand] = (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all = hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017</div><div = class=3D"">2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.sto<wbr = class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] = (org.ovirt.thread.pool-7-threa<wbr class=3D"">d-29) [2b66a42b] Could not = connect vds 'lago-basic-suite-master-host0<wbr class=3D"">' to pool = 'test-dc' - moving host to non-operational: null</div><div = class=3D"">2017-04-11 09:10:24,842-04 ERROR = [org.ovirt.engine.core.bll.sto<wbr = class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] = (org.ovirt.thread.pool-7-threa<wbr class=3D"">d-28) [2b66a42b] Could not = connect vds 'lago-basic-suite-master-host1<wbr class=3D"">' to pool = 'test-dc' - moving host to non-operational: null</div><div = class=3D"">2017-04-11 09:10:24,904-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.se/" target=3D"_blank" = class=3D"">org.ovirt.engine.core.bll.Se</a><wbr = class=3D"">tNonOperationalVdsCommand] (default task-29) [2bc83b1c] = Running command: SetNonOperationalVdsCommand internal: true. Entities = affected : ID: e1b611f2-67fb-4acb-8261-071331<wbr class=3D"">34e0e7 = Type: VDS</div><div class=3D"">2017-04-11 09:10:24,909-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] = START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParamete<wbr = class=3D"">rs:{runAsync=3D'true', hostId=3D'e1b611f2-67fb-4acb-826<wbr = class=3D"">1-07133134e0e7', status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 5c45239</div><div class=3D"">2017-04-11 09:10:24,919-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] = FINISH, SetVdsStatusVDSCommand, log id: 5c45239</div><div = class=3D"">2017-04-11 09:10:24,922-04 DEBUG = [org.ovirt.engine.core.dal.dbb<wbr = class=3D"">roker.PostgresDbEngineDialect$<wbr = class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled = stored procedure. Call string is [{call = getvmsrunningonvds(?)}]</div><div class=3D"">2017-04-11 09:10:24,922-04 = DEBUG [org.ovirt.engine.core.dal.dbb<wbr = class=3D"">roker.PostgresDbEngineDialect$<wbr = class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall = for procedure [GetVmsRunningOnVds] compiled</div><div = class=3D"">2017-04-11 09:10:24,985-04 WARN = [org.ovirt.engine.core.dal.db<wbr = class=3D"">broker.auditloghandling.AuditL<wbr class=3D"">ogDirector] = (default task-29) [2bc83b1c] EVENT_ID: = VDS_SET_NONOPERATIONAL_DOMAIN(<wbr class=3D"">522), Correlation ID: = 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e98269<wbr class=3D"">44b9a0, = Call Stack: null, Custom Event ID: -1, Message: Host = lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi = attached to the Data Center test-dc. Setting Host state to = Non-Operational.</div><div class=3D"">2017-04-11 09:10:24,985-04 DEBUG = [org.ovirt.engine.core.utils.t<wbr class=3D"">imer.FixedDelayJobListener] = (DefaultQuartzScheduler4) [] Rescheduling<span = class=3D"Apple-converted-space"> </span><a = href=3D"http://default.org/" target=3D"_blank" = class=3D"">DEFAULT.org</a>.ovirt.engine.core.<wbr = class=3D"">vdsbroker.VdsManager.onTimer#-<wbr = class=3D"">9223372036854775741 as there is no unfired trigger.</div><div = class=3D"">2017-04-11 09:10:25,036-04 INFO [<a = href=3D"http://org.ovirt.engine.core.bll.se/" target=3D"_blank" = class=3D"">org.ovirt.engine.core.bll.Se</a><wbr = class=3D"">tNonOperationalVdsCommand] (default task-29) [2ffa5981] = Running command: SetNonOperationalVdsCommand internal: true. Entities = affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0<wbr class=3D"">dafe34 = Type: VDS</div><div class=3D"">2017-04-11 09:10:25,039-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] = START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParamete<wbr = class=3D"">rs:{runAsync=3D'true', hostId=3D'87a5f691-ea5c-4692-a4a<wbr = class=3D"">b-52dad0dafe34', status=3D'NonOperational', = nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', = stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: = 706ffc3b</div></div><div class=3D""><br class=3D""></div><div = class=3D"">.....</div><div class=3D""><br class=3D""></div><div = class=3D""><div class=3D"">2017-04-11 09:10:25,366-04 INFO = [org.ovirt.engine.core.vdsbro<wbr = class=3D"">ker.vdsbroker.SpmStopVDSComman<wbr class=3D"">d] (default = task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: = 62f82cfb</div><div class=3D"">2017-04-11 09:10:25,366-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.reactors.stomp.impl.Message] (default task-29) [2ffa5981] = UNSUBSCRIBE</div><div class=3D"">id:4cd59c67-1b57-4456-9a16-52a<wbr = class=3D"">72ac6f689</div><div class=3D""><br class=3D""></div><div = class=3D"">\00</div><div class=3D"">2017-04-11 09:10:25,367-04 DEBUG = [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.reactors.stomp.StompCommonCli<wbr class=3D"">ent] (default = task-29) [2ffa5981] Message sent: UNSUBSCRIBE</div></div><div = class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote"><div class=3D""><div = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-h5">= On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola<span = class=3D"Apple-converted-space"> </span><span dir=3D"ltr" = class=3D""><<a href=3D"mailto:sbonazzo@redhat.com" target=3D"_blank" = class=3D"">sbonazzo@redhat.com</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""></div></div><blockquote class=3D"gmail_quote" style=3D"margin: = 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = class=3D""><div = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-h5">= <div dir=3D"ltr" class=3D"">Link to Job:<span = class=3D"Apple-converted-space"> </span><a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 303" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/t<wbr = class=3D"">est-repo_ovirt_experimental_ma<wbr class=3D"">ster/6303</a><pre= = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; margin-top: 0px; = margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 14px;"><span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D""><br class=3D""></b></span></pre><pre = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; margin-top: 0px; = margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 14px;"><span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">OST is failing in the last 50 builds</b></span></pre><pre = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; margin-top: 0px; = margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 14px;"><span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D""><br class=3D""></b></span></pre><pre = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; margin-top: 0px; = margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 14px;"><br = class=3D""></pre><pre = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; margin-top: 0px; = margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 14px;"><span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(0, 205, 205);" class=3D""> = # add_secondary_storage_domains: </span> <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(205, 0, 0);" class=3D"">Error = while running thread <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] Traceback (most = recent call last): <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/lago/utils.py", line = 57, in _ret_via_queue <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] = queue.put({'return': func()}) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 427, in add_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.090</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">(prefix, SD_NFS_NAME, = SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version=3D'v4_2') <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">_4(prefix, sd_nfs_name, = nfs_host_name, mount_path, sd_format, sd_type, nfs_version) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] = host=3D_random_host_from_dc_4(ap<wbr class=3D"">i, DC_NAME), <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] return = random.choice(_hosts_in_dc_4(a<wbr class=3D"">pi, dc_name)) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.091</b> </span>[basic_suit_el7] raise = RuntimeError('Could not find hosts that are up in DC %s' % dc_name) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] RuntimeError: Could = not find hosts that are up in DC test-dc</span> <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(205, 0, 0);" class=3D"">Error = while running thread <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] Traceback (most = recent call last): <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/lago/utils.py", line = 57, in _ret_via_queue <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] = queue.put({'return': func()}) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 583, in add_iso_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">(prefix, SD_ISO_NAME, = SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format=3D'v1', sd_type=3D'iso', = nfs_version=3D'v3') <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.092</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">_4(prefix, sd_nfs_name, = nfs_host_name, mount_path, sd_format, sd_type, nfs_version) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] = host=3D_random_host_from_dc_4(ap<wbr class=3D"">i, DC_NAME), <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] return = random.choice(_hosts_in_dc_4(a<wbr class=3D"">pi, dc_name)) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] raise = RuntimeError('Could not find hosts that are up in DC %s' % dc_name) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.093</b> </span>[basic_suit_el7] RuntimeError: Could = not find hosts that are up in DC test-dc</span> <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(205, 0, 0);" class=3D"">Error = while running thread <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] Traceback (most = recent call last): <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/lago/utils.py", line = 57, in _ret_via_queue <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] = queue.put({'return': func()}) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 587, in add_templates_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">(prefix, = SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, = sd_format=3D'v1', sd_type=3D'export', nfs_version=3D'v4_1') <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">_4(prefix, sd_nfs_name, = nfs_host_name, mount_path, sd_format, sd_type, nfs_version) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.094</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] = host=3D_random_host_from_dc_4(ap<wbr class=3D"">i, DC_NAME), <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] return = random.choice(_hosts_in_dc_4(a<wbr class=3D"">pi, dc_name)) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] raise = RuntimeError('Could not find hosts that are up in DC %s' % dc_name) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] RuntimeError: Could = not find hosts that are up in DC test-dc</span> <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(205, 0, 0);" class=3D"">Error = while running thread <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.095</b> </span>[basic_suit_el7] Traceback (most = recent call last): <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/lago/utils.py", line = 57, in _ret_via_queue <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] = queue.put({'return': func()}) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 436, in add_second_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] = SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 441, in add_generic_nfs_storage_domain <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] = add_generic_nfs_storage_domain<wbr class=3D"">_4(prefix, sd_nfs_name, = nfs_host_name, mount_path, sd_format, sd_type, nfs_version) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] = host=3D_random_host_from_dc_4(ap<wbr class=3D"">i, DC_NAME), <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.096</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 106, in _random_host_from_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] return = random.choice(_hosts_in_dc_4(a<wbr class=3D"">pi, dc_name)) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 100, in _hosts_in_dc_4 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] raise = RuntimeError('Could not find hosts that are up in DC %s' % dc_name) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] RuntimeError: Could = not find hosts that are up in DC test-dc</span> <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] <span = style=3D"box-sizing: border-box; color: rgb(205, 0, 0);" class=3D"">Error = while running thread <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] Traceback (most = recent call last): <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.097</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/lago/utils.py", line = 57, in _ret_via_queue <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] = queue.put({'return': func()}) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 803, in import_template_from_glance <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] = generic_import_from_glance(api<wbr class=3D"">, = image_name=3DCIRROS_IMAGE_NAME, image_ext=3D'_glance_template', = as_template=3DTrue) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] File = "/home/jenkins/workspace/test-<wbr = class=3D"">repo_ovirt_experimental_master<wbr = class=3D"">/ovirt-system-tests/basic-suit<wbr = class=3D"">e-master/test-scenarios/002_bo<wbr class=3D"">otstrap.py", = line 637, in generic_import_from_glance <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] = target_image.import_image(impo<wbr class=3D"">rt_action) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/br<wbr class=3D"">okers.py", line = 26017, in import_image <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] = headers=3D{"Correlation-Id":corr<wbr class=3D"">elation_id} <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/pr<wbr class=3D"">oxy.py", line = 122, in request <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.098</b> </span>[basic_suit_el7] = persistent_auth=3Dself.__persist<wbr class=3D"">ent_auth <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/co<wbr = class=3D"">nnectionspool.py", line 79, in do_request <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] persistent_auth) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] File = "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/ovirtsdk/infrastructure/co<wbr = class=3D"">nnectionspool.py", line 162, in __do_request <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] raise = errors.RequestError(response_c<wbr class=3D"">ode, response_reason, = response_body) <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] RequestError:=20 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] status: 400 <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] reason: Bad Request <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] detail: Cannot = import Virtual Disk: Storage Domain cannot be accessed. <span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">00:18:22.099</b> </span>[basic_suit_el7] -Please check that = at least one Host is operational and Data Center state is = up.</span></pre><span = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735HOEnZb"><font color=3D"#888888" class=3D""><div = class=3D""><br class=3D""></div>--<span = class=3D"Apple-converted-space"> </span><br class=3D""><div = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail-m_30= 46267183852181735m_5254582280271388056gmail_signature"><div dir=3D"ltr" = class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div dir=3D"ltr" = class=3D""><div dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div = style=3D"font-family: overpass, sans-serif; font-weight: bold; margin: = 0px; padding: 0px; font-size: 14px; text-transform: uppercase;" = class=3D""><span class=3D"">SANDRO</span> <span = class=3D"">BONAZZOLA</span></div><p style=3D"font-family: overpass, = sans-serif; font-size: 10px; margin: 0px 0px 4px; text-transform: = uppercase;" class=3D""><span class=3D"">ASSOCIATE MANAGER, SOFTWARE = ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><div = style=3D"font-family: overpass, sans-serif; margin: 0px; font-size: = 10px; color: rgb(153, 153, 153);" class=3D""><a = href=3D"https://www.redhat.com/" target=3D"_blank" style=3D"color: = rgb(0, 136, 206); margin: 0px;" class=3D"">Red Hat <span = class=3D"">EMEA</span></a></div><table border=3D"0" style=3D"font-family: = overpass, sans-serif; font-size: inherit;" class=3D""><tbody = class=3D""><tr class=3D""><td width=3D"100px" class=3D""><a = href=3D"https://red.ht/sig" target=3D"_blank" class=3D""><img width=3D"90"= height=3D"auto" class=3D""></a></td><td style=3D"font-size: 10px;" = class=3D""><div class=3D""><a href=3D"https://redhat.com/trusted" = target=3D"_blank" style=3D"color: rgb(204, 0, 0); font-weight: bold;" = class=3D"">TRIED. TESTED. = TRUSTED.</a></div></td></tr></tbody></table></div></div></div></div></div>= </div></div></font></span></div><br = class=3D""></div></div>______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br = class=3D""></div><br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br = class=3D""><br clear=3D"all" class=3D""><br class=3D""></div></div><span = class=3D"gmail-m_-7272531751306879750gmail-HOEnZb"><font color=3D"#888888"= class=3D"">--<span class=3D"Apple-converted-space"> </span><br = class=3D""><div = class=3D"gmail-m_-7272531751306879750gmail-m_5006786018485841575gmail_sign= ature">Barak Korren<br class=3D""><a href=3D"mailto:bkorren@redhat.com" = target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">RHCE, = RHCi, RHV-DevOps Team<br class=3D""><a = href=3D"https://ifireball.wordpress.com/" target=3D"_blank" = class=3D"">https://ifireball.wordpress.co<wbr = class=3D"">m/</a></div></font></span></div></div></div></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br = class=3D""></div></div>______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a></div></blockquote></div><br = class=3D""></div></div></div></blockquote></div><br style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><br clear=3D"all" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><div style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = class=3D""></div><span style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: = inline !important;" class=3D"">--<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div class=3D"gmail_signature"= style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;"><div dir=3D"ltr" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div = dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div = style=3D"font-family: overpass, sans-serif; font-weight: bold; margin: = 0px; padding: 0px; font-size: 14px; text-transform: uppercase;" = class=3D""><span class=3D"">SANDRO</span> <span = class=3D"">BONAZZOLA</span></div><p style=3D"font-family: overpass, = sans-serif; font-size: 10px; margin: 0px 0px 4px; text-transform: = uppercase;" class=3D""><span class=3D"">ASSOCIATE MANAGER, SOFTWARE = ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><div = style=3D"font-family: overpass, sans-serif; margin: 0px; font-size: = 10px; color: rgb(153, 153, 153);" class=3D""><a = href=3D"https://www.redhat.com/" target=3D"_blank" style=3D"color: = rgb(0, 136, 206); margin: 0px;" class=3D"">Red Hat <span = class=3D"">EMEA</span></a></div><table border=3D"0" style=3D"font-family: = overpass, sans-serif; font-size: inherit;" class=3D""><tbody = class=3D""><tr class=3D""><td width=3D"100px" class=3D""><a = href=3D"https://red.ht/sig" target=3D"_blank" class=3D""><img = src=3D"https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red= -hat-black.png" width=3D"90" height=3D"auto" class=3D""></a></td><td = style=3D"font-size: 10px;" class=3D""><div class=3D""><a = href=3D"https://redhat.com/trusted" target=3D"_blank" style=3D"color: = rgb(204, 0, 0); font-weight: bold;" class=3D"">TRIED. TESTED. = TRUSTED.</a></div></td></tr></tbody></table></div></div></div></div></div>= </div></div></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_9B8B03C8-0839-4F3A-94B7-1252B9DCF326--
On 12 April 2017 at 12:44, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Well, Im not sure who’s handling that exactly, but vdsm-jsonrpc-java is getting into OST runs despite the OST failing since Saturday. That should not happen
There is no mechanism for blocking stuff from going into 'experimental' ATM (Please differentiate between the 'experimental' flow that automates the testing and the OST which is the set of testing tools that can be used in various configurations). 'experimental' generally runs on the latest builds of all packages. We're working on something that will allow blocking failed changes, but it still some ways off (OVIRT-1077). btw, in the meantime, me and Tomas are working on narrowing down the
original failure.
Is there a quick fix that could be put in place to get 'experimental' back to work for everything else? -- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 12:10 PM, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 12:44, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Well, Im not sure who’s handling that exactly, but vdsm-jsonrpc-java is getting into OST runs despite the OST failing since Saturday. That should not happen
There is no mechanism for blocking stuff from going into 'experimental' ATM (Please differentiate between the 'experimental' flow that automates the testing and the OST which is the set of testing tools that can be used in various configurations). 'experimental' generally runs on the latest builds of all packages.
This sounds like using snapshots and we moved away from them to relay on versions. I do not want other people to use my changes if I do not specify so in the engine spec and pom. Please make sure to use only tagged version.
We're working on something that will allow blocking failed changes, but it still some ways off (OVIRT-1077).
btw, in the meantime, me and Tomas are working on narrowing down the
original failure.
Is there a quick fix that could be put in place to get 'experimental' back to work for everything else?
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
This sounds like using snapshots and we moved away from them to relay on versions. I do not want other people to use my changes if I do not specify so in the engine spec and pom. Please make sure to use only tagged version.
If you don't want your changes to be used don't merge them. If we limit the CI system to only use tagged stuff it becomes useless. -- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 12:30 PM, Barak Korren <bkorren@redhat.com> wrote:
This sounds like using snapshots and we moved away from them to relay on versions. I do not want other people to use my changes if I do not
specify
so in the engine spec and pom. Please make sure to use only tagged version.
If you don't want your changes to be used don't merge them.
I need to merge them and release before I can make the engine build to succeed and if the are api changes or dependency changes like in this thread I do not want my changes to be used without the engine update. Sorry but we can't workaround it without making sure that we are using correct versions.
If we limit the CI system to only use tagged stuff it becomes useless.
I makes sense for OST.
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On 12 April 2017 at 13:33, Piotr Kliczewski <pkliczew@redhat.com> wrote:
I need to merge them and release before I can make the engine build to succeed and if the are api changes or dependency changes like in this thread I do not want my changes to be used without the engine update.
Sorry but we can't workaround it without making sure that we are using correct versions.
So this is a case where change A in project X needs change B in project Y. We will eventually allow some syntax of specifying something like this (probably in the commit message like OpenStack) to ensure they are merged and tested together. In the meantime the closest approximation we have is merging such changes as simultaneously as possible. Consider what it means when an API-breaking change is merged into vdsm master for example without the required engine-side change. It means that from that point on, all further vdsm changes will need to wait until the engine change is merged.
If we limit the CI system to only use tagged stuff it becomes useless.
I makes sense for OST.
As I've commented before, you need to be clear what you mean when you say 'OST'. If you clone the OST repo and run it, or use the manual job you end up using the releases repos or (for master) the 'tested' repos. The point of the 'experimental' flow is to tell you if the sum total of oVirt's 'master' branches (or more general, branches by version) is stable or not. Not running it on the latest patches would not make sense. -- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 1:44 PM, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 13:33, Piotr Kliczewski <pkliczew@redhat.com> wrote:
I need to merge them and release before I can make the engine build to succeed and if the are api changes or dependency changes like in this thread I do not
want
my changes to be used without the engine update.
Sorry but we can't workaround it without making sure that we are using correct versions.
So this is a case where change A in project X needs change B in project Y. We will eventually allow some syntax of specifying something like this (probably in the commit message like OpenStack) to ensure they are merged and tested together. In the meantime the closest approximation we have is merging such changes as simultaneously as possible.
It is not possible due to maven central taking time to propagate the artifact.
Consider what it means when an API-breaking change is merged into vdsm master for example without the required engine-side change. It means that from that point on, all further vdsm changes will need to wait until the engine change is merged.
If we limit the CI system to only use tagged stuff it becomes useless.
I makes sense for OST.
As I've commented before, you need to be clear what you mean when you say 'OST'. If you clone the OST repo and run it, or use the manual job you end up using the releases repos or (for master) the 'tested' repos.
Which seems not to be the case for the last failure.
The point of the 'experimental' flow is to tell you if the sum total of oVirt's 'master' branches (or more general, branches by version) is stable or not. Not running it on the latest patches would not make sense.
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
--Apple-Mail=_7CDDE311-0F91-4C3F-9F1A-87378292C2ED Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 12 Apr 2017, at 12:10, Barak Korren <bkorren@redhat.com> wrote: =20 =20 =20 On 12 April 2017 at 12:44, Michal Skrivanek = <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: =20 Well, Im not sure who=E2=80=99s handling that exactly, but = vdsm-jsonrpc-java is getting into OST runs despite the OST failing since = Saturday. That should not happen =20 There is no mechanism for blocking stuff from going into = 'experimental' ATM (Please differentiate between the 'experimental' = flow that automates the testing and the OST which is the set of testing = tools that can be used in various configurations). 'experimental' generally runs on the latest builds of all packages. We're working on something that will allow blocking failed changes, = but it still some ways off (OVIRT-1077). =20 btw, in the meantime, me and Tomas are working on narrowing down the = original failure. =20 Is there a quick fix that could be put in place to get 'experimental' = back to work for everything else?
we finally managed to narrow it down and confirm the set of patches = causing the problem on Saturday (had to work around the jsonrpc-java and = some other stuff) engine artifacts[1] passed manual OST[2] now (and sorry Ido for aborting = your job) patches should be taken in shortly Thanks, michal [1] = http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_6= 4/3475/ = <http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_= 64/3475/> [2] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244 = <http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244>
=20 =20 --=20 Barak Korren bkorren@redhat.com <mailto:bkorren@redhat.com> RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/>
--Apple-Mail=_7CDDE311-0F91-4C3F-9F1A-87378292C2ED Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2017, at 12:10, Barak Korren <<a = href=3D"mailto:bkorren@redhat.com" class=3D"">bkorren@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 12 April 2017 at 12:44, Michal = Skrivanek <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D""><div class=3D""><span = class=3D""><div class=3D""><br class=3D""></div></span>Well, Im not sure = who=E2=80=99s handling that exactly, but vdsm-jsonrpc-java is getting = into OST runs despite the OST failing since Saturday. That should not = happen</div></div></blockquote><div class=3D""><br class=3D""></div><div = class=3D"">There is no mechanism for blocking stuff from going into = 'experimental' ATM (Please differentiate between the = 'experimental' flow that automates the testing and the OST which is the = set of testing tools that can be used in various configurations).<br = class=3D""></div><div class=3D"">'experimental' generally runs on the = latest builds of all packages.<br class=3D"">We're working on something = that will allow blocking failed changes, but it still some ways off = (OVIRT-1077).<br class=3D""><br class=3D""></div><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div style=3D"word-wrap:break-word" = class=3D""><div class=3D"">btw, in the meantime, me and Tomas are = working on narrowing down the original = failure.</div></div></blockquote><div class=3D""><br class=3D""></div><div= class=3D"">Is there a quick fix that could be put in place to get = 'experimental' back to work for everything else?<br = class=3D""></div></div></div></div></div></blockquote><div><br = class=3D""></div>we finally managed to narrow it down and confirm the = set of patches causing the problem on Saturday (had to work around the = jsonrpc-java and some other stuff)</div><div>engine artifacts[1] passed = manual OST[2] now (and sorry Ido for aborting your = job)</div><div>patches should be taken in shortly</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""></div><div>[1] <a = href=3D"http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-e= l7-x86_64/3475/" = class=3D"">http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifact= s-el7-x86_64/3475/</a></div><div>[2] <a = href=3D"http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244" = class=3D"">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244</a><= /div><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><div class=3D""><br class=3D""> </div></div>-- = <br class=3D""><div class=3D"gmail_signature" = data-smartmail=3D"gmail_signature">Barak Korren<br class=3D""><a = href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a><br class=3D"">RHCE, RHCi, RHV-DevOps = Team<br class=3D""><a href=3D"https://ifireball.wordpress.com/" = target=3D"_blank" class=3D"">https://ifireball.wordpress.com/</a></div> </div></div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_7CDDE311-0F91-4C3F-9F1A-87378292C2ED--
--Apple-Mail=_E16D14D1-0B57-447A-B5F3-7816A2A77971 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 12 Apr 2017, at 12:10, Barak Korren <bkorren@redhat.com = <mailto:bkorren@redhat.com>> wrote: =20 =20 =20 On 12 April 2017 at 12:44, Michal Skrivanek = <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: =20 Well, Im not sure who=E2=80=99s handling that exactly, but = vdsm-jsonrpc-java is getting into OST runs despite the OST failing since = Saturday. That should not happen =20 There is no mechanism for blocking stuff from going into = 'experimental' ATM (Please differentiate between the 'experimental' = flow that automates the testing and the OST which is the set of testing = tools that can be used in various configurations). 'experimental' generally runs on the latest builds of all packages. We're working on something that will allow blocking failed changes, = but it still some ways off (OVIRT-1077). =20 btw, in the meantime, me and Tomas are working on narrowing down the = original failure. =20 Is there a quick fix that could be put in place to get 'experimental' = back to work for everything else? =20 we finally managed to narrow it down and confirm the set of patches = causing the problem on Saturday (had to work around the jsonrpc-java and = some other stuff) engine artifacts[1] passed manual OST[2] now (and sorry Ido for = aborting your job)
On 12 Apr 2017, at 13:39, Michal Skrivanek = <michal.skrivanek@redhat.com> wrote: =20 =20 patches should be taken in shortly
so, they are all in, OST run, but failed on the jsonrpc-java package Can you please manually replace it with the last working version: = http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/r= pm/el7/noarch/vdsm-jsonrpc-java-1.4.5-1.20170330074257.gitbef06c7.el7.cent= os.noarch.rpm = <http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/= rpm/el7/noarch/vdsm-jsonrpc-java-1.4.5-1.20170330074257.gitbef06c7.el7.cen= tos.noarch.rpm> to see if the revert works?
=20 Thanks, michal =20 [1] = http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_6= 4/3475/ = <http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_= 64/3475/> [2] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244 = <http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244> =20
=20 =20 --=20 Barak Korren bkorren@redhat.com <mailto:bkorren@redhat.com> RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/>
--Apple-Mail=_E16D14D1-0B57-447A-B5F3-7816A2A77971 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2017, at 13:39, Michal Skrivanek <<a = href=3D"mailto:michal.skrivanek@redhat.com" = class=3D"">michal.skrivanek@redhat.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D""><br = class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2017, at 12:10, Barak Korren <<a = href=3D"mailto:bkorren@redhat.com" class=3D"">bkorren@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 12 April 2017 at 12:44, Michal = Skrivanek <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D""><div class=3D""><span = class=3D""><div class=3D""><br class=3D""></div></span>Well, Im not sure = who=E2=80=99s handling that exactly, but vdsm-jsonrpc-java is getting = into OST runs despite the OST failing since Saturday. That should not = happen</div></div></blockquote><div class=3D""><br class=3D""></div><div = class=3D"">There is no mechanism for blocking stuff from going into = 'experimental' ATM (Please differentiate between the = 'experimental' flow that automates the testing and the OST which is the = set of testing tools that can be used in various configurations).<br = class=3D""></div><div class=3D"">'experimental' generally runs on the = latest builds of all packages.<br class=3D"">We're working on something = that will allow blocking failed changes, but it still some ways off = (OVIRT-1077).<br class=3D""><br class=3D""></div><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div style=3D"word-wrap:break-word" = class=3D""><div class=3D"">btw, in the meantime, me and Tomas are = working on narrowing down the original = failure.</div></div></blockquote><div class=3D""><br class=3D""></div><div= class=3D"">Is there a quick fix that could be put in place to get = 'experimental' back to work for everything else?<br = class=3D""></div></div></div></div></div></blockquote><div class=3D""><br = class=3D""></div>we finally managed to narrow it down and confirm the = set of patches causing the problem on Saturday (had to work around the = jsonrpc-java and some other stuff)</div><div class=3D"">engine = artifacts[1] passed manual OST[2] now (and sorry Ido for aborting your = job)</div><div class=3D"">patches should be taken in = shortly</div></div></div></blockquote><div><br class=3D""></div>so, they = are all in, OST run, but failed on the jsonrpc-java = package</div><div>Can you please manually replace it with the last = working version:</div><div><a = href=3D"http://resources.ovirt.org/repos/ovirt/experimental/master/latest.= tested/rpm/el7/noarch/vdsm-jsonrpc-java-1.4.5-1.20170330074257.gitbef06c7.= el7.centos.noarch.rpm" = class=3D"">http://resources.ovirt.org/repos/ovirt/experimental/master/late= st.tested/rpm/el7/noarch/vdsm-jsonrpc-java-1.4.5-1.20170330074257.gitbef06= c7.el7.centos.noarch.rpm</a></div><div><br class=3D""></div><div>to see = if the revert works?</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">michal</div><div class=3D""><br = class=3D""></div><div class=3D"">[1] <a = href=3D"http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-e= l7-x86_64/3475/" = class=3D"">http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifact= s-el7-x86_64/3475/</a></div><div class=3D"">[2] <a = href=3D"http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244" = class=3D"">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244</a><= /div><div class=3D""><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><div class=3D""><br = class=3D""> </div></div>-- <br class=3D""><div = class=3D"gmail_signature" data-smartmail=3D"gmail_signature">Barak = Korren<br class=3D""><a href=3D"mailto:bkorren@redhat.com" = target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">RHCE, = RHCi, RHV-DevOps Team<br class=3D""><a = href=3D"https://ifireball.wordpress.com/" target=3D"_blank" = class=3D"">https://ifireball.wordpress.com/</a></div> </div></div> </div></blockquote></div><br class=3D""></div></div></blockquote></div><br= class=3D""></body></html>= --Apple-Mail=_E16D14D1-0B57-447A-B5F3-7816A2A77971--
On Wed, Apr 12, 2017 at 3:36 PM, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 12 Apr 2017, at 13:39, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 12 Apr 2017, at 12:10, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 12:44, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Well, Im not sure who’s handling that exactly, but vdsm-jsonrpc-java is getting into OST runs despite the OST failing since Saturday. That should not happen
There is no mechanism for blocking stuff from going into 'experimental' ATM (Please differentiate between the 'experimental' flow that automates the testing and the OST which is the set of testing tools that can be used in various configurations). 'experimental' generally runs on the latest builds of all packages. We're working on something that will allow blocking failed changes, but it still some ways off (OVIRT-1077).
btw, in the meantime, me and Tomas are working on narrowing down the original failure.
Is there a quick fix that could be put in place to get 'experimental' back to work for everything else?
we finally managed to narrow it down and confirm the set of patches causing the problem on Saturday (had to work around the jsonrpc-java and some other stuff) engine artifacts[1] passed manual OST[2] now (and sorry Ido for aborting your job) patches should be taken in shortly
so, they are all in, OST run, but failed on the jsonrpc-java package Can you please manually replace it with the last working version: http://resources.ovirt.org/repos/ovirt/experimental/master/latest.tested/rpm...
Or we could merge https://gerrit.ovirt.org/#/c/75333/ to fix this issue.
to see if the revert works?
Thanks, michal
[1] http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64/... [2] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/244
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 12 April 2017 at 16:42, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Or we could merge https://gerrit.ovirt.org/#/c/75333/ to fix this issue.
Maybe I'm missing something, why not go for it? -- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 3:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 16:42, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Or we could merge https://gerrit.ovirt.org/#/c/75333/ to fix this issue.
Maybe I'm missing something, why not go for it?
We need an ack
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
On Wed, Apr 12, 2017 at 4:55 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Wed, Apr 12, 2017 at 3:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 16:42, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Or we could merge https://gerrit.ovirt.org/#/c/75333/ to fix this
issue.
Maybe I'm missing something, why not go for it?
We need an ack
You got it
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Apr 12, 2017 at 3:57 PM, Tal Nisan <tnisan@redhat.com> wrote:
On Wed, Apr 12, 2017 at 4:55 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Wed, Apr 12, 2017 at 3:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 12 April 2017 at 16:42, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Or we could merge https://gerrit.ovirt.org/#/c/75333/ to fix this
issue.
Maybe I'm missing something, why not go for it?
We need an ack
You got it
Merged
-- Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_D617B269-0A9D-4704-9C35-6EBA290A1187 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii back to green[1] thanks Tomas and Piotr! [1] = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6332/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6332/>
On 12 Apr 2017, at 15:58, Piotr Kliczewski <pkliczew@redhat.com> = wrote: =20 =20 =20 On Wed, Apr 12, 2017 at 3:57 PM, Tal Nisan <tnisan@redhat.com = <mailto:tnisan@redhat.com>> wrote: =20 =20 On Wed, Apr 12, 2017 at 4:55 PM, Piotr Kliczewski = <piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote: On Wed, Apr 12, 2017 at 3:45 PM, Barak Korren <bkorren@redhat.com = <mailto:bkorren@redhat.com>> wrote:
On 12 April 2017 at 16:42, Piotr Kliczewski = <piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote:
Or we could merge https://gerrit.ovirt.org/#/c/75333/ =
<https://gerrit.ovirt.org/#/c/75333/> to fix this issue.
Maybe I'm missing something, why not go for it?
=20 We need an ack You got it=20 =20 Merged =20 =20
-- Barak Korren bkorren@redhat.com <mailto:bkorren@redhat.com> RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/>
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_D617B269-0A9D-4704-9C35-6EBA290A1187 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">back to green[1]<div class=3D"">thanks Tomas and = Piotr!</div><div class=3D""><br class=3D""></div><div = class=3D"">[1] <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6= 332/" = class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_maste= r/6332/</a></div><div class=3D""><br class=3D""><div><blockquote = type=3D"cite" class=3D""><div class=3D"">On 12 Apr 2017, at 15:58, Piotr = Kliczewski <<a href=3D"mailto:pkliczew@redhat.com" = class=3D"">pkliczew@redhat.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D""><br class=3D""><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Wed, Apr 12, 2017 at 3:57 PM, Tal Nisan <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:tnisan@redhat.com" = target=3D"_blank" class=3D"">tnisan@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D""><br class=3D""><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote"><span class=3D"">On Wed, Apr 12, 2017 at 4:55 PM, = Piotr Kliczewski <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:piotr.kliczewski@gmail.com" target=3D"_blank" = class=3D"">piotr.kliczewski@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On = Wed, Apr 12, 2017 at 3:45 PM, Barak Korren <<a = href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a>> wrote:<br class=3D""> > On 12 April 2017 at 16:42, Piotr Kliczewski <<a = href=3D"mailto:piotr.kliczewski@gmail.com" target=3D"_blank" = class=3D"">piotr.kliczewski@gmail.com</a>> wrote:<br class=3D""> >><br class=3D""> >> Or we could merge <a href=3D"https://gerrit.ovirt.org/#/c/75333/"= rel=3D"noreferrer" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/7<wbr class=3D"">5333/</a> to = fix this issue.<br class=3D""> >><br class=3D""> > Maybe I'm missing something, why not go for it?<br class=3D""> ><br class=3D""> <br class=3D""> </span>We need an ack<br class=3D""></blockquote></span><div = class=3D"">You got it </div></div></div></div></blockquote><div = class=3D""><br class=3D""></div><div class=3D"">Merged<br = class=3D""></div><div class=3D""> </div><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div dir=3D"ltr" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"> <div class=3D"m_2990112596878590662HOEnZb"><div = class=3D"m_2990112596878590662h5"><span class=3D""><br class=3D""> ><br class=3D""> > --<br class=3D""> > Barak Korren<br class=3D""> > <a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a><br class=3D""> > RHCE, RHCi, RHV-DevOps Team<br class=3D""> > <a href=3D"https://ifireball.wordpress.com/" rel=3D"noreferrer" = target=3D"_blank" class=3D"">https://ifireball.wordpress.co<wbr = class=3D"">m/</a><br class=3D""></span><span class=3D""> ______________________________<wbr class=3D"">_________________<br = class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> </span></div></div></blockquote></div><br class=3D""></div></div> </blockquote></div><br class=3D""></div></div> _______________________________________________<br class=3D"">Devel = mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote=
</div><br class=3D""></div></body></html>=
--Apple-Mail=_D617B269-0A9D-4704-9C35-6EBA290A1187--
participants (7)
-
Barak Korren -
Michal Skrivanek -
Piotr Kliczewski -
Piotr Kliczewski -
Sandro Bonazzola -
Tal Nisan -
Yaniv Kaul