Re: [ovirt-devel] [OST Failure Report][oVirt master][2017-04-11] add_secondary_storage_domains

12 Apr 2017

      --Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8
...
On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul@redhat.com> wrote:
=20
=20
=20
On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren@redhat.com =
<mailto:bkorren@redhat.com>> wrote:
This seems to have been the Secodary SD issue I reported on Sunday:
http://lists.ovirt.org/pipermail/devel/2017-April/030139.html =
<http://lists.ovirt.org/pipermail/devel/2017-April/030139.html>
=20
That was up until build #6307 where AddHost started failing:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307>
=20
That's an entirely different game:
2017-04-11 11:24:36,783-04 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] =
(org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: =
java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
2017-04-11 11:24:36,783-04 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] =
(org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: =
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
...
org/apache/commons/lang/StringUtils
  at java.util.concurrent.FutureTask.report(FutureTask.java:122) =
[rt.jar:1.8.0_121]
  at java.util.concurrent.FutureTask.get(FutureTask.java:206) =
[rt.jar:1.8.0_121]
  at =
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo=
mmand.java:69) [vdsbroker.jar:]
  at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos=
tSetupNetworkPoller.java:56) [bll.jar:]
  at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet=
upNetworkPoller.java:41) [bll.jar:]
=20
...
Caused by: java.lang.NoClassDefFoundError: =
org/apache/commons/lang/StringUtils
  at =
org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationI=
d(Message.java:75) [vdsm-jsonrpc-java-client.jar:]
  at =
org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SS=
LStompClient.java:85) [vdsm-jsonrpc-java-client.jar:]
  at =
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) =
[vdsm-jsonrpc-java-client.jar:]
  at =
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91=
) [vdsbroker.jar:]
  at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundP=
...
at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca=
ll(JsonRpcVdsServer.java:484) [vdsbroker.jar:]
  at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca=
ll(JsonRpcVdsServer.java:473) [vdsbroker.jar:]
=20
=20
=20
=20
=20
On 11 April 2017 at 18:12, Piotr Kliczewski =
<piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote:
I looks like SD activate failed on the engine side and the spm =
connection was gently closed by the engine:
=20
2017-04-11 09:10:24,807-04 INFO  [org.ovirt.engine.core.bll.st =
<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCo=
mmand] (default task-29) [2b66a42b] ActivateStorage Domain. After =
Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017
2017-04-11 09:10:24,842-04 ERROR =
[org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] =
(org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds =
'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to =
non-operational: null
2017-04-11 09:10:24,842-04 ERROR =
[org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] =
(org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds =
'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to =
non-operational: null
2017-04-11 09:10:24,904-04 INFO  [org.ovirt.engine.core.bll.Se =
<http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] =
(default task-29) [2bc83b1c] Running command: =
SetNonOperationalVdsCommand internal: true. Entities affected :  ID: =
e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS
2017-04-11 09:10:24,909-04 INFO  =
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default =
task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName =3D =
lago-basic-suite-master-host0, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', =
hostId=3D'e1b611f2-67fb-4acb-8261-07133134e0e7', =
status=3D'NonOperational', =
nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', =
stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: =
5c45239
2017-04-11 09:10:24,919-04 INFO  =
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default =
task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239
2017-04-11 09:10:24,922-04 DEBUG =
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple=
JdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call =
string is [{call getvmsrunningonvds(?)}]
2017-04-11 09:10:24,922-04 DEBUG =
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple=
JdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure =
[GetVmsRunningOnVds] compiled
2017-04-11 09:10:24,985-04 WARN  =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(default task-29) [2bc83b1c] EVENT_ID: =
VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: =
d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: =
-1, Message: Host lago-basic-suite-master-host0 cannot access the =
Storage Domain(s) iscsi attached to the Data Center test-dc. Setting =
Host state to Non-Operational.
2017-04-11 09:10:24,985-04 DEBUG =
[org.ovirt.engine.core.utils.timer.FixedDelayJobListener] =
(DefaultQuartzScheduler4) [] Rescheduling =
DEFAULT.org.ovirt.engine.core.vdsbroker.VdsManager.onTimer#-92233720368547=
75741 as there is no unfired trigger.
2017-04-11 09:10:25,036-04 INFO  [org.ovirt.engine.core.bll.Se =
<http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] =
(default task-29) [2ffa5981] Running command: =
SetNonOperationalVdsCommand internal: true. Entities affected :  ID: =
87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS
2017-04-11 09:10:25,039-04 INFO  =
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default =
task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName =3D =
lago-basic-suite-master-host1, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', =
hostId=3D'87a5f691-ea5c-4692-a4ab-52dad0dafe34', =
status=3D'NonOperational', =
nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', =
stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: =
706ffc3b
=20
.....
=20
2017-04-11 09:10:25,366-04 INFO  =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default =
task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb
2017-04-11 09:10:25,366-04 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default =
task-29) [2ffa5981] UNSUBSCRIBE
id:4cd59c67-1b57-4456-9a16-52a72ac6f689
=20
\00
2017-04-11 09:10:25,367-04 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] =
(default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
=20
=20
=20
On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo@redhat.com =
<mailto:sbonazzo@redhat.com>> wrote:
Link to Job: =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303 =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303>
=20
OST is failing in the last 50 builds
=20
=20
00:18:22.090 [basic_suit_el7]   # add_secondary_storage_domains:=20
00:18:22.090 [basic_suit_el7] Error while running thread
00:18:22.090 [basic_suit_el7] Traceback (most recent call last):
00:18:22.090 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in =
_ret_via_queue
00:18:22.090 [basic_suit_el7]     queue.put({'return': func()})
00:18:22.090 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in =
add_nfs_storage_domain
00:18:22.090 [basic_suit_el7]     =
add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, =
SD_NFS_PATH, nfs_version=3D'v4_2')
00:18:22.091 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in =
add_generic_nfs_storage_domain
00:18:22.091 [basic_suit_el7]     =
add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, =
mount_path, sd_format, sd_type, nfs_version)
00:18:22.091 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in =
add_generic_nfs_storage_domain_4
00:18:22.091 [basic_suit_el7]     host=3D_random_host_from_dc_4(api, =
DC_NAME),
00:18:22.091 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in =
_random_host_from_dc_4
00:18:22.091 [basic_suit_el7]     return =
random.choice(_hosts_in_dc_4(api, dc_name))
00:18:22.091 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in =
_hosts_in_dc_4
00:18:22.091 [basic_suit_el7]     raise RuntimeError('Could not find =
hosts that are up in DC %s' % dc_name)
00:18:22.092 [basic_suit_el7] RuntimeError: Could not find hosts that =
are up in DC test-dc
00:18:22.092 [basic_suit_el7] Error while running thread
00:18:22.092 [basic_suit_el7] Traceback (most recent call last):
00:18:22.092 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in =
_ret_via_queue
00:18:22.092 [basic_suit_el7]     queue.put({'return': func()})
00:18:22.092 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in =
add_iso_storage_domain
00:18:22.092 [basic_suit_el7]     =
add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, =
SD_ISO_PATH, sd_format=3D'v1', sd_type=3D'iso', nfs_version=3D'v3')
00:18:22.092 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in =
add_generic_nfs_storage_domain
00:18:22.092 [basic_suit_el7]     =
add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, =
mount_path, sd_format, sd_type, nfs_version)
00:18:22.093 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in =
add_generic_nfs_storage_domain_4
00:18:22.093 [basic_suit_el7]     host=3D_random_host_from_dc_4(api, =
DC_NAME),
00:18:22.093 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in =
_random_host_from_dc_4
00:18:22.093 [basic_suit_el7]     return =
random.choice(_hosts_in_dc_4(api, dc_name))
00:18:22.093 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in =
_hosts_in_dc_4
00:18:22.093 [basic_suit_el7]     raise RuntimeError('Could not find =
hosts that are up in DC %s' % dc_name)
00:18:22.093 [basic_suit_el7] RuntimeError: Could not find hosts that =
are up in DC test-dc
00:18:22.094 [basic_suit_el7] Error while running thread
00:18:22.094 [basic_suit_el7] Traceback (most recent call last):
00:18:22.094 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in =
_ret_via_queue
00:18:22.094 [basic_suit_el7]     queue.put({'return': func()})
00:18:22.094 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in =
add_templates_storage_domain
00:18:22.094 [basic_suit_el7]     =
add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, =
SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format=3D'v1', =
sd_type=3D'export', nfs_version=3D'v4_1')
00:18:22.094 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in =
add_generic_nfs_storage_domain
00:18:22.094 [basic_suit_el7]     =
add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, =
mount_path, sd_format, sd_type, nfs_version)
00:18:22.094 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in =
add_generic_nfs_storage_domain_4
00:18:22.095 [basic_suit_el7]     host=3D_random_host_from_dc_4(api, =
DC_NAME),
00:18:22.095 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in =
_random_host_from_dc_4
00:18:22.095 [basic_suit_el7]     return =
random.choice(_hosts_in_dc_4(api, dc_name))
00:18:22.095 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in =
_hosts_in_dc_4
00:18:22.095 [basic_suit_el7]     raise RuntimeError('Could not find =
hosts that are up in DC %s' % dc_name)
00:18:22.095 [basic_suit_el7] RuntimeError: Could not find hosts that =
are up in DC test-dc
00:18:22.095 [basic_suit_el7] Error while running thread
00:18:22.095 [basic_suit_el7] Traceback (most recent call last):
00:18:22.096 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in =
_ret_via_queue
00:18:22.096 [basic_suit_el7]     queue.put({'return': func()})
00:18:22.096 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in =
add_second_nfs_storage_domain
00:18:22.096 [basic_suit_el7]     SD_NFS_HOST_NAME, =
SD_SECOND_NFS_PATH)
00:18:22.096 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in =
add_generic_nfs_storage_domain
00:18:22.096 [basic_suit_el7]     =
add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, =
mount_path, sd_format, sd_type, nfs_version)
00:18:22.096 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in =
add_generic_nfs_storage_domain_4
00:18:22.096 [basic_suit_el7]     host=3D_random_host_from_dc_4(api, =
DC_NAME),
00:18:22.096 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in =
_random_host_from_dc_4
00:18:22.097 [basic_suit_el7]     return =
random.choice(_hosts_in_dc_4(api, dc_name))
00:18:22.097 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in =
_hosts_in_dc_4
00:18:22.097 [basic_suit_el7]     raise RuntimeError('Could not find =
hosts that are up in DC %s' % dc_name)
00:18:22.097 [basic_suit_el7] RuntimeError: Could not find hosts that =
are up in DC test-dc
00:18:22.097 [basic_suit_el7] Error while running thread
00:18:22.097 [basic_suit_el7] Traceback (most recent call last):
00:18:22.097 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in =
_ret_via_queue
00:18:22.098 [basic_suit_el7]     queue.put({'return': func()})
00:18:22.098 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in =
import_template_from_glance
00:18:22.098 [basic_suit_el7]     generic_import_from_glance(api, =
image_name=3DCIRROS_IMAGE_NAME, image_ext=3D'_glance_template', =
as_template=3DTrue)
00:18:22.098 [basic_suit_el7]   File =
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-=
tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in =
generic_import_from_glance
00:18:22.098 [basic_suit_el7]     =
target_image.import_image(import_action)
00:18:22.098 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", =
...
00:18:22.098 [basic_suit_el7]     =
...
00:18:22.098 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", =
...
00:18:22.098 [basic_suit_el7]     =
...
00:18:22.099 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
...
00:18:22.099 [basic_suit_el7]     persistent_auth)
00:18:22.099 [basic_suit_el7]   File =
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.=
this one started in =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/ =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/>

there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what =
seems to be mismatched
Piotr/Sandro, are all artifacts published correctly?

Thanks,
michal

oll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:]
line 26017, in import_image
headers=3D{"Correlation-Id":correlation_id}
line 122, in request
persistent_auth=3Dself.__persistent_auth
py", line 79, in do_request
py", line 162, in __do_request
...
00:18:22.099 [basic_suit_el7]     raise =
errors.RequestError(response_code, response_reason, response_body)
00:18:22.099 [basic_suit_el7] RequestError:=20
00:18:22.099 [basic_suit_el7] status: 400
00:18:22.099 [basic_suit_el7] reason: Bad Request
00:18:22.099 [basic_suit_el7] detail: Cannot import Virtual Disk: =
Storage Domain cannot be accessed.
00:18:22.099 [basic_suit_el7] -Please check that at least one Host is =
operational and Data Center state is up.
=20
--=20
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat=C2=A0EMEA <https://www.redhat.com/>
 <https://red.ht/sig>=09
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________
Devel mailing list
Devel@ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
=20
=20
_______________________________________________
Devel mailing list
Devel@ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
=20
=20
=20
--=20
Barak Korren
bkorren@redhat.com <mailto:bkorren@redhat.com>
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/>
_______________________________________________
Devel mailing list
Devel@ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
=20
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div =
class=3D"">On 12 Apr 2017, at 08:37, Yaniv Kaul <<a =
href=3D"mailto:ykaul@redhat.com" class=3D"">ykaul@redhat.com</a>> =
wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div =
dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On Wed, Apr 12, 2017 at 9:24 AM, =
Barak Korren <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:bkorren@redhat.com" target=3D"_blank" =
class=3D"">bkorren@redhat.com</a>></span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
dir=3D"ltr" class=3D""><div class=3D"">This seems to have been the =
Secodary SD issue I reported on Sunday:<br class=3D""><a =
href=3D"http://lists.ovirt.org/pipermail/devel/2017-April/030139.html" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">pipermail/devel/2017-April/<wbr class=3D"">030139.html</a><br =
class=3D""><br class=3D""></div>That was up until build #6307 where =
AddHost started failing:<br class=3D""><a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6=
307" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/<wbr =
class=3D"">test-repo_ovirt_experimental_<wbr =
class=3D"">master/6307</a></div></blockquote><div class=3D""><br =
class=3D""></div><div class=3D"">That's an entirely different =
game:</div><div class=3D""><b class=3D"">2017-04-11 11:24:36,783-04 =
ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] =
(org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: =
java.lang.NoClassDefFoundError: =
org/apache/commons/lang/StringUtils</b></div><div class=3D"">2017-04-11 =
11:24:36,783-04 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] =
(org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: =
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: =
</div></div></div></div></div></blockquote><div><br class=3D""></div>this =
one started in <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6=
307/" =
class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_maste=
r/6307/</a></div><div><br class=3D""></div><div>there are 2 patches =
linkded, one is in vdsm-jsonrpc-java, which is what seems to be =
mismatched</div><div>Piotr/Sandro, are all artifacts published =
correctly?</div><div><br =
class=3D""></div><div>Thanks,</div><div>michal</div><div><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><div =
class=3D"gmail_quote"><div =
class=3D"">org/apache/commons/lang/StringUtils</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at java.util.concurrent.FutureTask.report(FutureTask.java:122) =
[rt.jar:1.8.0_121]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at java.util.concurrent.FutureTask.get(FutureTask.java:206) =
[rt.jar:1.8.0_121]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo=
mmand.java:69) [vdsbroker.jar:]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos=
tSetupNetworkPoller.java:56) [bll.jar:]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet=
upNetworkPoller.java:41) [bll.jar:]</div><div class=3D""><br =
class=3D""></div><div class=3D"">...</div><div class=3D"">Caused by: =
java.lang.NoClassDefFoundError: =
org/apache/commons/lang/StringUtils</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationI=
d(Message.java:75) [vdsm-jsonrpc-java-client.jar:]</div><div =
class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">=
	</span>at =
org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SS=
LStompClient.java:85) [vdsm-jsonrpc-java-client.jar:]</div><div =
class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">=
	</span>at =
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) =
[vdsm-jsonrpc-java-client.jar:]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.j=
ava:91) [vdsbroker.jar:]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundP=
oll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:]</div><div =
class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">=
	</span>at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca=
ll(JsonRpcVdsServer.java:484) [vdsbroker.jar:]</div><div class=3D""><span =
class=3D"gmail-Apple-tab-span" style=3D"white-space:pre">	=
</span>at =
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.ca=
ll(JsonRpcVdsServer.java:473) [vdsbroker.jar:]</div><div class=3D""><br =
class=3D""></div><div class=3D""> </div><blockquote =
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px =
solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D""><br =
class=3D""><br class=3D""><div class=3D""><div class=3D""><div =
class=3D"gmail_extra"><div class=3D""><div class=3D"gmail-h5"><br =
class=3D""><div class=3D"gmail_quote">On 11 April 2017 at 18:12, Piotr =
Kliczewski <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:piotr.kliczewski@gmail.com" target=3D"_blank" =
class=3D"">piotr.kliczewski@gmail.com</a>></span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
dir=3D"ltr" class=3D"">I looks like SD activate failed on the engine =
side and the spm connection was gently closed by the engine:<div =
class=3D""><br class=3D""></div><div class=3D""><div class=3D"">2017-04-11=
 09:10:24,807-04 INFO  [<a =
href=3D"http://org.ovirt.engine.core.bll.st/" =
class=3D"">org.ovirt.engine.core.bll.st</a><wbr =
class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand] =
(default task-29) [2b66a42b] ActivateStorage Domain. After Connect all =
hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017</div><div =
class=3D"">2017-04-11 09:10:24,842-04 ERROR =
[org.ovirt.engine.core.bll.sto<wbr =
class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] =
(org.ovirt.thread.pool-7-threa<wbr class=3D"">d-29) [2b66a42b] Could not =
connect vds 'lago-basic-suite-master-host0<wbr class=3D"">' to pool =
'test-dc' - moving host to non-operational: null</div><div =
class=3D"">2017-04-11 09:10:24,842-04 ERROR =
[org.ovirt.engine.core.bll.sto<wbr =
class=3D"">rage.pool.RefreshPoolSingleAsy<wbr class=3D"">ncOperation] =
(org.ovirt.thread.pool-7-threa<wbr class=3D"">d-28) [2b66a42b] Could not =
connect vds 'lago-basic-suite-master-host1<wbr class=3D"">' to pool =
'test-dc' - moving host to non-operational: null</div><div =
class=3D"">2017-04-11 09:10:24,904-04 INFO  [<a =
href=3D"http://org.ovirt.engine.core.bll.se/" =
class=3D"">org.ovirt.engine.core.bll.Se</a><wbr =
class=3D"">tNonOperationalVdsCommand] (default task-29) [2bc83b1c] =
Running command: SetNonOperationalVdsCommand internal: true. Entities =
affected :  ID: e1b611f2-67fb-4acb-8261-071331<wbr class=3D"">34e0e7 =
Type: VDS</div><div class=3D"">2017-04-11 09:10:24,909-04 INFO =
 [org.ovirt.engine.core.vdsbro<wbr =
class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] =
START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D =
lago-basic-suite-master-host0, SetVdsStatusVDSCommandParamete<wbr =
class=3D"">rs:{runAsync=3D'true', hostId=3D'e1b611f2-67fb-4acb-826<wbr =
class=3D"">1-07133134e0e7', status=3D'NonOperational', =
nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', =
stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: =
5c45239</div><div class=3D"">2017-04-11 09:10:24,919-04 INFO =
 [org.ovirt.engine.core.vdsbro<wbr =
class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] =
FINISH, SetVdsStatusVDSCommand, log id: 5c45239</div><div =
class=3D"">2017-04-11 09:10:24,922-04 DEBUG =
[org.ovirt.engine.core.dal.dbb<wbr =
class=3D"">roker.PostgresDbEngineDialect$<wbr =
class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled =
stored procedure. Call string is [{call =
getvmsrunningonvds(?)}]</div><div class=3D"">2017-04-11 09:10:24,922-04 =
DEBUG [org.ovirt.engine.core.dal.dbb<wbr =
class=3D"">roker.PostgresDbEngineDialect$<wbr =
class=3D"">PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall =
for procedure [GetVmsRunningOnVds] compiled</div><div =
class=3D"">2017-04-11 09:10:24,985-04 WARN =
 [org.ovirt.engine.core.dal.db<wbr =
class=3D"">broker.auditloghandling.AuditL<wbr class=3D"">ogDirector] =
(default task-29) [2bc83b1c] EVENT_ID: =
VDS_SET_NONOPERATIONAL_DOMAIN(<wbr class=3D"">522), Correlation ID: =
2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e98269<wbr class=3D"">44b9a0, =
Call Stack: null, Custom Event ID: -1, Message: Host =
lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi =
attached to the Data Center test-dc. Setting Host state to =
Non-Operational.</div><div class=3D"">2017-04-11 09:10:24,985-04 DEBUG =
[org.ovirt.engine.core.utils.t<wbr class=3D"">imer.FixedDelayJobListener] =
(DefaultQuartzScheduler4) [] Rescheduling <a href=3D"http://DEFAULT.org" =
class=3D"">DEFAULT.org</a>.ovirt.engine.core.<wbr =
class=3D"">vdsbroker.VdsManager.onTimer#-<wbr =
class=3D"">9223372036854775741 as there is no unfired trigger.</div><div =
class=3D"">2017-04-11 09:10:25,036-04 INFO  [<a =
href=3D"http://org.ovirt.engine.core.bll.se/" =
class=3D"">org.ovirt.engine.core.bll.Se</a><wbr =
class=3D"">tNonOperationalVdsCommand] (default task-29) [2ffa5981] =
Running command: SetNonOperationalVdsCommand internal: true. Entities =
affected :  ID: 87a5f691-ea5c-4692-a4ab-52dad0<wbr class=3D"">dafe34 =
Type: VDS</div><div class=3D"">2017-04-11 09:10:25,039-04 INFO =
 [org.ovirt.engine.core.vdsbro<wbr =
class=3D"">ker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] =
START, SetVdsStatusVDSCommand(HostNam<wbr class=3D"">e =3D =
lago-basic-suite-master-host1, SetVdsStatusVDSCommandParamete<wbr =
class=3D"">rs:{runAsync=3D'true', hostId=3D'87a5f691-ea5c-4692-a4a<wbr =
class=3D"">b-52dad0dafe34', status=3D'NonOperational', =
nonOperationalReason=3D'STORAGE_<wbr class=3D"">DOMAIN_UNREACHABLE', =
stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: =
706ffc3b</div></div><div class=3D""><br class=3D""></div><div =
class=3D"">.....</div><div class=3D""><br class=3D""></div><div =
class=3D""><div class=3D"">2017-04-11 09:10:25,366-04 INFO =
 [org.ovirt.engine.core.vdsbro<wbr =
class=3D"">ker.vdsbroker.SpmStopVDSComman<wbr class=3D"">d] (default =
task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: =
62f82cfb</div><div class=3D"">2017-04-11 09:10:25,366-04 DEBUG =
[org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.reactors.stomp.impl.Message] (default task-29) [2ffa5981] =
UNSUBSCRIBE</div><div class=3D"">id:4cd59c67-1b57-4456-9a16-52a<wbr =
class=3D"">72ac6f689</div><div class=3D""><br class=3D""></div><div =
class=3D"">\00</div><div class=3D"">2017-04-11 09:10:25,367-04 DEBUG =
[org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.reactors.stomp.StompCommonCli<wbr class=3D"">ent] (default =
task-29) [2ffa5981] Message sent: UNSUBSCRIBE</div></div><div =
class=3D""><br class=3D""></div><div class=3D""><br =
class=3D""></div></div><div class=3D"gmail_extra"><br class=3D""><div =
class=3D"gmail_quote"><div class=3D""><div =
class=3D"gmail-m_5006786018485841575gmail-h5">On Tue, Apr 11, 2017 at =
4:49 PM, Sandro Bonazzola <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:sbonazzo@redhat.com" target=3D"_blank" =
class=3D"">sbonazzo@redhat.com</a>></span> wrote:<br =
class=3D""></div></div><blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex"><div class=3D""><div =
class=3D"gmail-m_5006786018485841575gmail-h5"><div dir=3D"ltr" =
class=3D"">Link to Job: <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6=
303" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/job/t<wbr =
class=3D"">est-repo_ovirt_experimental_ma<wbr class=3D"">ster/6303</a><pre=
 =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-console-output" =
style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-=
bottom:0px;color:rgb(51,51,51);font-size:14px"><span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D""><br =
class=3D""></b></span></pre><pre =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-console-output" =
style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-=
bottom:0px;color:rgb(51,51,51);font-size:14px"><span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">OST is failing in the last 50 =
builds</b></span></pre><pre =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-console-output" =
style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-=
bottom:0px;color:rgb(51,51,51);font-size:14px"><span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D""><br =
class=3D""></b></span></pre><pre =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-console-output" =
style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-=
bottom:0px;color:rgb(51,51,51);font-size:14px"><br class=3D""></pre><pre =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-console-output" =
style=3D"box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-=
bottom:0px;color:rgb(51,51,51);font-size:14px"><span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(0,205,205)" class=3D"">  # =
add_secondary_storage_domains: </span>
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error =
while running thread
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7] Traceback (most recent call last):
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7]     queue.put({'return': func()})
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 427, in add_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.090</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, =
nfs_version=3D'v4_2')
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 441, in add_generic_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, =
sd_type, nfs_version)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]     host=3D_random_host_from_dc_4(ap<wbr =
class=3D"">i, DC_NAME),
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 106, in _random_host_from_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]     return random.choice(_hosts_in_dc_4(a<wbr =
class=3D"">pi, dc_name))
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 100, in _hosts_in_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.091</b> =
</span>[basic_suit_el7]     raise RuntimeError('Could not find hosts =
that are up in DC %s' % dc_name)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7] RuntimeError: Could not find hosts that are up =
in DC test-dc</span>
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error =
while running thread
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7] Traceback (most recent call last):
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]     queue.put({'return': func()})
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 583, in add_iso_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, =
sd_format=3D'v1', sd_type=3D'iso', nfs_version=3D'v3')
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 441, in add_generic_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.092</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, =
sd_type, nfs_version)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]     host=3D_random_host_from_dc_4(ap<wbr =
class=3D"">i, DC_NAME),
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 106, in _random_host_from_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]     return random.choice(_hosts_in_dc_4(a<wbr =
class=3D"">pi, dc_name))
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 100, in _hosts_in_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7]     raise RuntimeError('Could not find hosts =
that are up in DC %s' % dc_name)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.093</b> =
</span>[basic_suit_el7] RuntimeError: Could not find hosts that are up =
in DC test-dc</span>
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error =
while running thread
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7] Traceback (most recent call last):
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]     queue.put({'return': func()})
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 587, in add_templates_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, =
SD_TEMPLATES_PATH, sd_format=3D'v1', sd_type=3D'export', =
nfs_version=3D'v4_1')
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 441, in add_generic_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, =
sd_type, nfs_version)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.094</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7]     host=3D_random_host_from_dc_4(ap<wbr =
class=3D"">i, DC_NAME),
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 106, in _random_host_from_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7]     return random.choice(_hosts_in_dc_4(a<wbr =
class=3D"">pi, dc_name))
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 100, in _hosts_in_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7]     raise RuntimeError('Could not find hosts =
that are up in DC %s' % dc_name)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7] RuntimeError: Could not find hosts that are up =
in DC test-dc</span>
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error =
while running thread
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.095</b> =
</span>[basic_suit_el7] Traceback (most recent call last):
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]     queue.put({'return': func()})
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 436, in add_second_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]     SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 441, in add_generic_nfs_storage_domain
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]     add_generic_nfs_storage_domain<wbr =
class=3D"">_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, =
sd_type, nfs_version)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 490, in add_generic_nfs_storage_domain<wbr class=3D"">_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]     host=3D_random_host_from_dc_4(ap<wbr =
class=3D"">i, DC_NAME),
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.096</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 106, in _random_host_from_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7]     return random.choice(_hosts_in_dc_4(a<wbr =
class=3D"">pi, dc_name))
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 100, in _hosts_in_dc_4
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7]     raise RuntimeError('Could not find hosts =
that are up in DC %s' % dc_name)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7] RuntimeError: Could not find hosts that are up =
in DC test-dc</span>
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7] <span =
style=3D"box-sizing:border-box;color:rgb(205,0,0)" class=3D"">Error =
while running thread
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7] Traceback (most recent call last):
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.097</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/lago/utils.py", line 57, in _ret_via_queue
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]     queue.put({'return': func()})
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 803, in import_template_from_glance
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]     generic_import_from_glance(api<wbr =
class=3D"">, image_name=3DCIRROS_IMAGE_NAME, =
image_ext=3D'_glance_template', as_template=3DTrue)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]   File "/home/jenkins/workspace/test-<wbr =
class=3D"">repo_ovirt_experimental_master<wbr =
class=3D"">/ovirt-system-tests/basic-suit<wbr =
class=3D"">e-master/test-scenarios/002_<wbr class=3D"">bootstrap.py", =
line 637, in generic_import_from_glance
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]     target_image.import_image(impo<wbr =
class=3D"">rt_action)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/ovirtsdk/infrastructure/br<wbr class=3D"">okers.py", line =
26017, in import_image
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]     headers=3D{"Correlation-Id":corr<wbr =
class=3D"">elation_id}
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/ovirtsdk/infrastructure/pr<wbr class=3D"">oxy.py", line =
122, in request
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.098</b> =
</span>[basic_suit_el7]     persistent_auth=3Dself.__persist<wbr =
class=3D"">ent_auth
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/ovirtsdk/infrastructure/co<wbr =
class=3D"">nnectionspool.py", line 79, in do_request
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7]     persistent_auth)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7]   File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/ovirtsdk/infrastructure/co<wbr =
class=3D"">nnectionspool.py", line 162, in __do_request
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7]     raise errors.RequestError(response_c<wbr =
class=3D"">ode, response_reason, response_body)
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7] RequestError:=20
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7] status: 400
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7] reason: Bad Request
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7] detail: Cannot import Virtual Disk: Storage =
Domain cannot be accessed.
<span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail-timestamp" style=3D"box-sizing:border-box"><b =
style=3D"box-sizing:border-box" class=3D"">00:18:22.099</b> =
</span>[basic_suit_el7] -Please check that at least one Host is =
operational and Data Center state is up.</span></pre><span =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735HOEnZb"><fo=
nt color=3D"#888888" class=3D""><div class=3D""><br class=3D""></div>-- =
<br class=3D""><div =
class=3D"gmail-m_5006786018485841575gmail-m_3046267183852181735m_525458228=
0271388056gmail_signature"><div dir=3D"ltr" class=3D""><div =
class=3D""><div dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div =
dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div =
style=3D"font-family: overpass, sans-serif; font-weight: bold; margin: =
0px; padding: 0px; font-size: 14px; text-transform: uppercase;" =
class=3D""><span class=3D"">SANDRO</span> <span =
class=3D"">BONAZZOLA</span></div><p style=3D"font-family: overpass, =
sans-serif; font-size: 10px; margin: 0px 0px 4px; text-transform: =
uppercase;" class=3D""><span class=3D"">ASSOCIATE MANAGER, SOFTWARE =
ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><div =
style=3D"font-family: overpass, sans-serif; margin: 0px; font-size: =
10px; color: rgb(153, 153, 153);" class=3D""><a =
href=3D"https://www.redhat.com/" style=3D"color:rgb(0,136,206);margin:0px"=
 target=3D"_blank" class=3D"">Red Hat <span =
class=3D"">EMEA</span></a></div><table style=3D"font-family: overpass, =
sans-serif; font-size: inherit;" border=3D"0" class=3D""><tbody =
class=3D""><tr class=3D""><td width=3D"100px" class=3D""><a =
href=3D"https://red.ht/sig" target=3D"_blank" class=3D""><img =
src=3D"https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red=
-hat-black.png" width=3D"90" height=3D"auto" class=3D""></a></td><td =
style=3D"font-size:10px" class=3D""><div class=3D""><a =
href=3D"https://redhat.com/trusted" =
style=3D"color:rgb(204,0,0);font-weight:bold" target=3D"_blank" =
class=3D"">TRIED. TESTED. =
TRUSTED.</a></div></td></tr></tbody></table></div></div></div></div></div>=
</div></div>
</font></span></div>
<br class=3D""></div></div>______________________________<wbr =
class=3D"">_________________<br class=3D"">
Devel mailing list<br class=3D"">
<a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
<a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br =
class=3D""></div>
<br class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">
Devel mailing list<br class=3D"">
<a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
<a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D""></blockquote></div><br =
class=3D""><br clear=3D"all" class=3D""><br class=3D""></div></div><span =
class=3D"gmail-HOEnZb"><font color=3D"#888888" class=3D"">-- <br =
class=3D""><div class=3D"gmail-m_5006786018485841575gmail_signature">Barak=
 Korren<br class=3D""><a href=3D"mailto:bkorren@redhat.com" =
target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">RHCE, =
RHCi, RHV-DevOps Team<br class=3D""><a =
href=3D"https://ifireball.wordpress.com/" target=3D"_blank" =
class=3D"">https://ifireball.wordpress.<wbr class=3D"">com/</a></div>
</font></span></div></div></div></div>
<br class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">
Devel mailing list<br class=3D"">
<a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br =
class=3D"">
<a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br=
 class=3D""></div></div>
_______________________________________________<br class=3D"">Devel =
mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org" =
class=3D"">Devel@ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote=
...
</div><br class=3D""></body></html>=
--Apple-Mail=_A73F9645-3BD9-4104-A39D-D19AEC539032--

Re: [ovirt-devel] [OST Failure Report][oVirt master][2017-04-11] add_secondary_storage_domains

Michal Skrivanek