<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>This seems to have been the Secodary SD issue I reported on Sunday:<br><a href="http://lists.ovirt.org/pipermail/devel/2017-April/030139.html" target="_blank">http://lists.ovirt.org/<wbr>pipermail/devel/2017-April/<wbr>030139.html</a><br><br></div>That was up until build #6307 where AddHost started failing:<br><a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/6307</a></div></blockquote><div><br></div><div>That's an entirely different game:</div><div><b>2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils</b></div><div>2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.8.0_121]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_121]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) [vdsbroker.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) [bll.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) [bll.jar:]</div><div><br></div><div>...</div><div>Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) [vdsbroker.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:]</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div><div><div class="gmail_extra"><div><div class="gmail-h5"><br><div class="gmail_quote">On 11 April 2017 at 18:12, Piotr Kliczewski <span dir="ltr"><<a href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:<div><br></div><div><div>2017-04-11 09:10:24,807-04 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017</div><div>2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto<wbr>rage.pool.RefreshPoolSingleAsy<wbr>ncOperation] (org.ovirt.thread.pool-7-threa<wbr>d-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0<wbr>' to pool 'test-dc' - moving host to non-operational: null</div><div>2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto<wbr>rage.pool.RefreshPoolSingleAsy<wbr>ncOperation] (org.ovirt.thread.pool-7-threa<wbr>d-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1<wbr>' to pool 'test-dc' - moving host to non-operational: null</div><div>2017-04-11 09:10:24,904-04 INFO [<a href="http://org.ovirt.engine.core.bll.Se">org.ovirt.engine.core.bll.Se</a><wbr>tNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-071331<wbr>34e0e7 Type: VDS</div><div>2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostNam<wbr>e = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParamete<wbr>rs:{runAsync='true', hostId='e1b611f2-67fb-4acb-826<wbr>1-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_<wbr>DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239</div><div>2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239</div><div>2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb<wbr>roker.PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}]</div><div>2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb<wbr>roker.PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled</div><div>2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.db<wbr>broker.auditloghandling.AuditL<wbr>ogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(<wbr>522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e98269<wbr>44b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational.</div><div>2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.t<wbr>imer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core.<wbr>vdsbroker.VdsManager.onTimer#-<wbr>9223372036854775741 as there is no unfired trigger.</div><div>2017-04-11 09:10:25,036-04 INFO [<a href="http://org.ovirt.engine.core.bll.Se">org.ovirt.engine.core.bll.Se</a><wbr>tNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0<wbr>dafe34 Type: VDS</div><div>2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostNam<wbr>e = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParamete<wbr>rs:{runAsync='true', hostId='87a5f691-ea5c-4692-a4a<wbr>b-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_<wbr>DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b</div></div><div><br></div><div>.....</div><div><br></div><div><div>2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.vdsbroker.SpmStopVDSComman<wbr>d] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb</div><div>2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE</div><div>id:4cd59c67-1b57-4456-9a16-52a<wbr>72ac6f689</div><div><br></div><div>\00</div><div>2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.stomp.StompCommonCli<wbr>ent] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_5006786018485841575gmail-h5">On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <span dir="ltr"><<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_5006786018485841575gmail-h5"><div dir="ltr">Link to Job: <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/6303</a><pre class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box"><br></b></span></pre><pre class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">OST is failing in the last 50 builds</b></span></pre><pre class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box"><br></b></span></pre><pre class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><br></pre><pre class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(0,205,205)"> # add_secondary_storage_domains: </span>
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] Traceback (most recent call last):
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] queue.put({'return': func()})
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 427, in add_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.090</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 441, in add_generic_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 490, in add_generic_nfs_storage_domain<wbr>_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] host=_random_host_from_dc_4(ap<wbr>i, DC_NAME),
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 106, in _random_host_from_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr>pi, dc_name))
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 100, in _hosts_in_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.091</b> </span>[basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc</span>
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] Traceback (most recent call last):
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] queue.put({'return': func()})
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 583, in add_iso_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 441, in add_generic_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.092</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 490, in add_generic_nfs_storage_domain<wbr>_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] host=_random_host_from_dc_4(ap<wbr>i, DC_NAME),
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 106, in _random_host_from_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr>pi, dc_name))
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 100, in _hosts_in_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.093</b> </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc</span>
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] Traceback (most recent call last):
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] queue.put({'return': func()})
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 587, in add_templates_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 441, in add_generic_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.094</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 490, in add_generic_nfs_storage_domain<wbr>_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] host=_random_host_from_dc_4(ap<wbr>i, DC_NAME),
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 106, in _random_host_from_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr>pi, dc_name))
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 100, in _hosts_in_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc</span>
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.095</b> </span>[basic_suit_el7] Traceback (most recent call last):
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] queue.put({'return': func()})
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 436, in add_second_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 441, in add_generic_nfs_storage_domain
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] add_generic_nfs_storage_domain<wbr>_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 490, in add_generic_nfs_storage_domain<wbr>_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] host=_random_host_from_dc_4(ap<wbr>i, DC_NAME),
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.096</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 106, in _random_host_from_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] return random.choice(_hosts_in_dc_4(a<wbr>pi, dc_name))
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 100, in _hosts_in_dc_4
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc</span>
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] <span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] Traceback (most recent call last):
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.097</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] queue.put({'return': func()})
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 803, in import_template_from_glance
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] generic_import_from_glance(api<wbr>, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] File "/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_master<wbr>/ovirt-system-tests/basic-suit<wbr>e-master/test-scenarios/002_<wbr>bootstrap.py", line 637, in generic_import_from_glance
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] target_image.import_image(impo<wbr>rt_action)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtsdk/infrastructure/br<wbr>okers.py", line 26017, in import_image
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] headers={"Correlation-Id":corr<wbr>elation_id}
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtsdk/infrastructure/pr<wbr>oxy.py", line 122, in request
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.098</b> </span>[basic_suit_el7] persistent_auth=self.__persist<wbr>ent_auth
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtsdk/infrastructure/co<wbr>nnectionspool.py", line 79, in do_request
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] persistent_auth)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtsdk/infrastructure/co<wbr>nnectionspool.py", line 162, in __do_request
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] raise errors.RequestError(response_c<wbr>ode, response_reason, response_body)
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] RequestError:
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] status: 400
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] reason: Bad Request
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.
<span class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">00:18:22.099</b> </span>[basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.</span></pre><span class="gmail-m_5006786018485841575gmail-m_3046267183852181735HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="gmail-m_5006786018485841575gmail-m_3046267183852181735m_5254582280271388056gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table></div></div></div></div></div></div></div>
</font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br><br clear="all"><br></div></div><span class="gmail-HOEnZb"><font color="#888888">-- <br><div class="gmail-m_5006786018485841575gmail_signature">Barak Korren<br><a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a><br>RHCE, RHCi, RHV-DevOps Team<br><a href="https://ifireball.wordpress.com/" target="_blank">https://ifireball.wordpress.<wbr>com/</a></div>
</font></span></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br></div></div>