<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Hi all,</div><div class="gmail_default" style="font-family:monospace,monospace">Happened again <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6017/console">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6017/console</a><br></div><div class="gmail_default"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 22, 2017 at 4:48 PM, Gil Shinar <span dir="ltr">&lt;<a href="mailto:gshinar@redhat.com" target="_blank">gshinar@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Looks like the actuall error is in VDSM log:<div><pre style="color:rgb(0,0,0)">2017-03-22 11:01:22,410-0400 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task=&#39;179424c1-b23d-40aa-<wbr>a091-3c197971f420&#39;) Unexpected error (task:871)
Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/task.py&quot;<wbr>, line 878, in _run
    return fn(*args, **kargs)
  File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/logUtils.py&quot;, line 52, in wrapper
    res = f(*args, **kwargs)
  File &quot;/usr/share/vdsm/storage/hsm.<wbr>py&quot;, line 3060, in getVolumeInfo
    volUUID=volUUID).getInfo()
  File &quot;/usr/share/vdsm/storage/sd.<wbr>py&quot;, line 748, in produceVolume
    volUUID)
  File &quot;/usr/share/vdsm/storage/<wbr>fileVolume.py&quot;, line 361, in __init__
    manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
  File &quot;/usr/share/vdsm/storage/<wbr>fileVolume.py&quot;, line 61, in __init__
    volUUID)
  File &quot;/usr/share/vdsm/storage/<wbr>volume.py&quot;, line 86, in __init__
    self.validate()
  File &quot;/usr/share/vdsm/storage/<wbr>volume.py&quot;, line 109, in validate
    self.validateVolumePath()
  File &quot;/usr/share/vdsm/storage/<wbr>fileVolume.py&quot;, line 120, in validateVolumePath
    raise se.VolumeDoesNotExist(self.<wbr>volUUID)
VolumeDoesNotExist: Volume does not exist: (u&#39;5133a146-d4bc-49a1-8358-<wbr>7e14522ffc4b&#39;,)
2017-03-22 11:01:22,413-0400 INFO  (jsonrpc/7) [storage.TaskManager.Task] (Task=&#39;179424c1-b23d-40aa-<wbr>a091-3c197971f420&#39;) aborting: Task is aborted: &#39;Volume does not exist&#39; - code 201 (task:1176)
2017-03-22 11:01:22,414-0400 ERROR (jsonrpc/7) [storage.Dispatcher] {&#39;status&#39;: {&#39;message&#39;: &quot;Volume does not exist: (u&#39;5133a146-d4bc-49a1-8358-<wbr>7e14522ffc4b&#39;,)&quot;, &#39;code&#39;: 201}} (dispatcher:78)</pre></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 22, 2017 at 10:57 AM, Gil Shinar <span dir="ltr">&lt;<a href="mailto:gshinar@redhat.com" target="_blank">gshinar@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It has happened again:<div><a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1033/artifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-suite-4.1/post-004_basic_sanity.py/" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_4.<wbr>1/1033/artifact/exported-artif<wbr>acts/basic-suit-4.1-el7/test_<wbr>logs/basic-suite-4.1/post-004_<wbr>basic_sanity.py/</a><br></div><div><br></div><div>Any news concerning this issue?</div><div><br></div><div>Thanks</div><span class="m_-2470281676985459154HOEnZb"><font color="#888888"><div>Gil</div></font></span></div><div class="m_-2470281676985459154HOEnZb"><div class="m_-2470281676985459154h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <span dir="ltr">&lt;<a href="mailto:awels@redhat.com" target="_blank">awels@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:<br>
&gt; On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David &lt;<a href="mailto:sbendavi@redhat.com" target="_blank">sbendavi@redhat.com</a>&gt;<br>
&gt;<br>
&gt; wrote:<br>
&gt; &gt; Hi,<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Test failed: [ 004_basic_sanity.snapshots_mer<wbr>ge ]<br>
&gt; &gt;<br>
&gt; &gt; Link to suspected patches: N/A<br>
&gt; &gt;<br>
&gt; &gt; Link to Job:<br>
</span>&gt; &gt;    1. <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_4.<wbr>1/1007</a><br>
&gt; &gt;    2. <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_</a><br>
<div><div class="m_-2470281676985459154m_398850444981516307h5">&gt; &gt;    master/5918<br>
&gt;<br>
&gt; They look similar, but master has an additional failure (unrelated to the<br>
&gt; snap merge failure).<br>
&gt; Allon/Tal - can you have someone from storage look at the failures?<br>
&gt; Alexander - can you take a look at the below?<br>
&gt;<br>
&gt;<br>
&gt; 2017-03-20 09:30:10,283-04 DEBUG<br>
&gt; [org.ovirt.engine.core.dal.dbb<wbr>roker.PostgresDbEngineDialect$<wbr>PostgresSimpleJd<br>
&gt; bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure<br>
&gt; [GetAllMacPoolRangesByMacPoolI<wbr>d] compiled<br>
&gt; 2017-03-20 09:30:10,283-04 INFO  [org.ovirt.engine.core.bll.Tag<wbr>sDirector]<br>
&gt; (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector<br>
&gt; 2017-03-20 09:30:10,275-04 ERROR<br>
&gt; [org.ovirt.engine.ui.frontend.<wbr>server.dashboard.DashboardData<wbr>Servlet.CacheUpd<br>
&gt; ate.Utilization] (EE-ManagedThreadFactory-defau<wbr>lt-Thread-2) [] Could not<br>
&gt; update the Utilization Cache: Error while running SQL query:<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.DashboardDataE<wbr>xception: Error<br>
&gt; while running SQL query<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<a href="http://erver.dashboard.dao.BaseDao.ru" target="_blank"><wbr>erver.dashboard.dao.BaseDao.ru</a><wbr>nQuery(BaseDao.j<br>
&gt; ava:60) [frontend.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.dao.HostDwhDao<wbr>.getTotalCpuMemC<br>
&gt; ount(HostDwhDao.java:78) [frontend.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.HourlySummaryH<wbr>elper.getTotalCp<br>
&gt; uMemCount(HourlySummaryHelper.<wbr>java:43) [frontend.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.HourlySummaryH<wbr>elper.getCpuMemS<br>
&gt; ummary(HourlySummaryHelper.jav<wbr>a:21) [frontend.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.DashboardDataS<wbr>ervlet.lookupGlo<br>
&gt; balUtilization(DashboardDataSe<wbr>rvlet.java:287) [frontend.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.ui.frontend.s<wbr>erver.dashboard.DashboardDataS<wbr>ervlet.getDashbo<br>
&gt; ard(DashboardDataServlet.java:<wbr>261) [frontend.jar:]<br>
&gt;<br>
<br>
</div></div>So it looks like a communication failure with the DWH. But when I look at the<br>
associated logs, the error is not in there, nor do the time stamps match. Is<br>
this error from the same run or a previous one? I was trying to get more<br>
information than a partial stack trace.<br>
<br>
&gt; &gt;    1.<br>
&gt; &gt;<br>
&gt; &gt; Link to all logs:<br>
&gt; &gt;    1. <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_</a><br>
&gt; &gt;    4.1/1007/artifact/exported-art<wbr>ifacts/basic-suit-4.1-el7/<br>
&gt; &gt;    test_logs/basic-suite-4.1/<a href="http://post-004_basic_sanity.py/" rel="noreferrer" target="_blank">post<wbr>-004_basic_sanity.py/</a><br>
&gt; &gt;    &lt;<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/art" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_4<wbr>.1/1007/art</a><br>
&gt; &gt;    ifact/exported-artifacts/basic<wbr>-suit-4.1-el7/test_logs/basic-<wbr>suite-4.1/p<br>
&gt; &gt;    <a href="http://ost-004_basic_sanity.py/" rel="noreferrer" target="_blank">ost-004_basic_sanity.py/</a>&gt; 2.<br>
&gt; &gt;    <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_</a><br>
&gt; &gt;    master/5918/artifact/exported-<wbr>artifacts/basic-suit-master-<br>
&gt; &gt;    el7/test_logs/basic-suite-mast<wbr>er/<a href="http://post-004_basic_sanity.py/" rel="noreferrer" target="_blank">post-004_basic_sanity.py/</a><br>
&gt; &gt;    &lt;<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_m<wbr>aster/5918/</a><br>
&gt; &gt;    artifact/exported-artifacts/ba<wbr>sic-suit-master-el7/test_logs/<wbr>basic-suite<br>
&gt; &gt;    -master/<a href="http://post-004_basic_sanity.py/" rel="noreferrer" target="_blank">post-004_basic_sanity.<wbr>py/</a>&gt;&gt;<br>
<div class="m_-2470281676985459154m_398850444981516307HOEnZb"><div class="m_-2470281676985459154m_398850444981516307h5">&gt; &gt; Error snippet from the log:<br>
&gt; &gt;<br>
&gt; &gt; &lt;error&gt;<br>
&gt; &gt;<br>
&gt; &gt; ovirtlago.testlib: ERROR: Unhandled exception in &lt;function &lt;lambda&gt; at<br>
&gt; &gt; 0x446a758&gt;<br>
&gt; &gt;<br>
&gt; &gt; Traceback (most recent call last):<br>
&gt; &gt;   File &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py&quot;, line 217,<br>
&gt; &gt;<br>
&gt; &gt; in assert_equals_within<br>
&gt; &gt;<br>
&gt; &gt;     res = func()<br>
&gt; &gt;<br>
&gt; &gt;   File &quot;/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_4.1/<br>
&gt; &gt;<br>
&gt; &gt; ovirt-system-tests/basic-suite<wbr>-4.1/test-scenarios/004_basic_<wbr>sanity.py&quot;,<br>
&gt; &gt; line 449, in &lt;lambda&gt;<br>
&gt; &gt;<br>
&gt; &gt;     api.vms.get(VM0_NAME).disks.g<wbr>et(disk_name).status.state == &#39;ok&#39;<br>
&gt; &gt;<br>
&gt; &gt; AttributeError: &#39;NoneType&#39; object has no attribute &#39;state&#39;<br>
&gt; &gt; lago.utils: ERROR: Error while running thread<br>
&gt; &gt;<br>
&gt; &gt; Traceback (most recent call last):<br>
&gt; &gt;   File &quot;/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py&quot;, line 57, in<br>
&gt; &gt;<br>
&gt; &gt; _ret_via_queue<br>
&gt; &gt;<br>
&gt; &gt;     queue.put({&#39;return&#39;: func()})<br>
&gt; &gt;<br>
&gt; &gt;   File &quot;/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_4.1/<br>
&gt; &gt;<br>
&gt; &gt; ovirt-system-tests/basic-suite<wbr>-4.1/test-scenarios/004_basic_<wbr>sanity.py&quot;,<br>
&gt; &gt; line 448, in snapshot_live_merge<br>
&gt; &gt;<br>
&gt; &gt;     lambda:<br>
&gt; &gt;   File &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py&quot;, line 264,<br>
&gt; &gt;<br>
&gt; &gt; in assert_true_within_long<br>
&gt; &gt;<br>
&gt; &gt;     assert_equals_within_long(fun<wbr>c, True, allowed_exceptions)<br>
&gt; &gt;<br>
&gt; &gt;   File &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py&quot;, line 251,<br>
&gt; &gt;<br>
&gt; &gt; in assert_equals_within_long<br>
&gt; &gt;<br>
&gt; &gt;     func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exc<wbr>eptions<br>
&gt; &gt;<br>
&gt; &gt;   File &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py&quot;, line 217,<br>
&gt; &gt;<br>
&gt; &gt; in assert_equals_within<br>
&gt; &gt;<br>
&gt; &gt;     res = func()<br>
&gt; &gt;<br>
&gt; &gt;   File &quot;/home/jenkins/workspace/test-<wbr>repo_ovirt_experimental_4.1/<br>
&gt; &gt;<br>
&gt; &gt; ovirt-system-tests/basic-suite<wbr>-4.1/test-scenarios/004_basic_<wbr>sanity.py&quot;,<br>
&gt; &gt; line 449, in &lt;lambda&gt;<br>
&gt; &gt;<br>
&gt; &gt;     api.vms.get(VM0_NAME).disks.g<wbr>et(disk_name).status.state == &#39;ok&#39;<br>
&gt; &gt;<br>
&gt; &gt; AttributeError: &#39;NoneType&#39; object has no attribute &#39;state&#39;<br>
&gt; &gt;<br>
&gt; &gt; &lt;/error&gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Best Regards,<br>
&gt; &gt;<br>
&gt; &gt; Shlomi Ben-David | Software Engineer | Red Hat ISRAEL<br>
&gt; &gt; RHCSA | RHCVA | RHCE<br>
&gt; &gt; IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)<br>
&gt; &gt;<br>
&gt; &gt; OPEN SOURCE - 1 4 011 &amp;&amp; 011 4 1<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; ______________________________<wbr>_________________<br>
&gt; &gt; Devel mailing list<br>
&gt; &gt; <a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
&gt; &gt; <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
<br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">Pavel Zhukov</div><div dir="ltr">Software Engineer</div><div dir="ltr">RHEV Devops</div><div dir="ltr">IRC: landgraf</div><div><br></div></div></div></div></div>
</div>