<div dir="ltr">Sorry, this is the new execution link:<div><a href="https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-storage/1048/testReport/">https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-storage/1048/testReport/</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 23, 2018 at 1:23 AM, Elad Ben Aharon <span dir="ltr"><<a href="mailto:ebenahar@redhat.com" target="_blank">ebenahar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, I've triggered another execution [1] due to some issues I saw in the first which are not related to the patch.<div><br></div><div>The success rate is 78% which is low comparing to tier1 executions with code from downstream builds (95-100% success rates) [2].</div><div><br></div><div>From what I could see so far, there is an issue with move and copy operations to and from Gluster domains. For example [3].</div><div><br></div><div>The logs are attached.</div><div><br></div><div><br></div><div>[1]</div><div><font color="#1155cc"><u><a href="https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/testReport/" target="_blank">https://rhv-jenkins.rhev-ci-vm<wbr>s.eng.rdu2.redhat.com/job/rhv-<wbr>4.2-ge-runner-tier1-after-upgr<wbr>ade/7/testReport/</a></u></font><br></div><div><br></div><div><br></div><div><br></div><div>[2]</div><div><a href="https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/" target="_blank">https://rhv-jenkins.rhev-ci-vm<wbr>s.eng.rdu2.redhat.com/job/rhv-<wbr>4.2-ge-runner-tier1-after-upgr<wbr>ade/7/</a><br></div><div><br></div><div><br></div><div><br></div><div>[3]</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH deleteImage error=Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835<wbr>-5f603e682f33, domain=e5fd29c8-52ba-467e-be09<wbr>-ca40ff054dd4' from=: </span><br>:ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2<wbr>630-5976-4935, task_id=3d5f2a8a-881c-409e-93e<wbr>9-aaa643c10e42 (api:51) <br>2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='3d5f2a8a-881c-409e-93e9<wbr>-aaa643c10e42') Unexpected error (task:875) <br>Traceback (most recent call last): <br> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/task.py", line 882, in _run <br> return fn(*args, **kargs) <br> File "<string>", line 2, in deleteImage <br> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 49, in method <br> ret = func(*args, **kwargs) <br> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/hsm.py", line 1503, in deleteImage <br> raise se.ImageDoesNotExistInSD(imgUU<wbr>ID, sdUUID) <br>ImageDoesNotExistInSD: Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835<wbr>-5f603e682f33, domain=e5fd29c8-52ba-467e-be09<wbr>-ca40ff054dd4' <br>2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='3d5f2a8a-881c-409e-93e9<wbr>-aaa643c10e42') aborting: Task is aborted: "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835<wbr>-<br>5f603e682f33, domain=e5fd29c8-52ba-467e-be09<wbr>-ca40ff054dd4'" - code 268 (task:1181) <br>2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH deleteImage error=Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835<wbr>-5f603e682f33, domain=e5fd29c8-52ba-467e-be09<wbr>-ca40ff054d<br>d4' (dispatcher:82)<br></span><br></div><div><div class="h5"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <span dir="ltr"><<a href="mailto:ebenahar@redhat.com" target="_blank">ebenahar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Triggered a sanity tier1 execution [1] using [2], which covers all the requested areas, on iSCSI, NFS and Gluster. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">I'll update with the results.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div>[1]</div><a href="https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2_dev/job/rhv-4.2-ge-flow-storage/1161/" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://rhv-jenkins.rhev-ci-vm<wbr>s.eng.rdu2.redhat.com/view/4.2<wbr>_dev/job/rhv-4.2-ge-flow-stora<wbr>ge/1161/</a><div><br></div><div>[2]</div><div><a href="https://gerrit.ovirt.org/#/c/89830/" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>9830/</a><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">vdsm-4.30.0-291.git77aef9a.el7<wbr>.x86_64</span><br></span><br></div><div><div class="m_2057066576539246219m_-2623767660581280952gmail-m_-7173335272469596554gmail-h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <span dir="ltr"><<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Martin,<br>
<br>
I see [1] requires a rebase, can you please take care?<br>
</blockquote>
<br></span>
Should be rebased.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
At the moment, our automation is stable only on iSCSI, NFS, Gluster and FC.<br>
Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's not<br>
stable enough at the moment.<br>
</blockquote>
<br></span>
That is still pretty good.<div class="m_2057066576539246219m_-2623767660581280952gmail-m_-7173335272469596554gmail-m_-2951423565569528281gmail-m_1458218254404552003HOEnZb"><div class="m_2057066576539246219m_-2623767660581280952gmail-m_-7173335272469596554gmail-m_-2951423565569528281gmail-m_1458218254404552003h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
[1] <a href="https://gerrit.ovirt.org/#/c/89830/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>9830/</a><br>
<br>
<br>
Thanks<br>
<br>
On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi, sorry if I misunderstood, I waited for more input regarding what areas<br>
have to be tested here.<br>
<br>
</blockquote>
<br>
I'd say that you have quite a bit of freedom in this regard. GlusterFS<br>
should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite<br>
that covers basic operations (start & stop VM, migrate it), snapshots<br>
and merging them, and whatever else would be important for storage<br>
sanity.<br>
<br>
mpolednik<br>
<br>
<br>
On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a>><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
wrote:<br>
<br>
On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder,<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
will<br>
have to check, since usually, we don't execute our automation on them.<br>
<br>
<br>
</blockquote>
Any update on this? I believe the gluster tests were successful, OST<br>
passes fine and unit tests pass fine, that makes the storage backends<br>
test the last required piece.<br>
<br>
<br>
On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <<a href="mailto:ratamir@redhat.com" target="_blank">ratamir@redhat.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
+Elad<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>><br>
wrote:<br>
<br>
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Please make sure to run as much OST suites on this patch as possible<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
before merging ( using 'ci please build' )<br>
<br>
<br>
But note that OST is not a way to verify the patch.<br>
</blockquote>
<br>
Such changes require testing with all storage types we support.<br>
<br>
Nir<br>
<br>
On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <<br>
<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a><br>
><br>
<br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Hey,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I've created a patch[0] that is finally able to activate libvirt's<br>
dynamic_ownership for VDSM while not negatively affecting<br>
functionality of our storage code.<br>
<br>
That of course comes with quite a bit of code removal, mostly in<br>
the<br>
area of host devices, hwrng and anything that touches devices;<br>
bunch<br>
of test changes and one XML generation caveat (storage is handled<br>
by<br>
VDSM, therefore disk relabelling needs to be disabled on the VDSM<br>
level).<br>
<br>
Because of the scope of the patch, I welcome storage/virt/network<br>
people to review the code and consider the implication this change<br>
has<br>
on current/future features.<br>
<br>
[0] <a href="https://gerrit.ovirt.org/#/c/89830/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>9830/</a><br>
<br>
<br>
In particular: dynamic_ownership was set to 0 prehistorically (as<br>
</blockquote>
<br>
</blockquote>
part<br>
</blockquote>
of <a href="https://bugzilla.redhat.com/show_bug.cgi?id=554961" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=554961</a> ) because<br>
libvirt,<br>
running as root, was not able to play properly with root-squash nfs<br>
mounts.<br>
<br>
Have you attempted this use case?<br>
<br>
I join to Nir's request to run this with storage QE.<br>
<br>
<br>
<br>
</blockquote>
<br>
--<br>
<br>
<br>
Raz Tamir<br>
Manager, RHV QE<br>
<br>
<br>
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
</div></div></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>