7:53 a.m.
--Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Hi Elad,
why did you install vdsm-hook-allocate_net?
adding Dan as I think the hook is not supposed to fail this badly in any =
case
Thanks,
michal
On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
=20
Start VM fails on:
=20
2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm] =
(vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path: =
'dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/=
images/6cdabfe5-=20
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
-> =
u'*dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c6=
2/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-=20=
f35a01928211' (storagexml:334)=20
2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START =
getSpmStatus(spUUID=3D'940fe6f3-b0c6-4d0c-a921-198e7819c1cc', =
options=3DNone) from=3D::ffff:10.35.161.127,53512, =
task_id=3Dc70ace39-dbfe-4f5c-ae49-a1e3a82c=20
2758 (api:46)=20
2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root] =
/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=3D2 =
err=3Dvm net allocation hook: [unexpected error]: Traceback (most recent =
call last):=20
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line
105, in <module>=20
> main()=20
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line 93,
in main=20
> allocate_random_network(device_xml)=20
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line 62,
in allocate_random_network=20
> net =3D _get_random_network()=20
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line 50,
in _get_random_network=20
> available_nets =3D _parse_nets()=20
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line 46,
in _parse_nets=20
return [net for net in os.environ[AVAIL_NETS_KEY].split()]=20
File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__=20
raise KeyError(key)=20
KeyError: 'equivnets'=20
=20
=20
(hooks:110)=20
2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm] =
(vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process =
failed (vm:943)=20
Traceback (most recent call last):=20
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in =
_startUnderlyingVm=20
self._run()=20
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, =
in
_run=20
domxml =3D hooks.before_vm_start(self._buildDomainXML(),=20
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, =
in
_buildDomainXML=20
dom, self.id <http://self.id/>;,
self._custom['custom'])=20
File =
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py", line =
240, in replace_device_xml_with_hooks_xml=20
dev_custom)=20
File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line =
134, in before_device_create=20
params=3DcustomProperties)=20
File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line =
120, in _runHooksDir=20
raise exception.HookError(err)=20
HookError: Hook Error: ('vm net allocation hook: [unexpected error]: =
Traceback (most recent call last):\n File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line =
105, in <module>\n main()\n
File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", =
line 93,
in main\n allocate_random_network(device_xml)\n File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, =
i
n allocate_random_network\n net =3D _get_random_network()\n File
=
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50,
=
in _get_random_network\n available_nets =3D _parse_nets()\n File =
"/us
r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
46, =
in _parse_nets\n return [net for net in =
os.environ[AVAIL_NETS_KEY].split()]\n File =
"/usr/lib64/python2.7/UserDict.py", line 23, in __getit
em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
=20
=20
=20
Hence, the success rate was 28% against 100% running with d/s (d/s). =
If needed,
I'll compare against the latest master, but I think you get =
the picture with d/s.
=20
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64=20
libvirt-3.9.0-14.el7_5.3.x86_64=20
qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64=20
kernel 3.10.0-862.el7.x86_64
rhel7.5
=20
=20
Logs attached
=20
On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <ebenahar(a)redhat.com =
<mailto:ebenahar@redhat.com>> wrote:
nvm, found gluster 3.12 repo, managed to install vdsm
=20
On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <ebenahar(a)redhat.com =
<mailto:ebenahar@redhat.com>> wrote:
No, vdsm requires it:
=20
Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64 =
(/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)=20
Requires: glusterfs-fuse >=3D 3.12=20
Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64 (@rhv-4.2.3)
=20
Therefore, vdsm package installation is skipped upon force install.
=20
On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek =
<michal.skrivanek(a)redhat.com
<mailto:michal.skrivanek@redhat.com>> =
wrote:
=20
=20
> On 5 May 2018, at 00:38, Elad Ben Aharon <ebenahar(a)redhat.com =
<mailto:ebenahar@redhat.com>> wrote:
>=20
> Hi guys,=20
>=20
> The vdsm build from the patch requires glusterfs-fuse > 3.12. This is =
while the latest 4.2.3-5 d/s build requires 3.8.4 (3.4.0.59rhs-1.el7)
=20
because it is still oVirt, not a downstream build. We can=E2=80=99t =
really do
downstream builds with unmerged changes:/
=20
> Trying to get this gluster-fuse build, so far no luck.
> Is this requirement intentional?=20
=20
it should work regardless, I guess you can force install it without =
the
dependency
=20
>=20
> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek =
<michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
> Hi Elad,
> to make it easier to compare, Martin backported the change to 4.2 so =
it is
actually comparable with a run without that patch. Would you =
please try that out?=20
> It would be best to have 4.2 upstream and this[1] run to really =
minimize the noise.
>=20
> Thanks,
> michal
>=20
> [1] =
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_64=
/28/ =
<http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_6=
4/28/>
>=20
>> On 27 Apr 2018, at 09:23, Martin Polednik <mpolednik(a)redhat.com =
<mailto:mpolednik@redhat.com>> wrote:
>>=20
>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>>> I will update with the results of the next tier1 execution on =
latest
4.2.3
>>=20
>> That isn't master but old branch though. Could you run it against
>> *current* VDSM master?
>>=20
>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik =
<mpolednik(a)redhat.com <mailto:mpolednik@redhat.com>>
>>> wrote:
>>>=20
>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>>>>=20
>>>>> Hi, I've triggered another execution [1] due to some issues I saw
=
in the
>>>>> first which are not related to the patch.
>>>>>=20
>>>>> The success rate is 78% which is low comparing to tier1 =
executions with
>>>>> code from downstream builds (95-100% success
rates) [2].
>>>>>=20
>>>>=20
>>>> Could you run the current master (without the dynamic_ownership =
patch)
>>>> so that we have viable comparision?
>>>>=20
>>>> =46rom what I could see so far, there is an issue with move and =
copy
>>>>> operations to and from Gluster domains. For
example [3].
>>>>>=20
>>>>> The logs are attached.
>>>>>=20
>>>>>=20
>>>>> [1]
>>>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv =
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv>
>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv =
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv>
>>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> [2]
>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ =
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/>
>>>>>=20
>>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> [3]
>>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>>>>> deleteImage error=3DImage does not exist in domain:
>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> from=3D:
>>>>> :ffff:10.35.161.182,40936, =
flow_id=3Ddisks_syncAction_ba6b2630-5976-4935,
>>>>> task_id=3D3d5f2a8a-881c-409e-93e9-aaa643c10e42
(api:51)
>>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) =
[storage.TaskManager.Task]
>>>>>
(Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error =
(task:875)
>>>>> Traceback (most recent call last):
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", =
line 882,
>>>>> in
>>>>> _run
>>>>> return fn(*args, **kargs)
>>>>> File "<string>", line 2, in deleteImage
>>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line =
49, in
>>>>> method
>>>>> ret =3D func(*args, **kwargs)
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line =
1503,
>>>>> in
>>>>> deleteImage
>>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>>>> ImageDoesNotExistInSD: Image does not exist in domain:
>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>>=20
>>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) =
[storage.TaskManager.Task]
>>>>>
(Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is =
aborted:
>>>>> "Image does not exist in domain:
'image=3Dcabb8846-7a4b-4244-9835-
>>>>> 5f603e682f33,
domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'" - =
code 268
>>>>> (task:1181)
>>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) =
[storage.Dispatcher] FINISH
>>>>> deleteImage error=3DImage does not exist in
domain:
>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=3De5fd29c8-52ba-467e-be09
>>>>> -ca40ff054d
>>>>> d4' (dispatcher:82)
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon =
<ebenahar(a)redhat.com <mailto:ebenahar@redhat.com>>
>>>>> wrote:
>>>>>=20
>>>>> Triggered a sanity tier1 execution [1] using [2], which covers =
all the
>>>>>> requested areas, on iSCSI, NFS and Gluster.
>>>>>> I'll update with the results.
>>>>>>=20
>>>>>> [1]
>>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2 =
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2>
>>>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>>>=20
>>>>>> [2]
>>>>>> https://gerrit.ovirt.org/#/c/89830/ =
<https://gerrit.ovirt.org/#/c/89830/>
>>>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik =
<mpolednik(a)redhat.com <mailto:mpolednik@redhat.com>>
>>>>>> wrote:
>>>>>>=20
>>>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>>>>=20
>>>>>>> Hi Martin,
>>>>>>>>=20
>>>>>>>> I see [1] requires a rebase, can you please take care?
>>>>>>>>=20
>>>>>>>>=20
>>>>>>> Should be rebased.
>>>>>>>=20
>>>>>>> At the moment, our automation is stable only on iSCSI, NFS, =
Gluster and
>>>>>>>=20
>>>>>>>> FC.
>>>>>>>> Ceph is not supported and Cinder will be stabilized soon,
=
AFAIR, it's
>>>>>>>> not
>>>>>>>> stable enough at the moment.
>>>>>>>>=20
>>>>>>>>=20
>>>>>>> That is still pretty good.
>>>>>>>=20
>>>>>>>=20
>>>>>>> [1] https://gerrit.ovirt.org/#/c/89830/ =
<https://gerrit.ovirt.org/#/c/89830/>
>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> Thanks
>>>>>>>>=20
>>>>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik =
<mpolednik(a)redhat.com <mailto:mpolednik@redhat.com>
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> Hi, sorry if I misunderstood, I waited for more input
=
regarding what
>>>>>>>>>=20
>>>>>>>>>> areas
>>>>>>>>>> have to be tested here.
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> I'd say that you have quite a bit of freedom
in this regard.
>>>>>>>>> GlusterFS
>>>>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder
with =
some suite
>>>>>>>>> that covers basic operations
(start & stop VM, migrate it), =
snapshots
>>>>>>>>> and merging them, and whatever
else would be important for =
storage
>>>>>>>>> sanity.
>>>>>>>>>=20
>>>>>>>>> mpolednik
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik
<
>>>>>>>>> mpolednik(a)redhat.com
<mailto:mpolednik@redhat.com>
>>>>>>>>> >
>>>>>>>>>=20
>>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>> We can test this on iSCSI, NFS and GlusterFS.
As for ceph =
and
>>>>>>>>>>> cinder,
>>>>>>>>>>>=20
>>>>>>>>>>> will
>>>>>>>>>>>> have to check, since usually, we
don't execute our =
automation on
>>>>>>>>>>>> them.
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> Any update on this? I believe the gluster
tests were =
successful,
>>>>>>>>>>>> OST
>>>>>>>>>>>>=20
>>>>>>>>>>> passes fine and unit tests pass fine, that
makes the =
storage
>>>>>>>>>>> backends
>>>>>>>>>>> test the last required piece.
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir =
<ratamir(a)redhat.com <mailto:ratamir@redhat.com>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> +Elad
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan
Kenigsberg =
<danken(a)redhat.com <mailto:danken@redhat.com>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir
Soffer =
<nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal
Edri =
<eedri(a)redhat.com <mailto:eedri@redhat.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Please make sure to run as much
OST suites on this patch =
as
>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> before merging ( using
'ci please build' )
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> But note that OST is not
a way to verify the patch.
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Such changes require testing
with all storage types we =
support.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Nir
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09
PM, Martin Polednik <
>>>>>>>>>>>>>>> mpolednik(a)redhat.com
<mailto:mpolednik@redhat.com>
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> I've created a
patch[0] that is finally able to =
activate
>>>>>>>>>>>>>>>>>
libvirt's
>>>>>>>>>>>>>>>>> dynamic_ownership for
VDSM while not negatively =
affecting
>>>>>>>>>>>>>>>>>
functionality of our storage code.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> That of course comes
with quite a bit of code =
removal, mostly
>>>>>>>>>>>>>>>>>
in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> area of host devices,
hwrng and anything that touches =
devices;
>>>>>>>>>>>>>>>>>
bunch
>>>>>>>>>>>>>>>>> of test changes and
one XML generation caveat =
(storage is
>>>>>>>>>>>>>>>>>
handled
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> VDSM, therefore disk
relabelling needs to be disabled =
on the
>>>>>>>>>>>>>>>>>
VDSM
>>>>>>>>>>>>>>>>> level).
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Because of the scope
of the patch, I welcome
>>>>>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>>>>>> people to review the
code and consider the =
implication this
>>>>>>>>>>>>>>>>>
change
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>> on current/future
features.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/ =
<https://gerrit.ovirt.org/#/c/89830/>
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> In particular:
dynamic_ownership was set to 0 =
prehistorically
>>>>>>>>>>>>>>>>>
(as
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=3D554961 =
<https://bugzilla.redhat.com/show_bug.cgi?id=3D554961> ) because
>>>>>>>>>>>>>> libvirt,
>>>>>>>>>>>>>> running as root, was not able to
play properly with =
root-squash
>>>>>>>>>>>>>> nfs
>>>>>>>>>>>>>> mounts.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Have you attempted this use
case?
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> I join to Nir's request to
run this with storage QE.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Raz Tamir
>>>>>>>>>>>>> Manager, RHV QE
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>>=20
>>=20
>=20
>=20
=20
=20
=20
=20
<logs.tar.gz>
--Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hi
=
Elad,<div class=3D"">why did you install =
vdsm-hook-allocate_net?</div><div class=3D""><br
class=3D""></div><div =
class=3D"">adding Dan as I think the hook is not supposed to fail this =
badly in any case</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Thanks,</div><div class=3D"">michal<br
class=3D""><div><br =
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D"">On 5 May =
2018, at 19:22, Elad Ben Aharon <<a
href=3D"mailto:ebenahar@redhat.com"=
class=3D"">ebenahar(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D"">Start VM fails on:<div class=3D""><br
class=3D""></div><div =
class=3D""><span style=3D"font-family:monospace"
class=3D""><span =
style=3D"background-color: rgb(255, 255, 255);"
class=3D"">2018-05-05 =
17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm] =
(vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path: =
'dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/=
images/6cdabfe5- </span><br =
class=3D"">d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a0192821=
1' -> =
u'*dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c6=
2/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425- =
<br class=3D"">f35a01928211' (storagexml:334) <br
class=3D"">2018-05-05 =
17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START =
getSpmStatus(spUUID=3D'940fe6f3-b0c6-4d0c-a921-198e7819c1cc', =
options=3DNone) from=3D::ffff:10.35.161.127,53512, =
task_id=3Dc70ace39-dbfe-4f5c-ae49-a1e3a82c <br class=3D"">2758 (api:46) =
<br class=3D"">2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce)
=
[root] /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: =
rc=3D2 err=3Dvm net allocation hook: [unexpected error]: Traceback (most =
recent call last): <br class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line =
105, in <module> <br
class=3D""> main() <br =
class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 93, =
in main <br =
class=3D""> allocate_random_network(device_xml)
<br =
class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, =
in allocate_random_network <br
class=3D""> net =3D =
_get_random_network() <br class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50, =
in _get_random_network <br
class=3D""> available_nets =3D=
_parse_nets() <br class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46, =
in _parse_nets <br class=3D""> return [net
for net in =
os.environ[AVAIL_NETS_KEY].split()] <br class=3D""> File =
"/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__ <br =
class=3D""> raise KeyError(key) <br
class=3D"">KeyError: =
'equivnets' <br class=3D"">
<br class=3D"">
<br class=3D"">(hooks:110) <br class=3D"">2018-05-05
17:53:27,915+0300 =
<span style=3D"color:rgb(255,255,255);background-color:rgb(0,0,0)" =
class=3D"">ERROR</span><span style=3D"background-color:
rgb(255, 255, =
255);" class=3D""> (vm/e6ce66ce) [virt.vm] =
(vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process =
failed (vm:943) </span><br class=3D"">Traceback (most recent call
last): =
<br class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in =
_startUnderlyingVm <br
class=3D""> self._run() <br =
class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, in _run =
<br class=3D""> domxml =3D =
hooks.before_vm_start(self._buildDomainXML(), <br
class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, in =
_buildDomainXML <br class=3D""> dom, <a =
href=3D"http://self.id/" class=3D"">self.id</a>,
self._custom['custom']) =
<br class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py", line =
240, in replace_device_xml_with_hooks_xml <br =
class=3D""> dev_custom) <br
class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 134, in =
before_device_create <br =
class=3D""> params=3DcustomProperties) <br =
class=3D""> File =
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 120, in =
_runHooksDir <br class=3D""> raise =
exception.HookError(err) <br class=3D"">HookError: Hook Error: ('vm
net =
allocation hook: [unexpected error]: Traceback (most recent call =
last):\n File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line =
105, in <module>\n main()\n<br =
class=3D""> File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 93, =
in main\n allocate_random_network(device_xml)\n =
File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, =
i<br class=3D"">n allocate_random_network\n
net =3D =
_get_random_network()\n File =
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50, =
in _get_random_network\n available_nets =3D =
_parse_nets()\n File "/us<br =
class=3D"">r/libexec/vdsm/hooks/before_device_create/10_allocate_net",
=
line 46, in _parse_nets\n return [net for net in =
os.environ[AVAIL_NETS_KEY].split()]\n File =
"/usr/lib64/python2.7/UserDict.py", line 23, in __getit<br =
class=3D"">em__\n raise
KeyError(key)\nKeyError: =
\'equivnets\'\n\n\n',)<br class=3D""></span><br
class=3D""></div><div =
class=3D""><br class=3D""></div><div
class=3D""><br class=3D""></div><div =
class=3D"">Hence, the success rate was 28% against 100% running with d/s =
(d/s). If needed, I'll compare against the latest master, but I think =
you get the picture with d/s.</div><div class=3D""><br =
class=3D""></div><div class=3D""><span
style=3D"font-family:monospace" =
class=3D""><span style=3D"background-color: rgb(255, 255, 255);"
=
class=3D"">vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64 </span><br =
class=3D"">libvirt-3.9.0-14.el7_5.3.x86_64 <br =
class=3D"">qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64 <br
class=3D"">kernel =
3.10.0-862.el7.x86_64</span></div><div class=3D""><span =
style=3D"font-family:monospace" class=3D"">rhel7.5<br =
class=3D""></span><br class=3D""></div><div
class=3D""><br =
class=3D""></div><div class=3D"">Logs
attached</div></div><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On Sat, =
May 5, 2018 at 1:26 PM, Elad Ben Aharon <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:ebenahar@redhat.com"
target=3D"_blank" =
class=3D"">ebenahar(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D"">nvm, found gluster 3.12 repo, managed to install =
vdsm</div><div class=3D"HOEnZb"><div
class=3D"h5"><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On Sat, =
May 5, 2018 at 1:12 PM, Elad Ben Aharon <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:ebenahar@redhat.com"
target=3D"_blank" =
class=3D"">ebenahar(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D"">No, vdsm requires it:<div
class=3D""><br =
class=3D""></div><div class=3D""><span
style=3D"font-family:monospace" =
class=3D""><span style=3D"background-color: rgb(255, 255, 255);"
=
class=3D"">Error: Package: vdsm-4.20.27-3.gitfee7810.el7.<wbr =
class=3D"">centos.x86_64 (/vdsm-4.20.27-3.gitfee7810.el<wbr =
class=3D"">7.centos.x86_64) </span><br =
class=3D""> Req=
uires: glusterfs-fuse >=3D 3.12 <br =
class=3D""> Ins=
talled: glusterfs-fuse-3.8.4-54.8.el7.<wbr class=3D"">x86_64 =
(@rhv-4.2.3)<br class=3D""></span><br
class=3D""></div><div =
class=3D"">Therefore, vdsm package installation is skipped upon force =
install.</div></div><div
class=3D"m_8270803836802176999HOEnZb"><div =
class=3D"m_8270803836802176999h5"><div
class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On Sat, May 5, 2018 at
11:42 AM, =
Michal Skrivanek <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek(a)redhat.com</a>&gt;</span>
wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word;line-break:after-white-space"
class=3D""><br=
class=3D""><div class=3D""><span
class=3D""><br class=3D""><blockquote =
type=3D"cite" class=3D""><div class=3D"">On 5 May
2018, at 00:38, Elad =
Ben Aharon <<a href=3D"mailto:ebenahar@redhat.com"
target=3D"_blank" =
class=3D"">ebenahar(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788A=
pple-interchange-newline"><div class=3D""><div
dir=3D"ltr" class=3D"">Hi =
guys, <div class=3D""><br
class=3D""></div><div class=3D"">The =
vdsm build from the patch requires glusterfs-fuse > 3.12. This =
is while the latest 4.2.3-5 d/s build requires 3.8.4 (<span =
style=3D"font-family:monospace" class=3D""><span =
style=3D"background-color:rgb(255,255,255)" =
class=3D"">3.4.0.59rhs-1.el7)</span><br =
class=3D""></span></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div></span>because it is still oVirt, not a
downstream =
build. We can=E2=80=99t really do downstream builds with unmerged =
changes:/</div><div class=3D""><span
class=3D""><br class=3D""><blockquote=
type=3D"cite" class=3D""><div class=3D""><div
dir=3D"ltr" class=3D""><div=
class=3D""><font face=3D"monospace"
class=3D"">Trying to get this =
gluster-fuse build, so far no luck.</font></div><div
class=3D""><font =
face=3D"monospace" class=3D"">Is this requirement =
intentional? </font></div></div></div></blockquote><div
=
class=3D""><br class=3D""></div></span>it should
work regardless, I =
guess you can force install it without the dependency</div><div =
class=3D""><div =
class=3D"m_8270803836802176999m_4224343900157515506h5"><div
class=3D""><br=
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D""><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On Fri, =
May 4, 2018 at 2:38 PM, Michal Skrivanek <span dir=3D"ltr" =
class=3D""><<a
href=3D"mailto:michal.skrivanek@redhat.com" =
target=3D"_blank"
class=3D"">michal.skrivanek(a)redhat.com</a>&gt;</span> =
wrote:<br class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word;line-break:after-white-space"
class=3D"">Hi =
Elad,<div class=3D"">to make it easier to compare, Martin backported the
=
change to 4.2 so it is actually comparable with a run without that =
patch. Would you please try that out? </div><div
class=3D"">It =
would be best to have 4.2 upstream and this[1] run to really minimize =
the noise.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Thanks,</div><div
class=3D"">michal</div><div class=3D""><br =
class=3D""></div><div class=3D"">[1] <a =
href=3D"http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-dem...
7-x86_64/28/" target=3D"_blank"
class=3D"">http://jenkins.ovirt.org/j<wbr =
class=3D"">ob/vdsm_4.2_build-artifacts-on<wbr =
class=3D"">-demand-el7-x86_64/28/</a></div><div
class=3D""><br =
class=3D""><div class=3D""><blockquote
type=3D"cite" class=3D""><div =
class=3D""><div =
class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788h=
5"><div class=3D"">On 27 Apr 2018, at 09:23, Martin Polednik
<<a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788m=
_-2464431127513935993Apple-interchange-newline"></div></div><div =
class=3D""><div class=3D""><div
class=3D""><div =
class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788h=
5">On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:<br =
class=3D""><blockquote type=3D"cite" class=3D"">I
will update with the =
results of the next tier1 execution on latest 4.2.3<br =
class=3D""></blockquote><br class=3D"">That isn't
master but old branch =
though. Could you run it against<br class=3D"">*current* VDSM
master?<br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D"">On Mon, =
Apr 23, 2018 at 3:56 PM, Martin Polednik <<a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a>&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D"">On =
23/04/18 01:23 +0300, Elad Ben Aharon wrote:<br class=3D""><br =
class=3D""><blockquote type=3D"cite" class=3D"">Hi,
I've triggered =
another execution [1] due to some issues I saw in the<br class=3D"">first
=
which are not related to the patch.<br class=3D""><br
class=3D"">The =
success rate is 78% which is low comparing to tier1 executions with<br =
class=3D"">code from downstream builds (95-100% success rates) [2].<br =
class=3D""><br class=3D""></blockquote><br
class=3D"">Could you run the =
current master (without the dynamic_ownership patch)<br class=3D"">so =
that we have viable comparision?<br class=3D""><br
class=3D"">=46rom =
what I could see so far, there is an issue with move and copy<br =
class=3D""><blockquote type=3D"cite"
class=3D"">operations to and from =
Gluster domains. For example [3].<br class=3D""><br
class=3D"">The logs =
are attached.<br class=3D""><br class=3D""><br
class=3D"">[1]<br =
class=3D"">*<a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv&... =
target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-v<wbr
=
class=3D"">ms.eng.rdu2.redhat.com/job/rhv</a><br =
class=3D"">-4.2-ge-runner-tier1-after-upg<wbr =
class=3D"">rade/7/testReport/<br class=3D""><<a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv&... =
target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-v<wbr
=
class=3D"">ms.eng.rdu2.redhat.com/job/rhv</a><br =
class=3D"">-4.2-ge-runner-tier1-after-upg<wbr =
class=3D"">rade/7/testReport/>*<br class=3D""><br
class=3D""><br =
class=3D""><br class=3D"">[2]<br
class=3D""><a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/" =
target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-vm<wbr
=
class=3D"">s.eng.rdu2.redhat.com/job/</a><br
class=3D""><br =
class=3D"">rhv-4.2-ge-runner-tier1-after-<wbr
class=3D"">upgrade/7/<br =
class=3D""><br class=3D""><br
class=3D""><br class=3D"">[3]<br =
class=3D"">2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) =
[vdsm.api] FINISH<br class=3D"">deleteImage error=3DImage does not exist
=
in domain:<br class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr =
class=3D"">-5f603e682f33,<br
class=3D"">domain=3De5fd29c8-52ba-467e-be09<w=
br class=3D"">-ca40ff054dd4'<br class=3D"">from=3D:<br
=
class=3D"">:ffff:10.35.161.182,40936,
flow_id=3Ddisks_syncAction_ba6b2<wbr=
class=3D"">630-5976-4935,<br =
class=3D"">task_id=3D3d5f2a8a-881c-409e-93e<wbr
class=3D"">9-aaa643c10e42 =
(api:51)<br class=3D"">2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) =
[storage.TaskManager.Task]<br =
class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9<wbr
class=3D"">-aaa643c10e42')=
Unexpected error (task:875)<br class=3D"">Traceback (most recent call =
last):<br class=3D"">File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/vdsm/storage/task.py", line 882,<br
class=3D"">in<br =
class=3D"">_run<br class=3D""> return fn(*args,
**kargs)<br =
class=3D"">File "<string>", line 2, in
deleteImage<br =
class=3D"">File "/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/vdsm/common/api.py", line 49, in<br
class=3D"">method<br =
class=3D""> ret =3D func(*args, **kwargs)<br
class=3D"">File =
"/usr/lib/python2.7/site-packa<wbr
class=3D"">ges/vdsm/storage/hsm.py", =
line 1503,<br class=3D"">in<br
class=3D"">deleteImage<br class=3D""> =
raise se.ImageDoesNotExistInSD(imgUU<wbr class=3D"">ID,
sdUUID)<br =
class=3D"">ImageDoesNotExistInSD: Image does not exist in domain:<br =
class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr =
class=3D"">-5f603e682f33,<br
class=3D"">domain=3De5fd29c8-52ba-467e-be09<w=
br class=3D"">-ca40ff054dd4'<br class=3D""><br
class=3D"">2018-04-22 =
13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]<br =
class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9<wbr
class=3D"">-aaa643c10e42')=
aborting: Task is aborted:<br class=3D"">"Image does not exist in =
domain: 'image=3Dcabb8846-7a4b-4244-9835<wbr class=3D"">-<br =
class=3D"">5f603e682f33, domain=3De5fd29c8-52ba-467e-be09<wbr =
class=3D"">-ca40ff054dd4'" - code 268<br
class=3D"">(task:1181)<br =
class=3D"">2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) =
[storage.Dispatcher] FINISH<br class=3D"">deleteImage error=3DImage does
=
not exist in domain:<br
class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr =
class=3D"">-5f603e682f33,<br
class=3D"">domain=3De5fd29c8-52ba-467e-be09<b=
r class=3D"">-ca40ff054d<br class=3D"">d4'
(dispatcher:82)<br =
class=3D""><br class=3D""><br
class=3D""><br class=3D"">On Thu, Apr 19, =
2018 at 5:34 PM, Elad Ben Aharon <<a =
href=3D"mailto:ebenahar@redhat.com" target=3D"_blank" =
class=3D"">ebenahar(a)redhat.com</a>&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D"">Triggered a sanity tier1 execution
[1] using =
[2], which covers all the<br class=3D""><blockquote
type=3D"cite" =
class=3D"">requested areas, on iSCSI, NFS and Gluster.<br
class=3D"">I'll =
update with the results.<br class=3D""><br
class=3D"">[1]<br class=3D""><a=
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2... =
target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-vm<wbr
=
class=3D"">s.eng.rdu2.redhat.com/view/4.2</a><br =
class=3D"">_dev/job/rhv-4.2-ge-flow-stora<wbr
class=3D"">ge/1161/<br =
class=3D""><br class=3D"">[2]<br
class=3D""><a =
href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/#/c/8<wbr
class=3D"">9830/</a><br =
class=3D"">vdsm-4.30.0-291.git77aef9a.el7<wbr
class=3D"">.x86_64<br =
class=3D""><br class=3D""><br
class=3D""><br class=3D"">On Thu, Apr 19, =
2018 at 3:07 PM, Martin Polednik <<a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a>&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D"">On 19/04/18 14:54 +0300, Elad Ben
Aharon =
wrote:<br class=3D""><blockquote type=3D"cite"
class=3D""><br =
class=3D"">Hi Martin,<br class=3D""><blockquote
type=3D"cite" =
class=3D""><br class=3D"">I see [1] requires a rebase, can
you please =
take care?<br class=3D""><br class=3D""><br
class=3D""></blockquote>Should=
be rebased.<br class=3D""><br class=3D"">At the moment,
our automation =
is stable only on iSCSI, NFS, Gluster and<br class=3D""><br =
class=3D""><blockquote type=3D"cite"
class=3D"">FC.<br class=3D"">Ceph =
is not supported and Cinder will be stabilized soon, AFAIR, it's<br =
class=3D"">not<br class=3D"">stable enough at the
moment.<br =
class=3D""><br class=3D""><br
class=3D""></blockquote>That is still =
pretty good.<br class=3D""><br class=3D""><br
class=3D"">[1] <a =
href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/#/c/8<wbr
class=3D"">9830/</a><br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D""><br =
class=3D""><br class=3D"">Thanks<br
class=3D""><br class=3D"">On Wed, =
Apr 18, 2018 at 2:17 PM, Martin Polednik <<a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a><br
class=3D"">><br =
class=3D"">wrote:<br class=3D""><br
class=3D"">On 18/04/18 11:37 +0300, =
Elad Ben Aharon wrote:<br class=3D""><br
class=3D""><blockquote =
type=3D"cite" class=3D""><br class=3D"">Hi, sorry
if I misunderstood, I =
waited for more input regarding what<br class=3D""><br =
class=3D""><blockquote type=3D"cite"
class=3D"">areas<br class=3D"">have =
to be tested here.<br class=3D""><br class=3D""><br
class=3D"">I'd say =
that you have quite a bit of freedom in this regard.<br =
class=3D""></blockquote>GlusterFS<br class=3D"">should
be covered by =
Dennis, so iSCSI/NFS/ceph/cinder with some suite<br class=3D"">that =
covers basic operations (start & stop VM, migrate it), snapshots<br =
class=3D"">and merging them, and whatever else would be important for =
storage<br class=3D"">sanity.<br class=3D""><br
class=3D"">mpolednik<br =
class=3D""><br class=3D""><br class=3D"">On
Wed, Apr 18, 2018 at 11:16 =
AM, Martin Polednik <<br class=3D""><a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a><br
class=3D"">><br class=3D""><br =
class=3D"">wrote:<br class=3D""><blockquote
type=3D"cite" class=3D""><br =
class=3D"">On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:<br =
class=3D""><br class=3D""><br
class=3D""><blockquote type=3D"cite" =
class=3D"">We can test this on iSCSI, NFS and GlusterFS. As for ceph =
and<br class=3D"">cinder,<br class=3D""><br
class=3D"">will<br =
class=3D""><blockquote type=3D"cite"
class=3D"">have to check, since =
usually, we don't execute our automation on<br class=3D"">them.<br
=
class=3D""><br class=3D""><br class=3D"">Any
update on this? I believe =
the gluster tests were successful,<br class=3D"">OST<br
class=3D""><br =
class=3D""></blockquote>passes fine and unit tests pass fine, that
makes =
the storage<br class=3D"">backends<br class=3D"">test the
last required =
piece.<br class=3D""><br class=3D""><br
class=3D"">On Wed, Apr 11, 2018 =
at 4:38 PM, Raz Tamir <<a href=3D"mailto:ratamir@redhat.com" =
target=3D"_blank"
class=3D"">ratamir(a)redhat.com</a>&gt;<br =
class=3D"">wrote:<br class=3D""><br
class=3D""><br class=3D"">+Elad<br =
class=3D""><blockquote type=3D"cite"
class=3D""><br class=3D""><br =
class=3D"">On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <<a =
href=3D"mailto:danken@redhat.com" target=3D"_blank" =
class=3D"">danken(a)redhat.com</a><br
class=3D""><blockquote type=3D"cite" =
class=3D"">><br class=3D"">wrote:<br
class=3D""><br class=3D"">On =
Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <<a =
href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D""><br class=3D"">On
Wed, Apr 11, 2018 at 12:31 =
PM Eyal Edri <<a href=3D"mailto:eedri@redhat.com"
target=3D"_blank" =
class=3D"">eedri(a)redhat.com</a>&gt;<br
class=3D""><blockquote =
type=3D"cite" class=3D"">wrote:<br
class=3D""><br class=3D""><br =
class=3D"">Please make sure to run as much OST suites on this patch =
as<br class=3D""><blockquote type=3D"cite"
class=3D"">possible<br =
class=3D""><br class=3D"">before merging ( using 'ci
please build' )<br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D""><br =
class=3D""><br class=3D"">But note that OST is not a way to
verify the =
patch.<br class=3D""><br class=3D""><br
class=3D""></blockquote>Such =
changes require testing with all storage types we support.<br =
class=3D""><br class=3D"">Nir<br
class=3D""><br class=3D"">On Tue, Apr =
10, 2018 at 4:09 PM, Martin Polednik <<br class=3D""><a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a><br
class=3D"">><br class=3D""><br =
class=3D"">wrote:<br class=3D""><br
class=3D""><br class=3D""><blockquote =
type=3D"cite" class=3D"">Hey,<br class=3D""><br
class=3D""><br =
class=3D"">I've created a patch[0] that is finally able to activate<br
=
class=3D""><blockquote type=3D"cite"
class=3D"">libvirt's<br =
class=3D"">dynamic_ownership for VDSM while not negatively affecting<br =
class=3D"">functionality of our storage code.<br
class=3D""><br =
class=3D"">That of course comes with quite a bit of code removal, =
mostly<br class=3D"">in<br class=3D"">the<br
class=3D"">area of host =
devices, hwrng and anything that touches devices;<br
class=3D"">bunch<br =
class=3D"">of test changes and one XML generation caveat (storage is<br =
class=3D"">handled<br class=3D"">by<br
class=3D"">VDSM, therefore disk =
relabelling needs to be disabled on the<br class=3D"">VDSM<br =
class=3D"">level).<br class=3D""><br
class=3D"">Because of the scope of =
the patch, I welcome<br class=3D"">storage/virt/network<br =
class=3D"">people to review the code and consider the implication =
this<br class=3D"">change<br class=3D"">has<br
class=3D"">on =
current/future features.<br class=3D""><br class=3D"">[0]
<a =
href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/#/c/8<wbr
class=3D"">9830/</a><br =
class=3D""><br class=3D""><br class=3D"">In
particular: =
dynamic_ownership was set to 0 prehistorically<br
class=3D"">(as<br =
class=3D""><br class=3D""><br
class=3D""></blockquote>part<br =
class=3D""><br class=3D""></blockquote><br
class=3D"">of <a =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D554961" =
target=3D"_blank" class=3D"">https://bugzilla.redhat.com/sh<wbr
=
class=3D"">ow_bug.cgi?id=3D554961</a> ) because<br =
class=3D""></blockquote>libvirt,<br class=3D"">running
as root, was not =
able to play properly with root-squash<br class=3D"">nfs<br =
class=3D"">mounts.<br class=3D""><br
class=3D"">Have you attempted this =
use case?<br class=3D""><br class=3D"">I join to Nir's
request to run =
this with storage QE.<br class=3D""><br class=3D""><br
class=3D""><br =
class=3D""><br class=3D"">--<br
class=3D""></blockquote><br class=3D""><br=
class=3D"">Raz Tamir<br class=3D"">Manager, RHV QE<br
class=3D""><br =
class=3D""><br class=3D""><br
class=3D""><br class=3D""><br =
class=3D""></blockquote></blockquote></blockquote></blockquote></blockquot=
e></blockquote></blockquote><br
class=3D""></blockquote></blockquote><br =
class=3D""><br
class=3D""></blockquote></blockquote></div></div><span
=
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">Devel mailing
list<br =
class=3D""><a href=3D"mailto:Devel@ovirt.org"
target=3D"_blank" =
class=3D"">Devel(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D""><br
class=3D""><br =
class=3D""></span></div></div></blockquote></div><br
=
class=3D""></div></div></blockquote></div><br
class=3D""></div>
</div></blockquote></div><br =
class=3D""></div></div></div></blockquote></div><br
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
<span =
id=3D"cid:77382D15-7BFB-4164-A6D0-F8FA5BE5E692@mrkev"><logs.tar.gz><=
/span></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187--
5:14 a.m.
New subject: dynamic ownership changes
On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
<michal.skrivanek(a)redhat.com> wrote:
Hi Elad,
why did you install vdsm-hook-allocate_net?
adding Dan as I think the hook is not supposed to fail this badly in any
case
yep, this looks bad and deserves a little bug report. Installing this
little hook should not block vm startup.
But more importantly - what is the conclusion of this thread? Do we
have a green light from QE to take this in?
>
> Thanks,
> michal
>
> On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com> wrote:
>
> Start VM fails on:
>
> 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
> (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path:
>
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/images/6cdabfe5-
> d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211' ->
>
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-
> f35a01928211' (storagexml:334)
> 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
> getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc', options=None)
> from=::ffff:10.35.161.127,53512, task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
> 2758 (api:46)
> 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
> /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2 err=vm
> net allocation hook: [unexpected error]: Traceback (most recent call last):
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 105, in <module>
> main()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 93, in main
> allocate_random_network(device_xml)
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 62, in allocate_random_network
> net = _get_random_network()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 50, in _get_random_network
> available_nets = _parse_nets()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 46, in _parse_nets
> return [net for net in os.environ[AVAIL_NETS_KEY].split()]
> File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
> raise KeyError(key)
> KeyError: 'equivnets'
>
>
> (hooks:110)
> 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
> (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process failed
> (vm:943)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in
> _startUnderlyingVm
> self._run()
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, in
_run
> domxml = hooks.before_vm_start(self._buildDomainXML(),
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, in
> _buildDomainXML
> dom, self.id, self._custom['custom'])
> File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py",
> line 240, in replace_device_xml_with_hooks_xml
> dev_custom)
> File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 134,
in
> before_device_create
> params=customProperties)
> File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 120,
in
> _runHooksDir
> raise exception.HookError(err)
> HookError: Hook Error: ('vm net allocation hook: [unexpected error]:
> Traceback (most recent call last):\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 105,
in
> <module>\n main()\n
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
> 93, in main\n allocate_random_network(device_xml)\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, i
> n allocate_random_network\n net = _get_random_network()\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50,
in
> _get_random_network\n available_nets = _parse_nets()\n File "/us
> r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46, in
> _parse_nets\n return [net for net in
> os.environ[AVAIL_NETS_KEY].split()]\n File
> "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
> em__\n raise KeyError(key)\nKeyError: \'equivnets\'\n\n\n',)
>
>
>
> Hence, the success rate was 28% against 100% running with d/s (d/s). If
> needed, I'll compare against the latest master, but I think you get the
> picture with d/s.
>
> vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
> libvirt-3.9.0-14.el7_5.3.x86_64
> qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
> kernel 3.10.0-862.el7.x86_64
> rhel7.5
>
>
> Logs attached
>
> On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <ebenahar(a)redhat.com> wrote:
>>
>> nvm, found gluster 3.12 repo, managed to install vdsm
>>
>> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>>>
>>> No, vdsm requires it:
>>>
>>> Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>> Requires: glusterfs-fuse >= 3.12
>>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64 (@rhv-4.2.3)
>>>
>>> Therefore, vdsm package installation is skipped upon force install.
>>>
>>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>>> <michal.skrivanek(a)redhat.com> wrote:
>>>>
>>>>
>>>>
>>>> On 5 May 2018, at 00:38, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> The vdsm build from the patch requires glusterfs-fuse > 3.12. This is
>>>> while the latest 4.2.3-5 d/s build requires 3.8.4 (3.4.0.59rhs-1.el7)
>>>>
>>>>
>>>> because it is still oVirt, not a downstream build. We can’t really do
>>>> downstream builds with unmerged changes:/
>>>>
>>>> Trying to get this gluster-fuse build, so far no luck.
>>>> Is this requirement intentional?
>>>>
>>>>
>>>> it should work regardless, I guess you can force install it without the
>>>> dependency
>>>>
>>>>
>>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>>>> <michal.skrivanek(a)redhat.com> wrote:
>>>>>
>>>>> Hi Elad,
>>>>> to make it easier to compare, Martin backported the change to 4.2 so
it
>>>>> is actually comparable with a run without that patch. Would you
please try
>>>>> that out?
>>>>> It would be best to have 4.2 upstream and this[1] run to really
>>>>> minimize the noise.
>>>>>
>>>>> Thanks,
>>>>> michal
>>>>>
>>>>> [1]
>>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_6...
>>>>>
>>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com> wrote:
>>>>>
>>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> I will update with the results of the next tier1 execution on latest
>>>>> 4.2.3
>>>>>
>>>>>
>>>>> That isn't master but old branch though. Could you run it
against
>>>>> *current* VDSM master?
>>>>>
>>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> Hi, I've triggered another execution [1] due to some issues I saw
in
>>>>> the
>>>>> first which are not related to the patch.
>>>>>
>>>>> The success rate is 78% which is low comparing to tier1 executions
with
>>>>> code from downstream builds (95-100% success rates) [2].
>>>>>
>>>>>
>>>>> Could you run the current master (without the dynamic_ownership
patch)
>>>>> so that we have viable comparision?
>>>>>
>>>>> From what I could see so far, there is an issue with move and copy
>>>>>
>>>>> operations to and from Gluster domains. For example [3].
>>>>>
>>>>> The logs are attached.
>>>>>
>>>>>
>>>>> [1]
>>>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>>>
>>>>>
>>>>>
>>>>> [2]
>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>>>
>>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>>>
>>>>>
>>>>>
>>>>> [3]
>>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>>>>> deleteImage error=Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> from=:
>>>>> :ffff:10.35.161.182,40936,
flow_id=disks_syncAction_ba6b2630-5976-4935,
>>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>>>>> [storage.TaskManager.Task]
>>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
error
>>>>> (task:875)
>>>>> Traceback (most recent call last):
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
>>>>> in
>>>>> _run
>>>>> return fn(*args, **kargs)
>>>>> File "<string>", line 2, in deleteImage
>>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line 49, in
>>>>> method
>>>>> ret = func(*args, **kwargs)
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503,
>>>>> in
>>>>> deleteImage
>>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>>>> ImageDoesNotExistInSD: Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>>
>>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>>>>> [storage.TaskManager.Task]
>>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task
is
>>>>> aborted:
>>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-
>>>>> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'"
- code 268
>>>>> (task:1181)
>>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher]
>>>>> FINISH
>>>>> deleteImage error=Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09
>>>>> -ca40ff054d
>>>>> d4' (dispatcher:82)
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> Triggered a sanity tier1 execution [1] using [2], which covers all
the
>>>>>
>>>>> requested areas, on iSCSI, NFS and Gluster.
>>>>> I'll update with the results.
>>>>>
>>>>> [1]
>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>>
>>>>> [2]
>>>>> https://gerrit.ovirt.org/#/c/89830/
>>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> Hi Martin,
>>>>>
>>>>>
>>>>> I see [1] requires a rebase, can you please take care?
>>>>>
>>>>>
>>>>> Should be rebased.
>>>>>
>>>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster
and
>>>>>
>>>>> FC.
>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>>> not
>>>>> stable enough at the moment.
>>>>>
>>>>>
>>>>> That is still pretty good.
>>>>>
>>>>>
>>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>>
>>>>> areas
>>>>> have to be tested here.
>>>>>
>>>>>
>>>>> I'd say that you have quite a bit of freedom in this regard.
>>>>>
>>>>> GlusterFS
>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>> and merging them, and whatever else would be important for storage
>>>>> sanity.
>>>>>
>>>>> mpolednik
>>>>>
>>>>>
>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>> mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
>>>>> cinder,
>>>>>
>>>>> will
>>>>>
>>>>> have to check, since usually, we don't execute our automation on
>>>>> them.
>>>>>
>>>>>
>>>>> Any update on this? I believe the gluster tests were successful,
>>>>> OST
>>>>>
>>>>> passes fine and unit tests pass fine, that makes the storage
>>>>> backends
>>>>> test the last required piece.
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> +Elad
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>>>>>
>>>>> >
>>>>> wrote:
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Please make sure to run as much OST suites on this patch as
>>>>>
>>>>> possible
>>>>>
>>>>> before merging ( using 'ci please build' )
>>>>>
>>>>>
>>>>>
>>>>> But note that OST is not a way to verify the patch.
>>>>>
>>>>>
>>>>> Such changes require testing with all storage types we support.
>>>>>
>>>>> Nir
>>>>>
>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>> mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hey,
>>>>>
>>>>>
>>>>> I've created a patch[0] that is finally able to activate
>>>>>
>>>>> libvirt's
>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>> functionality of our storage code.
>>>>>
>>>>> That of course comes with quite a bit of code removal, mostly
>>>>> in
>>>>> the
>>>>> area of host devices, hwrng and anything that touches devices;
>>>>> bunch
>>>>> of test changes and one XML generation caveat (storage is
>>>>> handled
>>>>> by
>>>>> VDSM, therefore disk relabelling needs to be disabled on the
>>>>> VDSM
>>>>> level).
>>>>>
>>>>> Because of the scope of the patch, I welcome
>>>>> storage/virt/network
>>>>> people to review the code and consider the implication this
>>>>> change
>>>>> has
>>>>> on current/future features.
>>>>>
>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>>
>>>>> In particular: dynamic_ownership was set to 0 prehistorically
>>>>> (as
>>>>>
>>>>>
>>>>> part
>>>>>
>>>>>
>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>
>>>>> libvirt,
>>>>> running as root, was not able to play properly with root-squash
>>>>> nfs
>>>>> mounts.
>>>>>
>>>>> Have you attempted this use case?
>>>>>
>>>>> I join to Nir's request to run this with storage QE.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> Raz Tamir
>>>>> Manager, RHV QE
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
> <logs.tar.gz>
>
>
5:21 a.m.
New subject: dynamic ownership changes
Hi Dan,
In the last execution, the success rate was very low due to a large number
of failures on start VM caused, according to Michal, by the
vdsm-hook-allocate_net
that was installed on the host.
This is the latest status here, would you like me to re-execute? If so,
with or W/O vdsm-hook-allocate_net installed?
On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
<michal.skrivanek(a)redhat.com> wrote:
> Hi Elad,
> why did you install vdsm-hook-allocate_net?
>
> adding Dan as I think the hook is not supposed to fail this badly in any
> case
yep, this looks bad and deserves a little bug report. Installing this
little hook should not block vm startup.
But more importantly - what is the conclusion of this thread? Do we
have a green light from QE to take this in?
>
> Thanks,
> michal
>
> On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com> wrote:
>
> Start VM fails on:
>
> 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
> (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path:
> 'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
9a78-bdd13a843c62/images/6cdabfe5-
> d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211' ->
> u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
9834f235d1c8/7ef97445-30e6-4435-8425-
> f35a01928211' (storagexml:334)
> 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
> getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
options=None)
> from=::ffff:10.35.161.127,53512, task_id=c70ace39-dbfe-4f5c-
ae49-a1e3a82c
> 2758 (api:46)
> 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
> /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
err=vm
> net allocation hook: [unexpected error]: Traceback (most recent call
last):
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 105, in <module>
> main()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 93, in main
> allocate_random_network(device_xml)
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 62, in allocate_random_network
> net = _get_random_network()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 50, in _get_random_network
> available_nets = _parse_nets()
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 46, in _parse_nets
> return [net for net in os.environ[AVAIL_NETS_KEY].split()]
> File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
> raise KeyError(key)
> KeyError: 'equivnets'
>
>
> (hooks:110)
> 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
> (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process
failed
> (vm:943)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in
> _startUnderlyingVm
> self._run()
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, in
_run
> domxml = hooks.before_vm_start(self._buildDomainXML(),
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, in
> _buildDomainXML
> dom, self.id, self._custom['custom'])
> File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py",
> line 240, in replace_device_xml_with_hooks_xml
> dev_custom)
> File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
134, in
> before_device_create
> params=customProperties)
> File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
120, in
> _runHooksDir
> raise exception.HookError(err)
> HookError: Hook Error: ('vm net allocation hook: [unexpected error]:
> Traceback (most recent call last):\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
105, in
> <module>\n main()\n
> File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
> 93, in main\n allocate_random_network(device_xml)\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
62, i
> n allocate_random_network\n net = _get_random_network()\n File
> "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
50, in
> _get_random_network\n available_nets = _parse_nets()\n File "/us
> r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46, in
> _parse_nets\n return [net for net in
> os.environ[AVAIL_NETS_KEY].split()]\n File
> "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
> em__\n raise KeyError(key)\nKeyError: \'equivnets\'\n\n\n',)
>
>
>
> Hence, the success rate was 28% against 100% running with d/s (d/s). If
> needed, I'll compare against the latest master, but I think you get the
> picture with d/s.
>
> vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
> libvirt-3.9.0-14.el7_5.3.x86_64
> qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
> kernel 3.10.0-862.el7.x86_64
> rhel7.5
>
>
> Logs attached
>
> On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
>>
>> nvm, found gluster 3.12 repo, managed to install vdsm
>>
>> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>>>
>>> No, vdsm requires it:
>>>
>>> Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>> Requires: glusterfs-fuse >= 3.12
>>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64 (@rhv-4.2.3)
>>>
>>> Therefore, vdsm package installation is skipped upon force install.
>>>
>>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>>> <michal.skrivanek(a)redhat.com> wrote:
>>>>
>>>>
>>>>
>>>> On 5 May 2018, at 00:38, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> The vdsm build from the patch requires glusterfs-fuse > 3.12. This
is
>>>> while the latest 4.2.3-5 d/s build requires 3.8.4 (3.4.0.59rhs-1.el7)
>>>>
>>>>
>>>> because it is still oVirt, not a downstream build. We can’t really do
>>>> downstream builds with unmerged changes:/
>>>>
>>>> Trying to get this gluster-fuse build, so far no luck.
>>>> Is this requirement intentional?
>>>>
>>>>
>>>> it should work regardless, I guess you can force install it without
the
>>>> dependency
>>>>
>>>>
>>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>>>> <michal.skrivanek(a)redhat.com> wrote:
>>>>>
>>>>> Hi Elad,
>>>>> to make it easier to compare, Martin backported the change to 4.2
so
it
>>>>> is actually comparable with a run without that patch. Would you
please try
>>>>> that out?
>>>>> It would be best to have 4.2 upstream and this[1] run to really
>>>>> minimize the noise.
>>>>>
>>>>> Thanks,
>>>>> michal
>>>>>
>>>>> [1]
>>>>> http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
demand-el7-x86_64/28/
>>>>>
>>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com>
wrote:
>>>>>
>>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> I will update with the results of the next tier1 execution on
latest
>>>>> 4.2.3
>>>>>
>>>>>
>>>>> That isn't master but old branch though. Could you run it
against
>>>>> *current* VDSM master?
>>>>>
>>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik <
mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> Hi, I've triggered another execution [1] due to some issues I
saw in
>>>>> the
>>>>> first which are not related to the patch.
>>>>>
>>>>> The success rate is 78% which is low comparing to tier1 executions
with
>>>>> code from downstream builds (95-100% success rates) [2].
>>>>>
>>>>>
>>>>> Could you run the current master (without the dynamic_ownership
patch)
>>>>> so that we have viable comparision?
>>>>>
>>>>> From what I could see so far, there is an issue with move and copy
>>>>>
>>>>> operations to and from Gluster domains. For example [3].
>>>>>
>>>>> The logs are attached.
>>>>>
>>>>>
>>>>> [1]
>>>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>>>
>>>>>
>>>>>
>>>>> [2]
>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>>>
>>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>>>
>>>>>
>>>>>
>>>>> [3]
>>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>>>>> deleteImage error=Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> from=:
>>>>> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_
ba6b2630-5976-4935,
>>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>>>>> [storage.TaskManager.Task]
>>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
error
>>>>> (task:875)
>>>>> Traceback (most recent call last):
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
882,
>>>>> in
>>>>> _run
>>>>> return fn(*args, **kargs)
>>>>> File "<string>", line 2, in deleteImage
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
49, in
>>>>> method
>>>>> ret = func(*args, **kwargs)
>>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
1503,
>>>>> in
>>>>> deleteImage
>>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>>>> ImageDoesNotExistInSD: Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>>
>>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>>>>> [storage.TaskManager.Task]
>>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task
is
>>>>> aborted:
>>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-
>>>>> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'"
- code
268
>>>>> (task:1181)
>>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher]
>>>>> FINISH
>>>>> deleteImage error=Image does not exist in domain:
>>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> domain=e5fd29c8-52ba-467e-be09
>>>>> -ca40ff054d
>>>>> d4' (dispatcher:82)
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <
ebenahar(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> Triggered a sanity tier1 execution [1] using [2], which covers all
the
>>>>>
>>>>> requested areas, on iSCSI, NFS and Gluster.
>>>>> I'll update with the results.
>>>>>
>>>>> [1]
>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>>
>>>>> [2]
>>>>> https://gerrit.ovirt.org/#/c/89830/
>>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <
mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> Hi Martin,
>>>>>
>>>>>
>>>>> I see [1] requires a rebase, can you please take care?
>>>>>
>>>>>
>>>>> Should be rebased.
>>>>>
>>>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster
and
>>>>>
>>>>> FC.
>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>>> not
>>>>> stable enough at the moment.
>>>>>
>>>>>
>>>>> That is still pretty good.
>>>>>
>>>>>
>>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <
mpolednik(a)redhat.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> Hi, sorry if I misunderstood, I waited for more input regarding
what
>>>>>
>>>>> areas
>>>>> have to be tested here.
>>>>>
>>>>>
>>>>> I'd say that you have quite a bit of freedom in this regard.
>>>>>
>>>>> GlusterFS
>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>> and merging them, and whatever else would be important for storage
>>>>> sanity.
>>>>>
>>>>> mpolednik
>>>>>
>>>>>
>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>> mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>
>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
>>>>> cinder,
>>>>>
>>>>> will
>>>>>
>>>>> have to check, since usually, we don't execute our automation
on
>>>>> them.
>>>>>
>>>>>
>>>>> Any update on this? I believe the gluster tests were successful,
>>>>> OST
>>>>>
>>>>> passes fine and unit tests pass fine, that makes the storage
>>>>> backends
>>>>> test the last required piece.
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> +Elad
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>>>>>
>>>>> >
>>>>> wrote:
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Please make sure to run as much OST suites on this patch as
>>>>>
>>>>> possible
>>>>>
>>>>> before merging ( using 'ci please build' )
>>>>>
>>>>>
>>>>>
>>>>> But note that OST is not a way to verify the patch.
>>>>>
>>>>>
>>>>> Such changes require testing with all storage types we support.
>>>>>
>>>>> Nir
>>>>>
>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>> mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hey,
>>>>>
>>>>>
>>>>> I've created a patch[0] that is finally able to activate
>>>>>
>>>>> libvirt's
>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>> functionality of our storage code.
>>>>>
>>>>> That of course comes with quite a bit of code removal, mostly
>>>>> in
>>>>> the
>>>>> area of host devices, hwrng and anything that touches devices;
>>>>> bunch
>>>>> of test changes and one XML generation caveat (storage is
>>>>> handled
>>>>> by
>>>>> VDSM, therefore disk relabelling needs to be disabled on the
>>>>> VDSM
>>>>> level).
>>>>>
>>>>> Because of the scope of the patch, I welcome
>>>>> storage/virt/network
>>>>> people to review the code and consider the implication this
>>>>> change
>>>>> has
>>>>> on current/future features.
>>>>>
>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>>
>>>>> In particular: dynamic_ownership was set to 0 prehistorically
>>>>> (as
>>>>>
>>>>>
>>>>> part
>>>>>
>>>>>
>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>
>>>>> libvirt,
>>>>> running as root, was not able to play properly with root-squash
>>>>> nfs
>>>>> mounts.
>>>>>
>>>>> Have you attempted this use case?
>>>>>
>>>>> I join to Nir's request to run this with storage QE.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> Raz Tamir
>>>>> Manager, RHV QE
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
> <logs.tar.gz>
>
>
5:34 a.m.
New subject: dynamic ownership changes
On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <ebenahar(a)redhat.com> wrote:
Hi Dan,
In the last execution, the success rate was very low due to a large number
of failures on start VM caused, according to Michal, by the
vdsm-hook-allocate_net that was installed on the host.
This is the latest status here, would you like me to re-execute?
yes, of course. but you should rebase Polednik's code on top of
*current* ovirt-4.2.3 branch.
If so, with
or W/O vdsm-hook-allocate_net installed?
There was NO reason to have that installed. Please keep it (and any
other needless code) out of the test environment.
>
> On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
>>
>> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>> <michal.skrivanek(a)redhat.com> wrote:
>> > Hi Elad,
>> > why did you install vdsm-hook-allocate_net?
>> >
>> > adding Dan as I think the hook is not supposed to fail this badly in any
>> > case
>>
>> yep, this looks bad and deserves a little bug report. Installing this
>> little hook should not block vm startup.
>>
>> But more importantly - what is the conclusion of this thread? Do we
>> have a green light from QE to take this in?
>>
>>
>> >
>> > Thanks,
>> > michal
>> >
>> > On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com> wrote:
>> >
>> > Start VM fails on:
>> >
>> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
>> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda'
path:
>> >
>> >
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/images/6cdabfe5-
>> > d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211' ->
>> >
>> >
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-
>> > f35a01928211' (storagexml:334)
>> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
>> > getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>> > options=None)
>> > from=::ffff:10.35.161.127,53512,
>> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>> > 2758 (api:46)
>> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
>> > /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
>> > err=vm
>> > net allocation hook: [unexpected error]: Traceback (most recent call
>> > last):
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 105, in <module>
>> > main()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 93, in main
>> > allocate_random_network(device_xml)
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 62, in allocate_random_network
>> > net = _get_random_network()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 50, in _get_random_network
>> > available_nets = _parse_nets()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 46, in _parse_nets
>> > return [net for net in os.environ[AVAIL_NETS_KEY].split()]
>> > File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
>> > raise KeyError(key)
>> > KeyError: 'equivnets'
>> >
>> >
>> > (hooks:110)
>> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
>> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process
>> > failed
>> > (vm:943)
>> > Traceback (most recent call last):
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
872, in
>> > _startUnderlyingVm
>> > self._run()
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2861, in
>> > _run
>> > domxml = hooks.before_vm_start(self._buildDomainXML(),
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2254, in
>> > _buildDomainXML
>> > dom, self.id, self._custom['custom'])
>> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py",
>> > line 240, in replace_device_xml_with_hooks_xml
>> > dev_custom)
>> > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
line 134,
>> > in
>> > before_device_create
>> > params=customProperties)
>> > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
line 120,
>> > in
>> > _runHooksDir
>> > raise exception.HookError(err)
>> > HookError: Hook Error: ('vm net allocation hook: [unexpected error]:
>> > Traceback (most recent call last):\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
>> > 105, in
>> > <module>\n main()\n
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 93, in main\n allocate_random_network(device_xml)\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line 62,
>> > i
>> > n allocate_random_network\n net = _get_random_network()\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line 50,
>> > in
>> > _get_random_network\n available_nets = _parse_nets()\n File "/us
>> > r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46,
in
>> > _parse_nets\n return [net for net in
>> > os.environ[AVAIL_NETS_KEY].split()]\n File
>> > "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
>> > em__\n raise KeyError(key)\nKeyError: \'equivnets\'\n\n\n',)
>> >
>> >
>> >
>> > Hence, the success rate was 28% against 100% running with d/s (d/s). If
>> > needed, I'll compare against the latest master, but I think you get the
>> > picture with d/s.
>> >
>> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> > libvirt-3.9.0-14.el7_5.3.x86_64
>> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>> > kernel 3.10.0-862.el7.x86_64
>> > rhel7.5
>> >
>> >
>> > Logs attached
>> >
>> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> > wrote:
>> >>
>> >> nvm, found gluster 3.12 repo, managed to install vdsm
>> >>
>> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> >> wrote:
>> >>>
>> >>> No, vdsm requires it:
>> >>>
>> >>> Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>> >>> Requires: glusterfs-fuse >= 3.12
>> >>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64
(@rhv-4.2.3)
>> >>>
>> >>> Therefore, vdsm package installation is skipped upon force install.
>> >>>
>> >>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>> >>> <michal.skrivanek(a)redhat.com> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com> wrote:
>> >>>>
>> >>>> Hi guys,
>> >>>>
>> >>>> The vdsm build from the patch requires glusterfs-fuse > 3.12.
This is
>> >>>> while the latest 4.2.3-5 d/s build requires 3.8.4
(3.4.0.59rhs-1.el7)
>> >>>>
>> >>>>
>> >>>> because it is still oVirt, not a downstream build. We can’t
really do
>> >>>> downstream builds with unmerged changes:/
>> >>>>
>> >>>> Trying to get this gluster-fuse build, so far no luck.
>> >>>> Is this requirement intentional?
>> >>>>
>> >>>>
>> >>>> it should work regardless, I guess you can force install it
without
>> >>>> the
>> >>>> dependency
>> >>>>
>> >>>>
>> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>> >>>> <michal.skrivanek(a)redhat.com> wrote:
>> >>>>>
>> >>>>> Hi Elad,
>> >>>>> to make it easier to compare, Martin backported the change
to 4.2 so
>> >>>>> it
>> >>>>> is actually comparable with a run without that patch. Would
you
>> >>>>> please try
>> >>>>> that out?
>> >>>>> It would be best to have 4.2 upstream and this[1] run to
really
>> >>>>> minimize the noise.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> michal
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_6...
>> >>>>>
>> >>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>> I will update with the results of the next tier1 execution
on latest
>> >>>>> 4.2.3
>> >>>>>
>> >>>>>
>> >>>>> That isn't master but old branch though. Could you run
it against
>> >>>>> *current* VDSM master?
>> >>>>>
>> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>> Hi, I've triggered another execution [1] due to some
issues I saw in
>> >>>>> the
>> >>>>> first which are not related to the patch.
>> >>>>>
>> >>>>> The success rate is 78% which is low comparing to tier1
executions
>> >>>>> with
>> >>>>> code from downstream builds (95-100% success rates) [2].
>> >>>>>
>> >>>>>
>> >>>>> Could you run the current master (without the
dynamic_ownership
>> >>>>> patch)
>> >>>>> so that we have viable comparision?
>> >>>>>
>> >>>>> From what I could see so far, there is an issue with move
and copy
>> >>>>>
>> >>>>> operations to and from Gluster domains. For example [3].
>> >>>>>
>> >>>>> The logs are attached.
>> >>>>>
>> >>>>>
>> >>>>> [1]
>> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> [2]
>> >>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>> >>>>>
>> >>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> [3]
>> >>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api]
FINISH
>> >>>>> deleteImage error=Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >>>>> from=:
>> >>>>> :ffff:10.35.161.182,40936,
>> >>>>> flow_id=disks_syncAction_ba6b2630-5976-4935,
>> >>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>> >>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>> >>>>> [storage.TaskManager.Task]
>> >>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42')
Unexpected error
>> >>>>> (task:875)
>> >>>>> Traceback (most recent call last):
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
>> >>>>> 882,
>> >>>>> in
>> >>>>> _run
>> >>>>> return fn(*args, **kargs)
>> >>>>> File "<string>", line 2, in deleteImage
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49,
>> >>>>> in
>> >>>>> method
>> >>>>> ret = func(*args, **kwargs)
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
>> >>>>> 1503,
>> >>>>> in
>> >>>>> deleteImage
>> >>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> >>>>> ImageDoesNotExistInSD: Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >>>>>
>> >>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>> >>>>> [storage.TaskManager.Task]
>> >>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42')
aborting: Task is
>> >>>>> aborted:
>> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-
>> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code
>> >>>>> 268
>> >>>>> (task:1181)
>> >>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7)
[storage.Dispatcher]
>> >>>>> FINISH
>> >>>>> deleteImage error=Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09
>> >>>>> -ca40ff054d
>> >>>>> d4' (dispatcher:82)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon
>> >>>>> <ebenahar(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> Triggered a sanity tier1 execution [1] using [2], which
covers all
>> >>>>> the
>> >>>>>
>> >>>>> requested areas, on iSCSI, NFS and Gluster.
>> >>>>> I'll update with the results.
>> >>>>>
>> >>>>> [1]
>> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>> >>>>>
>> >>>>> [2]
>> >>>>> https://gerrit.ovirt.org/#/c/89830/
>> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hi Martin,
>> >>>>>
>> >>>>>
>> >>>>> I see [1] requires a rebase, can you please take care?
>> >>>>>
>> >>>>>
>> >>>>> Should be rebased.
>> >>>>>
>> >>>>> At the moment, our automation is stable only on iSCSI, NFS,
Gluster
>> >>>>> and
>> >>>>>
>> >>>>> FC.
>> >>>>> Ceph is not supported and Cinder will be stabilized soon,
AFAIR,
>> >>>>> it's
>> >>>>> not
>> >>>>> stable enough at the moment.
>> >>>>>
>> >>>>>
>> >>>>> That is still pretty good.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Thanks
>> >>>>>
>> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com
>> >>>>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hi, sorry if I misunderstood, I waited for more input
regarding what
>> >>>>>
>> >>>>> areas
>> >>>>> have to be tested here.
>> >>>>>
>> >>>>>
>> >>>>> I'd say that you have quite a bit of freedom in this
regard.
>> >>>>>
>> >>>>> GlusterFS
>> >>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with
some
>> >>>>> suite
>> >>>>> that covers basic operations (start & stop VM, migrate
it),
>> >>>>> snapshots
>> >>>>> and merging them, and whatever else would be important for
storage
>> >>>>> sanity.
>> >>>>>
>> >>>>> mpolednik
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>> >>>>> mpolednik(a)redhat.com
>> >>>>> >
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>> >>>>> cinder,
>> >>>>>
>> >>>>> will
>> >>>>>
>> >>>>> have to check, since usually, we don't execute our
automation on
>> >>>>> them.
>> >>>>>
>> >>>>>
>> >>>>> Any update on this? I believe the gluster tests were
successful,
>> >>>>> OST
>> >>>>>
>> >>>>> passes fine and unit tests pass fine, that makes the
storage
>> >>>>> backends
>> >>>>> test the last required piece.
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> +Elad
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>> >>>>>
>> >>>>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Please make sure to run as much OST suites on this patch as
>> >>>>>
>> >>>>> possible
>> >>>>>
>> >>>>> before merging ( using 'ci please build' )
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> But note that OST is not a way to verify the patch.
>> >>>>>
>> >>>>>
>> >>>>> Such changes require testing with all storage types we
support.
>> >>>>>
>> >>>>> Nir
>> >>>>>
>> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>> >>>>> mpolednik(a)redhat.com
>> >>>>> >
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hey,
>> >>>>>
>> >>>>>
>> >>>>> I've created a patch[0] that is finally able to
activate
>> >>>>>
>> >>>>> libvirt's
>> >>>>> dynamic_ownership for VDSM while not negatively affecting
>> >>>>> functionality of our storage code.
>> >>>>>
>> >>>>> That of course comes with quite a bit of code removal,
mostly
>> >>>>> in
>> >>>>> the
>> >>>>> area of host devices, hwrng and anything that touches
devices;
>> >>>>> bunch
>> >>>>> of test changes and one XML generation caveat (storage is
>> >>>>> handled
>> >>>>> by
>> >>>>> VDSM, therefore disk relabelling needs to be disabled on
the
>> >>>>> VDSM
>> >>>>> level).
>> >>>>>
>> >>>>> Because of the scope of the patch, I welcome
>> >>>>> storage/virt/network
>> >>>>> people to review the code and consider the implication this
>> >>>>> change
>> >>>>> has
>> >>>>> on current/future features.
>> >>>>>
>> >>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>> >>>>>
>> >>>>>
>> >>>>> In particular: dynamic_ownership was set to 0
prehistorically
>> >>>>> (as
>> >>>>>
>> >>>>>
>> >>>>> part
>> >>>>>
>> >>>>>
>> >>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
because
>> >>>>>
>> >>>>> libvirt,
>> >>>>> running as root, was not able to play properly with
root-squash
>> >>>>> nfs
>> >>>>> mounts.
>> >>>>>
>> >>>>> Have you attempted this use case?
>> >>>>>
>> >>>>> I join to Nir's request to run this with storage QE.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Raz Tamir
>> >>>>> Manager, RHV QE
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Devel mailing list
>> >>>>> Devel(a)ovirt.org
>> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>> > <logs.tar.gz>
>> >
>> >
>
>
7:30 a.m.
New subject: dynamic ownership changes
Hi Martin,
Can you please create a cerry pick patch that is based on 4.2?
Thanks
On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
wrote:
> Hi Dan,
>
> In the last execution, the success rate was very low due to a large
number
> of failures on start VM caused, according to Michal, by the
> vdsm-hook-allocate_net that was installed on the host.
>
> This is the latest status here, would you like me to re-execute?
yes, of course. but you should rebase Polednik's code on top of
*current* ovirt-4.2.3 branch.
> If so, with
> or W/O vdsm-hook-allocate_net installed?
There was NO reason to have that installed. Please keep it (and any
other needless code) out of the test environment.
>
> On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <danken(a)redhat.com>
wrote:
>>
>> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>> <michal.skrivanek(a)redhat.com> wrote:
>> > Hi Elad,
>> > why did you install vdsm-hook-allocate_net?
>> >
>> > adding Dan as I think the hook is not supposed to fail this badly in
any
>> > case
>>
>> yep, this looks bad and deserves a little bug report. Installing this
>> little hook should not block vm startup.
>>
>> But more importantly - what is the conclusion of this thread? Do we
>> have a green light from QE to take this in?
>>
>>
>> >
>> > Thanks,
>> > michal
>> >
>> > On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
>> >
>> > Start VM fails on:
>> >
>> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
>> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda'
path:
>> >
>> > 'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
9a78-bdd13a843c62/images/6cdabfe5-
>> > d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
->
>> >
>> > u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
9834f235d1c8/7ef97445-30e6-4435-8425-
>> > f35a01928211' (storagexml:334)
>> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
>> > getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>> > options=None)
>> > from=::ffff:10.35.161.127,53512,
>> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>> > 2758 (api:46)
>> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
>> > /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
>> > err=vm
>> > net allocation hook: [unexpected error]: Traceback (most recent call
>> > last):
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 105, in <module>
>> > main()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 93, in main
>> > allocate_random_network(device_xml)
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 62, in allocate_random_network
>> > net = _get_random_network()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 50, in _get_random_network
>> > available_nets = _parse_nets()
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 46, in _parse_nets
>> > return [net for net in os.environ[AVAIL_NETS_KEY].split()]
>> > File "/usr/lib64/python2.7/UserDict.py", line 23, in
__getitem__
>> > raise KeyError(key)
>> > KeyError: 'equivnets'
>> >
>> >
>> > (hooks:110)
>> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
>> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process
>> > failed
>> > (vm:943)
>> > Traceback (most recent call last):
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
872,
in
>> > _startUnderlyingVm
>> > self._run()
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2861,
in
>> > _run
>> > domxml = hooks.before_vm_start(self._buildDomainXML(),
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2254,
in
>> > _buildDomainXML
>> > dom, self.id, self._custom['custom'])
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
preprocess.py",
>> > line 240, in replace_device_xml_with_hooks_xml
>> > dev_custom)
>> > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
line
134,
>> > in
>> > before_device_create
>> > params=customProperties)
>> > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
line
120,
>> > in
>> > _runHooksDir
>> > raise exception.HookError(err)
>> > HookError: Hook Error: ('vm net allocation hook: [unexpected error]:
>> > Traceback (most recent call last):\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
>> > 105, in
>> > <module>\n main()\n
>> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> > line
>> > 93, in main\n allocate_random_network(device_xml)\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
62,
>> > i
>> > n allocate_random_network\n net = _get_random_network()\n File
>> > "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
50,
>> > in
>> > _get_random_network\n available_nets = _parse_nets()\n File "/us
>> > r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46,
in
>> > _parse_nets\n return [net for net in
>> > os.environ[AVAIL_NETS_KEY].split()]\n File
>> > "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
>> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>> >
>> >
>> >
>> > Hence, the success rate was 28% against 100% running with d/s (d/s).
If
>> > needed, I'll compare against the latest master, but I think you get
the
>> > picture with d/s.
>> >
>> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> > libvirt-3.9.0-14.el7_5.3.x86_64
>> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>> > kernel 3.10.0-862.el7.x86_64
>> > rhel7.5
>> >
>> >
>> > Logs attached
>> >
>> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> > wrote:
>> >>
>> >> nvm, found gluster 3.12 repo, managed to install vdsm
>> >>
>> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon
<ebenahar(a)redhat.com
>
>> >> wrote:
>> >>>
>> >>> No, vdsm requires it:
>> >>>
>> >>> Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>> >>> Requires: glusterfs-fuse >= 3.12
>> >>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64
(@rhv-4.2.3)
>> >>>
>> >>> Therefore, vdsm package installation is skipped upon force
install.
>> >>>
>> >>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>> >>> <michal.skrivanek(a)redhat.com> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com>
wrote:
>> >>>>
>> >>>> Hi guys,
>> >>>>
>> >>>> The vdsm build from the patch requires glusterfs-fuse >
3.12. This
is
>> >>>> while the latest 4.2.3-5 d/s build requires 3.8.4
(3.4.0.59rhs-1.el7)
>> >>>>
>> >>>>
>> >>>> because it is still oVirt, not a downstream build. We can’t
really
do
>> >>>> downstream builds with unmerged changes:/
>> >>>>
>> >>>> Trying to get this gluster-fuse build, so far no luck.
>> >>>> Is this requirement intentional?
>> >>>>
>> >>>>
>> >>>> it should work regardless, I guess you can force install it
without
>> >>>> the
>> >>>> dependency
>> >>>>
>> >>>>
>> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>> >>>> <michal.skrivanek(a)redhat.com> wrote:
>> >>>>>
>> >>>>> Hi Elad,
>> >>>>> to make it easier to compare, Martin backported the change
to 4.2
so
>> >>>>> it
>> >>>>> is actually comparable with a run without that patch. Would
you
>> >>>>> please try
>> >>>>> that out?
>> >>>>> It would be best to have 4.2 upstream and this[1] run to
really
>> >>>>> minimize the noise.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> michal
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> >>>>> http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
demand-el7-x86_64/28/
>> >>>>>
>> >>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>> I will update with the results of the next tier1 execution
on
latest
>> >>>>> 4.2.3
>> >>>>>
>> >>>>>
>> >>>>> That isn't master but old branch though. Could you run
it against
>> >>>>> *current* VDSM master?
>> >>>>>
>> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>> Hi, I've triggered another execution [1] due to some
issues I saw
in
>> >>>>> the
>> >>>>> first which are not related to the patch.
>> >>>>>
>> >>>>> The success rate is 78% which is low comparing to tier1
executions
>> >>>>> with
>> >>>>> code from downstream builds (95-100% success rates) [2].
>> >>>>>
>> >>>>>
>> >>>>> Could you run the current master (without the
dynamic_ownership
>> >>>>> patch)
>> >>>>> so that we have viable comparision?
>> >>>>>
>> >>>>> From what I could see so far, there is an issue with move
and copy
>> >>>>>
>> >>>>> operations to and from Gluster domains. For example [3].
>> >>>>>
>> >>>>> The logs are attached.
>> >>>>>
>> >>>>>
>> >>>>> [1]
>> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> [2]
>> >>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>> >>>>>
>> >>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> [3]
>> >>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api]
FINISH
>> >>>>> deleteImage error=Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >>>>> from=:
>> >>>>> :ffff:10.35.161.182,40936,
>> >>>>> flow_id=disks_syncAction_ba6b2630-5976-4935,
>> >>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>> >>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>> >>>>> [storage.TaskManager.Task]
>> >>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42')
Unexpected error
>> >>>>> (task:875)
>> >>>>> Traceback (most recent call last):
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
line
>> >>>>> 882,
>> >>>>> in
>> >>>>> _run
>> >>>>> return fn(*args, **kargs)
>> >>>>> File "<string>", line 2, in deleteImage
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
49,
>> >>>>> in
>> >>>>> method
>> >>>>> ret = func(*args, **kwargs)
>> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
>> >>>>> 1503,
>> >>>>> in
>> >>>>> deleteImage
>> >>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> >>>>> ImageDoesNotExistInSD: Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >>>>>
>> >>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>> >>>>> [storage.TaskManager.Task]
>> >>>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42')
aborting: Task is
>> >>>>> aborted:
>> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-
>> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" -
code
>> >>>>> 268
>> >>>>> (task:1181)
>> >>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7)
[storage.Dispatcher]
>> >>>>> FINISH
>> >>>>> deleteImage error=Image does not exist in domain:
>> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >>>>> domain=e5fd29c8-52ba-467e-be09
>> >>>>> -ca40ff054d
>> >>>>> d4' (dispatcher:82)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon
>> >>>>> <ebenahar(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> Triggered a sanity tier1 execution [1] using [2], which
covers all
>> >>>>> the
>> >>>>>
>> >>>>> requested areas, on iSCSI, NFS and Gluster.
>> >>>>> I'll update with the results.
>> >>>>>
>> >>>>> [1]
>> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>> >>>>>
>> >>>>> [2]
>> >>>>> https://gerrit.ovirt.org/#/c/89830/
>> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hi Martin,
>> >>>>>
>> >>>>>
>> >>>>> I see [1] requires a rebase, can you please take care?
>> >>>>>
>> >>>>>
>> >>>>> Should be rebased.
>> >>>>>
>> >>>>> At the moment, our automation is stable only on iSCSI,
NFS,
Gluster
>> >>>>> and
>> >>>>>
>> >>>>> FC.
>> >>>>> Ceph is not supported and Cinder will be stabilized soon,
AFAIR,
>> >>>>> it's
>> >>>>> not
>> >>>>> stable enough at the moment.
>> >>>>>
>> >>>>>
>> >>>>> That is still pretty good.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Thanks
>> >>>>>
>> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
>> >>>>> <mpolednik(a)redhat.com
>> >>>>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hi, sorry if I misunderstood, I waited for more input
regarding
what
>> >>>>>
>> >>>>> areas
>> >>>>> have to be tested here.
>> >>>>>
>> >>>>>
>> >>>>> I'd say that you have quite a bit of freedom in this
regard.
>> >>>>>
>> >>>>> GlusterFS
>> >>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with
some
>> >>>>> suite
>> >>>>> that covers basic operations (start & stop VM, migrate
it),
>> >>>>> snapshots
>> >>>>> and merging them, and whatever else would be important for
storage
>> >>>>> sanity.
>> >>>>>
>> >>>>> mpolednik
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>> >>>>> mpolednik(a)redhat.com
>> >>>>> >
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>> >>>>>
>> >>>>>
>> >>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>> >>>>> cinder,
>> >>>>>
>> >>>>> will
>> >>>>>
>> >>>>> have to check, since usually, we don't execute our
automation on
>> >>>>> them.
>> >>>>>
>> >>>>>
>> >>>>> Any update on this? I believe the gluster tests were
successful,
>> >>>>> OST
>> >>>>>
>> >>>>> passes fine and unit tests pass fine, that makes the
storage
>> >>>>> backends
>> >>>>> test the last required piece.
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> +Elad
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <
danken(a)redhat.com
>> >>>>>
>> >>>>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Please make sure to run as much OST suites on this patch
as
>> >>>>>
>> >>>>> possible
>> >>>>>
>> >>>>> before merging ( using 'ci please build' )
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> But note that OST is not a way to verify the patch.
>> >>>>>
>> >>>>>
>> >>>>> Such changes require testing with all storage types we
support.
>> >>>>>
>> >>>>> Nir
>> >>>>>
>> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>> >>>>> mpolednik(a)redhat.com
>> >>>>> >
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hey,
>> >>>>>
>> >>>>>
>> >>>>> I've created a patch[0] that is finally able to
activate
>> >>>>>
>> >>>>> libvirt's
>> >>>>> dynamic_ownership for VDSM while not negatively affecting
>> >>>>> functionality of our storage code.
>> >>>>>
>> >>>>> That of course comes with quite a bit of code removal,
mostly
>> >>>>> in
>> >>>>> the
>> >>>>> area of host devices, hwrng and anything that touches
devices;
>> >>>>> bunch
>> >>>>> of test changes and one XML generation caveat (storage is
>> >>>>> handled
>> >>>>> by
>> >>>>> VDSM, therefore disk relabelling needs to be disabled on
the
>> >>>>> VDSM
>> >>>>> level).
>> >>>>>
>> >>>>> Because of the scope of the patch, I welcome
>> >>>>> storage/virt/network
>> >>>>> people to review the code and consider the implication
this
>> >>>>> change
>> >>>>> has
>> >>>>> on current/future features.
>> >>>>>
>> >>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>> >>>>>
>> >>>>>
>> >>>>> In particular: dynamic_ownership was set to 0
prehistorically
>> >>>>> (as
>> >>>>>
>> >>>>>
>> >>>>> part
>> >>>>>
>> >>>>>
>> >>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
because
>> >>>>>
>> >>>>> libvirt,
>> >>>>> running as root, was not able to play properly with
root-squash
>> >>>>> nfs
>> >>>>> mounts.
>> >>>>>
>> >>>>> Have you attempted this use case?
>> >>>>>
>> >>>>> I join to Nir's request to run this with storage QE.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Raz Tamir
>> >>>>> Manager, RHV QE
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Devel mailing list
>> >>>>> Devel(a)ovirt.org
>> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>> > <logs.tar.gz>
>> >
>> >
>
>
8:26 a.m.
New subject: dynamic ownership changes
On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
Hi Martin,
Can you please create a cerry pick patch that is based on 4.2?
See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
(storage needs real env).
mpolednik
>
>Thanks
>
>On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
>
>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>> > Hi Dan,
>> >
>> > In the last execution, the success rate was very low due to a large
>> number
>> > of failures on start VM caused, according to Michal, by the
>> > vdsm-hook-allocate_net that was installed on the host.
>> >
>> > This is the latest status here, would you like me to re-execute?
>>
>> yes, of course. but you should rebase Polednik's code on top of
>> *current* ovirt-4.2.3 branch.
>>
>> > If so, with
>> > or W/O vdsm-hook-allocate_net installed?
>>
>> There was NO reason to have that installed. Please keep it (and any
>> other needless code) out of the test environment.
>>
>> >
>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>> >>
>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>> >> <michal.skrivanek(a)redhat.com> wrote:
>> >> > Hi Elad,
>> >> > why did you install vdsm-hook-allocate_net?
>> >> >
>> >> > adding Dan as I think the hook is not supposed to fail this badly
in
>> any
>> >> > case
>> >>
>> >> yep, this looks bad and deserves a little bug report. Installing this
>> >> little hook should not block vm startup.
>> >>
>> >> But more importantly - what is the conclusion of this thread? Do we
>> >> have a green light from QE to take this in?
>> >>
>> >>
>> >> >
>> >> > Thanks,
>> >> > michal
>> >> >
>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com> wrote:
>> >> >
>> >> > Start VM fails on:
>> >> >
>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive
'vda' path:
>> >> >
>> >> > 'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>> 9a78-bdd13a843c62/images/6cdabfe5-
>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211' ->
>> >> >
>> >> > u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>> >> > f35a01928211' (storagexml:334)
>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>> >> > options=None)
>> >> > from=::ffff:10.35.161.127,53512,
>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>> >> > 2758 (api:46)
>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
>> >> > /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
>> >> > err=vm
>> >> > net allocation hook: [unexpected error]: Traceback (most recent
call
>> >> > last):
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 105, in <module>
>> >> > main()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 93, in main
>> >> > allocate_random_network(device_xml)
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 62, in allocate_random_network
>> >> > net = _get_random_network()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 50, in _get_random_network
>> >> > available_nets = _parse_nets()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 46, in _parse_nets
>> >> > return [net for net in os.environ[AVAIL_NETS_KEY].split()]
>> >> > File "/usr/lib64/python2.7/UserDict.py", line 23, in
__getitem__
>> >> > raise KeyError(key)
>> >> > KeyError: 'equivnets'
>> >> >
>> >> >
>> >> > (hooks:110)
>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start
process
>> >> > failed
>> >> > (vm:943)
>> >> > Traceback (most recent call last):
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 872,
>> in
>> >> > _startUnderlyingVm
>> >> > self._run()
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 2861,
>> in
>> >> > _run
>> >> > domxml = hooks.before_vm_start(self._buildDomainXML(),
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 2254,
>> in
>> >> > _buildDomainXML
>> >> > dom, self.id, self._custom['custom'])
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>> preprocess.py",
>> >> > line 240, in replace_device_xml_with_hooks_xml
>> >> > dev_custom)
>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
>> 134,
>> >> > in
>> >> > before_device_create
>> >> > params=customProperties)
>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
>> 120,
>> >> > in
>> >> > _runHooksDir
>> >> > raise exception.HookError(err)
>> >> > HookError: Hook Error: ('vm net allocation hook: [unexpected
error]:
>> >> > Traceback (most recent call last):\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>> >> > 105, in
>> >> > <module>\n main()\n
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> >> > line
>> >> > 93, in main\n allocate_random_network(device_xml)\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>> 62,
>> >> > i
>> >> > n allocate_random_network\n net = _get_random_network()\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>> 50,
>> >> > in
>> >> > _get_random_network\n available_nets = _parse_nets()\n File
"/us
>> >> > r/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line 46,
>> in
>> >> > _parse_nets\n return [net for net in
>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
>> >> > "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>> >> >
>> >> >
>> >> >
>> >> > Hence, the success rate was 28% against 100% running with d/s
(d/s).
>> If
>> >> > needed, I'll compare against the latest master, but I think you
get
>> the
>> >> > picture with d/s.
>> >> >
>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>> >> > kernel 3.10.0-862.el7.x86_64
>> >> > rhel7.5
>> >> >
>> >> >
>> >> > Logs attached
>> >> >
>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> nvm, found gluster 3.12 repo, managed to install vdsm
>> >> >>
>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon
<ebenahar(a)redhat.com
>> >
>> >> >> wrote:
>> >> >>>
>> >> >>> No, vdsm requires it:
>> >> >>>
>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>> >> >>> Requires: glusterfs-fuse >= 3.12
>> >> >>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64
>> (@rhv-4.2.3)
>> >> >>>
>> >> >>> Therefore, vdsm package installation is skipped upon force
install.
>> >> >>>
>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>> >> >>> <michal.skrivanek(a)redhat.com> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> wrote:
>> >> >>>>
>> >> >>>> Hi guys,
>> >> >>>>
>> >> >>>> The vdsm build from the patch requires glusterfs-fuse
> 3.12. This
>> is
>> >> >>>> while the latest 4.2.3-5 d/s build requires 3.8.4
>> (3.4.0.59rhs-1.el7)
>> >> >>>>
>> >> >>>>
>> >> >>>> because it is still oVirt, not a downstream build. We
can’t really
>> do
>> >> >>>> downstream builds with unmerged changes:/
>> >> >>>>
>> >> >>>> Trying to get this gluster-fuse build, so far no luck.
>> >> >>>> Is this requirement intentional?
>> >> >>>>
>> >> >>>>
>> >> >>>> it should work regardless, I guess you can force
install it without
>> >> >>>> the
>> >> >>>> dependency
>> >> >>>>
>> >> >>>>
>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>> >> >>>> <michal.skrivanek(a)redhat.com> wrote:
>> >> >>>>>
>> >> >>>>> Hi Elad,
>> >> >>>>> to make it easier to compare, Martin backported the
change to 4.2
>> so
>> >> >>>>> it
>> >> >>>>> is actually comparable with a run without that
patch. Would you
>> >> >>>>> please try
>> >> >>>>> that out?
>> >> >>>>> It would be best to have 4.2 upstream and this[1]
run to really
>> >> >>>>> minimize the noise.
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>> michal
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>> demand-el7-x86_64/28/
>> >> >>>>>
>> >> >>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>> I will update with the results of the next tier1
execution on
>> latest
>> >> >>>>> 4.2.3
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> That isn't master but old branch though. Could
you run it against
>> >> >>>>> *current* VDSM master?
>> >> >>>>>
>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>> Hi, I've triggered another execution [1] due to
some issues I saw
>> in
>> >> >>>>> the
>> >> >>>>> first which are not related to the patch.
>> >> >>>>>
>> >> >>>>> The success rate is 78% which is low comparing to
tier1 executions
>> >> >>>>> with
>> >> >>>>> code from downstream builds (95-100% success rates)
[2].
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Could you run the current master (without the
dynamic_ownership
>> >> >>>>> patch)
>> >> >>>>> so that we have viable comparision?
>> >> >>>>>
>> >> >>>>> From what I could see so far, there is an issue
with move and copy
>> >> >>>>>
>> >> >>>>> operations to and from Gluster domains. For example
[3].
>> >> >>>>>
>> >> >>>>> The logs are attached.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> >> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [2]
>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>> >> >>>>>
>> >> >>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [3]
>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7)
[vdsm.api] FINISH
>> >> >>>>> deleteImage error=Image does not exist in domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >> >>>>> from=:
>> >> >>>>> :ffff:10.35.161.182,40936,
>> >> >>>>> flow_id=disks_syncAction_ba6b2630-5976-4935,
>> >> >>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42
(api:51)
>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>> >> >>>>> [storage.TaskManager.Task]
>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>> >> >>>>> (task:875)
>> >> >>>>> Traceback (most recent call last):
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
>> line
>> >> >>>>> 882,
>> >> >>>>> in
>> >> >>>>> _run
>> >> >>>>> return fn(*args, **kargs)
>> >> >>>>> File "<string>", line 2, in
deleteImage
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
>> 49,
>> >> >>>>> in
>> >> >>>>> method
>> >> >>>>> ret = func(*args, **kwargs)
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
>> >> >>>>> 1503,
>> >> >>>>> in
>> >> >>>>> deleteImage
>> >> >>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> >> >>>>> ImageDoesNotExistInSD: Image does not exist in
domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >> >>>>>
>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>> >> >>>>> [storage.TaskManager.Task]
>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is
>> >> >>>>> aborted:
>> >> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-
>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" -
>> code
>> >> >>>>> 268
>> >> >>>>> (task:1181)
>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7)
>> [storage.Dispatcher]
>> >> >>>>> FINISH
>> >> >>>>> deleteImage error=Image does not exist in domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
>> >> >>>>> -ca40ff054d
>> >> >>>>> d4' (dispatcher:82)
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon
>> >> >>>>> <ebenahar(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> Triggered a sanity tier1 execution [1] using [2],
which covers all
>> >> >>>>> the
>> >> >>>>>
>> >> >>>>> requested areas, on iSCSI, NFS and Gluster.
>> >> >>>>> I'll update with the results.
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> >> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>> >> >>>>>
>> >> >>>>> [2]
>> >> >>>>> https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hi Martin,
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I see [1] requires a rebase, can you please take
care?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Should be rebased.
>> >> >>>>>
>> >> >>>>> At the moment, our automation is stable only on
iSCSI, NFS,
>> Gluster
>> >> >>>>> and
>> >> >>>>>
>> >> >>>>> FC.
>> >> >>>>> Ceph is not supported and Cinder will be stabilized
soon, AFAIR,
>> >> >>>>> it's
>> >> >>>>> not
>> >> >>>>> stable enough at the moment.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> That is still pretty good.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Thanks
>> >> >>>>>
>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hi, sorry if I misunderstood, I waited for more
input regarding
>> what
>> >> >>>>>
>> >> >>>>> areas
>> >> >>>>> have to be tested here.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I'd say that you have quite a bit of freedom in
this regard.
>> >> >>>>>
>> >> >>>>> GlusterFS
>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with some
>> >> >>>>> suite
>> >> >>>>> that covers basic operations (start & stop VM,
migrate it),
>> >> >>>>> snapshots
>> >> >>>>> and merging them, and whatever else would be
important for storage
>> >> >>>>> sanity.
>> >> >>>>>
>> >> >>>>> mpolednik
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik
<
>> >> >>>>> mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> We can test this on iSCSI, NFS and GlusterFS. As
for ceph and
>> >> >>>>> cinder,
>> >> >>>>>
>> >> >>>>> will
>> >> >>>>>
>> >> >>>>> have to check, since usually, we don't execute
our automation on
>> >> >>>>> them.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Any update on this? I believe the gluster tests
were successful,
>> >> >>>>> OST
>> >> >>>>>
>> >> >>>>> passes fine and unit tests pass fine, that makes
the storage
>> >> >>>>> backends
>> >> >>>>> test the last required piece.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> +Elad
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<
>> danken(a)redhat.com
>> >> >>>>>
>> >> >>>>> >
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Please make sure to run as much OST suites on this
patch as
>> >> >>>>>
>> >> >>>>> possible
>> >> >>>>>
>> >> >>>>> before merging ( using 'ci please build' )
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> But note that OST is not a way to verify the
patch.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Such changes require testing with all storage types
we support.
>> >> >>>>>
>> >> >>>>> Nir
>> >> >>>>>
>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<
>> >> >>>>> mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hey,
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I've created a patch[0] that is finally able to
activate
>> >> >>>>>
>> >> >>>>> libvirt's
>> >> >>>>> dynamic_ownership for VDSM while not negatively
affecting
>> >> >>>>> functionality of our storage code.
>> >> >>>>>
>> >> >>>>> That of course comes with quite a bit of code
removal, mostly
>> >> >>>>> in
>> >> >>>>> the
>> >> >>>>> area of host devices, hwrng and anything that
touches devices;
>> >> >>>>> bunch
>> >> >>>>> of test changes and one XML generation caveat
(storage is
>> >> >>>>> handled
>> >> >>>>> by
>> >> >>>>> VDSM, therefore disk relabelling needs to be
disabled on the
>> >> >>>>> VDSM
>> >> >>>>> level).
>> >> >>>>>
>> >> >>>>> Because of the scope of the patch, I welcome
>> >> >>>>> storage/virt/network
>> >> >>>>> people to review the code and consider the
implication this
>> >> >>>>> change
>> >> >>>>> has
>> >> >>>>> on current/future features.
>> >> >>>>>
>> >> >>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> In particular: dynamic_ownership was set to 0
prehistorically
>> >> >>>>> (as
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> part
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>> >> >>>>>
>> >> >>>>> libvirt,
>> >> >>>>> running as root, was not able to play properly with
root-squash
>> >> >>>>> nfs
>> >> >>>>> mounts.
>> >> >>>>>
>> >> >>>>> Have you attempted this use case?
>> >> >>>>>
>> >> >>>>> I join to Nir's request to run this with
storage QE.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> --
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Raz Tamir
>> >> >>>>> Manager, RHV QE
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> _______________________________________________
>> >> >>>>> Devel mailing list
>> >> >>>>> Devel(a)ovirt.org
>> >> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >
>> >> > <logs.tar.gz>
>> >> >
>> >> >
>> >
>> >
>>
9:44 a.m.
New subject: dynamic ownership changes
Triggered a sanity automation execution using [1], which covers all the
requested areas, on iSCSI, NFS and Gluster.
I'll update with the results.
[1]
*https://gerrit.ovirt.org/#/c/90906/ <https://gerrit.ovirt.org/#/c/90906/>*
vdsm-4.20.28-6.gitc23aef6.el7.x86_64
On Tue, May 29, 2018 at 4:26 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
> Hi Martin,
>
> Can you please create a cerry pick patch that is based on 4.2?
>
See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
(storage needs real env).
mpolednik
> Thanks
>
> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg <danken(a)redhat.com>
> wrote:
>
> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>> > Hi Dan,
>> >
>> > In the last execution, the success rate was very low due to a large
>> number
>> > of failures on start VM caused, according to Michal, by the
>> > vdsm-hook-allocate_net that was installed on the host.
>> >
>> > This is the latest status here, would you like me to re-execute?
>>
>> yes, of course. but you should rebase Polednik's code on top of
>> *current* ovirt-4.2.3 branch.
>>
>> > If so, with
>> > or W/O vdsm-hook-allocate_net installed?
>>
>> There was NO reason to have that installed. Please keep it (and any
>> other needless code) out of the test environment.
>>
>> >
>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>> >>
>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>> >> <michal.skrivanek(a)redhat.com> wrote:
>> >> > Hi Elad,
>> >> > why did you install vdsm-hook-allocate_net?
>> >> >
>> >> > adding Dan as I think the hook is not supposed to fail this badly
in
>> any
>> >> > case
>> >>
>> >> yep, this looks bad and deserves a little bug report. Installing this
>> >> little hook should not block vm startup.
>> >>
>> >> But more importantly - what is the conclusion of this thread? Do we
>> >> have a green light from QE to take this in?
>> >>
>> >>
>> >> >
>> >> > Thanks,
>> >> > michal
>> >> >
>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> wrote:
>> >> >
>> >> > Start VM fails on:
>> >> >
>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive
'vda' path:
>> >> >
>> >> > 'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>> 9a78-bdd13a843c62/images/6cdabfe5-
>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
>> ->
>> >> >
>> >> > u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>> >> > f35a01928211' (storagexml:334)
>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START
>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>> >> > options=None)
>> >> > from=::ffff:10.35.161.127,53512,
>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>> >> > 2758 (api:46)
>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
>> >> > /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
>> >> > err=vm
>> >> > net allocation hook: [unexpected error]: Traceback (most recent
call
>> >> > last):
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 105, in <module>
>> >> > main()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 93, in main
>> >> > allocate_random_network(device_xml)
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 62, in allocate_random_network
>> >> > net = _get_random_network()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 50, in _get_random_network
>> >> > available_nets = _parse_nets()
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 46, in _parse_nets
>> >> > return [net for net in os.environ[AVAIL_NETS_KEY].split()]
>> >> > File "/usr/lib64/python2.7/UserDict.py", line 23, in
__getitem__
>> >> > raise KeyError(key)
>> >> > KeyError: 'equivnets'
>> >> >
>> >> >
>> >> > (hooks:110)
>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start
process
>> >> > failed
>> >> > (vm:943)
>> >> > Traceback (most recent call last):
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 872,
>> in
>> >> > _startUnderlyingVm
>> >> > self._run()
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line
>> 2861,
>> in
>> >> > _run
>> >> > domxml = hooks.before_vm_start(self._buildDomainXML(),
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line
>> 2254,
>> in
>> >> > _buildDomainXML
>> >> > dom, self.id, self._custom['custom'])
>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>> preprocess.py",
>> >> > line 240, in replace_device_xml_with_hooks_xml
>> >> > dev_custom)
>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
>> 134,
>> >> > in
>> >> > before_device_create
>> >> > params=customProperties)
>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line
>> 120,
>> >> > in
>> >> > _runHooksDir
>> >> > raise exception.HookError(err)
>> >> > HookError: Hook Error: ('vm net allocation hook: [unexpected
error]:
>> >> > Traceback (most recent call last):\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> line
>> >> > 105, in
>> >> > <module>\n main()\n
>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>> net",
>> >> > line
>> >> > 93, in main\n allocate_random_network(device_xml)\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> line
>> 62,
>> >> > i
>> >> > n allocate_random_network\n net = _get_random_network()\n File
>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>> line
>> 50,
>> >> > in
>> >> > _get_random_network\n available_nets = _parse_nets()\n File
"/us
>> >> > r/libexec/vdsm/hooks/before_device_create/10_allocate_net",
line
>> 46,
>> in
>> >> > _parse_nets\n return [net for net in
>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
>> >> > "/usr/lib64/python2.7/UserDict.py", line 23, in __getit
>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>> >> >
>> >> >
>> >> >
>> >> > Hence, the success rate was 28% against 100% running with d/s
(d/s).
>> If
>> >> > needed, I'll compare against the latest master, but I think you
get
>> the
>> >> > picture with d/s.
>> >> >
>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>> >> > kernel 3.10.0-862.el7.x86_64
>> >> > rhel7.5
>> >> >
>> >> >
>> >> > Logs attached
>> >> >
>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <
>> ebenahar(a)redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> nvm, found gluster 3.12 repo, managed to install vdsm
>> >> >>
>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <
>> ebenahar(a)redhat.com
>> >
>> >> >> wrote:
>> >> >>>
>> >> >>> No, vdsm requires it:
>> >> >>>
>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>> >> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>> >> >>> Requires: glusterfs-fuse >= 3.12
>> >> >>> Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64
>> (@rhv-4.2.3)
>> >> >>>
>> >> >>> Therefore, vdsm package installation is skipped upon force
>> install.
>> >> >>>
>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>> >> >>> <michal.skrivanek(a)redhat.com> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com>
>> wrote:
>> >> >>>>
>> >> >>>> Hi guys,
>> >> >>>>
>> >> >>>> The vdsm build from the patch requires glusterfs-fuse
> 3.12.
>> This
>> is
>> >> >>>> while the latest 4.2.3-5 d/s build requires 3.8.4
>> (3.4.0.59rhs-1.el7)
>> >> >>>>
>> >> >>>>
>> >> >>>> because it is still oVirt, not a downstream build. We
can’t
>> really
>> do
>> >> >>>> downstream builds with unmerged changes:/
>> >> >>>>
>> >> >>>> Trying to get this gluster-fuse build, so far no luck.
>> >> >>>> Is this requirement intentional?
>> >> >>>>
>> >> >>>>
>> >> >>>> it should work regardless, I guess you can force
install it
>> without
>> >> >>>> the
>> >> >>>> dependency
>> >> >>>>
>> >> >>>>
>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>> >> >>>> <michal.skrivanek(a)redhat.com> wrote:
>> >> >>>>>
>> >> >>>>> Hi Elad,
>> >> >>>>> to make it easier to compare, Martin backported the
change to
>> 4.2
>> so
>> >> >>>>> it
>> >> >>>>> is actually comparable with a run without that
patch. Would you
>> >> >>>>> please try
>> >> >>>>> that out?
>> >> >>>>> It would be best to have 4.2 upstream and this[1]
run to really
>> >> >>>>> minimize the noise.
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>> michal
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>> demand-el7-x86_64/28/
>> >> >>>>>
>> >> >>>>> On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com
>> >
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>> I will update with the results of the next tier1
execution on
>> latest
>> >> >>>>> 4.2.3
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> That isn't master but old branch though. Could
you run it
>> against
>> >> >>>>> *current* VDSM master?
>> >> >>>>>
>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>> Hi, I've triggered another execution [1] due to
some issues I
>> saw
>> in
>> >> >>>>> the
>> >> >>>>> first which are not related to the patch.
>> >> >>>>>
>> >> >>>>> The success rate is 78% which is low comparing to
tier1
>> executions
>> >> >>>>> with
>> >> >>>>> code from downstream builds (95-100% success rates)
[2].
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Could you run the current master (without the
dynamic_ownership
>> >> >>>>> patch)
>> >> >>>>> so that we have viable comparision?
>> >> >>>>>
>> >> >>>>> From what I could see so far, there is an issue
with move and
>> copy
>> >> >>>>>
>> >> >>>>> operations to and from Gluster domains. For example
[3].
>> >> >>>>>
>> >> >>>>> The logs are attached.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >> >>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> >> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [2]
>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>> >> >>>>>
>> >> >>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [3]
>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7)
[vdsm.api] FINISH
>> >> >>>>> deleteImage error=Image does not exist in domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >> >>>>> from=:
>> >> >>>>> :ffff:10.35.161.182,40936,
>> >> >>>>> flow_id=disks_syncAction_ba6b2630-5976-4935,
>> >> >>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42
(api:51)
>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>> >> >>>>> [storage.TaskManager.Task]
>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>> >> >>>>> (task:875)
>> >> >>>>> Traceback (most recent call last):
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
>> line
>> >> >>>>> 882,
>> >> >>>>> in
>> >> >>>>> _run
>> >> >>>>> return fn(*args, **kargs)
>> >> >>>>> File "<string>", line 2, in
deleteImage
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>> line
>> 49,
>> >> >>>>> in
>> >> >>>>> method
>> >> >>>>> ret = func(*args, **kwargs)
>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>> line
>> >> >>>>> 1503,
>> >> >>>>> in
>> >> >>>>> deleteImage
>> >> >>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> >> >>>>> ImageDoesNotExistInSD: Image does not exist in
domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> >> >>>>>
>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>> >> >>>>> [storage.TaskManager.Task]
>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is
>> >> >>>>> aborted:
>> >> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835
>> -
>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" -
>> code
>> >> >>>>> 268
>> >> >>>>> (task:1181)
>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7)
>> [storage.Dispatcher]
>> >> >>>>> FINISH
>> >> >>>>> deleteImage error=Image does not exist in domain:
>> >> >>>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
>> >> >>>>> -ca40ff054d
>> >> >>>>> d4' (dispatcher:82)
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon
>> >> >>>>> <ebenahar(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> Triggered a sanity tier1 execution [1] using [2],
which covers
>> all
>> >> >>>>> the
>> >> >>>>>
>> >> >>>>> requested areas, on iSCSI, NFS and Gluster.
>> >> >>>>> I'll update with the results.
>> >> >>>>>
>> >> >>>>> [1]
>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> >> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>> >> >>>>>
>> >> >>>>> [2]
>> >> >>>>> https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hi Martin,
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I see [1] requires a rebase, can you please take
care?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Should be rebased.
>> >> >>>>>
>> >> >>>>> At the moment, our automation is stable only on
iSCSI, NFS,
>> Gluster
>> >> >>>>> and
>> >> >>>>>
>> >> >>>>> FC.
>> >> >>>>> Ceph is not supported and Cinder will be stabilized
soon, AFAIR,
>> >> >>>>> it's
>> >> >>>>> not
>> >> >>>>> stable enough at the moment.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> That is still pretty good.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Thanks
>> >> >>>>>
>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
>> >> >>>>> <mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hi, sorry if I misunderstood, I waited for more
input regarding
>> what
>> >> >>>>>
>> >> >>>>> areas
>> >> >>>>> have to be tested here.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I'd say that you have quite a bit of freedom in
this regard.
>> >> >>>>>
>> >> >>>>> GlusterFS
>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with some
>> >> >>>>> suite
>> >> >>>>> that covers basic operations (start & stop VM,
migrate it),
>> >> >>>>> snapshots
>> >> >>>>> and merging them, and whatever else would be
important for
>> storage
>> >> >>>>> sanity.
>> >> >>>>>
>> >> >>>>> mpolednik
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik
<
>> >> >>>>> mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> We can test this on iSCSI, NFS and GlusterFS. As
for ceph and
>> >> >>>>> cinder,
>> >> >>>>>
>> >> >>>>> will
>> >> >>>>>
>> >> >>>>> have to check, since usually, we don't execute
our automation on
>> >> >>>>> them.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Any update on this? I believe the gluster tests
were successful,
>> >> >>>>> OST
>> >> >>>>>
>> >> >>>>> passes fine and unit tests pass fine, that makes
the storage
>> >> >>>>> backends
>> >> >>>>> test the last required piece.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> +Elad
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<
>> danken(a)redhat.com
>> >> >>>>>
>> >> >>>>> >
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <
>> nsoffer(a)redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Please make sure to run as much OST suites on this
patch as
>> >> >>>>>
>> >> >>>>> possible
>> >> >>>>>
>> >> >>>>> before merging ( using 'ci please build' )
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> But note that OST is not a way to verify the
patch.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Such changes require testing with all storage types
we support.
>> >> >>>>>
>> >> >>>>> Nir
>> >> >>>>>
>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<
>> >> >>>>> mpolednik(a)redhat.com
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hey,
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I've created a patch[0] that is finally able to
activate
>> >> >>>>>
>> >> >>>>> libvirt's
>> >> >>>>> dynamic_ownership for VDSM while not negatively
affecting
>> >> >>>>> functionality of our storage code.
>> >> >>>>>
>> >> >>>>> That of course comes with quite a bit of code
removal, mostly
>> >> >>>>> in
>> >> >>>>> the
>> >> >>>>> area of host devices, hwrng and anything that
touches devices;
>> >> >>>>> bunch
>> >> >>>>> of test changes and one XML generation caveat
(storage is
>> >> >>>>> handled
>> >> >>>>> by
>> >> >>>>> VDSM, therefore disk relabelling needs to be
disabled on the
>> >> >>>>> VDSM
>> >> >>>>> level).
>> >> >>>>>
>> >> >>>>> Because of the scope of the patch, I welcome
>> >> >>>>> storage/virt/network
>> >> >>>>> people to review the code and consider the
implication this
>> >> >>>>> change
>> >> >>>>> has
>> >> >>>>> on current/future features.
>> >> >>>>>
>> >> >>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> In particular: dynamic_ownership was set to 0
prehistorically
>> >> >>>>> (as
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> part
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>> >> >>>>>
>> >> >>>>> libvirt,
>> >> >>>>> running as root, was not able to play properly with
root-squash
>> >> >>>>> nfs
>> >> >>>>> mounts.
>> >> >>>>>
>> >> >>>>> Have you attempted this use case?
>> >> >>>>>
>> >> >>>>> I join to Nir's request to run this with
storage QE.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> --
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Raz Tamir
>> >> >>>>> Manager, RHV QE
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> _______________________________________________
>> >> >>>>> Devel mailing list
>> >> >>>>> Devel(a)ovirt.org
>> >> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >
>> >> > <logs.tar.gz>
>> >> >
>> >> >
>> >
>> >
>>
>>
4:47 a.m.
New subject: dynamic ownership changes
Execution is done, 59/65 cases passed. Latest 4.2.4 execution ended with
100% so failures were caused probably due to the changes done in the patch.
Failures are mainly on preview snapshots.
Execution info provided to Martin separately.
On Wed, May 30, 2018 at 5:44 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
Triggered a sanity automation execution using [1], which covers all
the
requested areas, on iSCSI, NFS and Gluster.
I'll update with the results.
[1]
*https://gerrit.ovirt.org/#/c/90906/ <https://gerrit.ovirt.org/#/c/90906/>*
vdsm-4.20.28-6.gitc23aef6.el7.x86_64
On Tue, May 29, 2018 at 4:26 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
> On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
>
>> Hi Martin,
>>
>> Can you please create a cerry pick patch that is based on 4.2?
>>
>
> See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
> (storage needs real env).
>
> mpolednik
>
>
>
>> Thanks
>>
>> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>>
>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>>> wrote:
>>> > Hi Dan,
>>> >
>>> > In the last execution, the success rate was very low due to a large
>>> number
>>> > of failures on start VM caused, according to Michal, by the
>>> > vdsm-hook-allocate_net that was installed on the host.
>>> >
>>> > This is the latest status here, would you like me to re-execute?
>>>
>>> yes, of course. but you should rebase Polednik's code on top of
>>> *current* ovirt-4.2.3 branch.
>>>
>>> > If so, with
>>> > or W/O vdsm-hook-allocate_net installed?
>>>
>>> There was NO reason to have that installed. Please keep it (and any
>>> other needless code) out of the test environment.
>>>
>>> >
>>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>> wrote:
>>> >>
>>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>>> >> <michal.skrivanek(a)redhat.com> wrote:
>>> >> > Hi Elad,
>>> >> > why did you install vdsm-hook-allocate_net?
>>> >> >
>>> >> > adding Dan as I think the hook is not supposed to fail this
badly
>>> in
>>> any
>>> >> > case
>>> >>
>>> >> yep, this looks bad and deserves a little bug report. Installing
this
>>> >> little hook should not block vm startup.
>>> >>
>>> >> But more importantly - what is the conclusion of this thread? Do we
>>> >> have a green light from QE to take this in?
>>> >>
>>> >>
>>> >> >
>>> >> > Thanks,
>>> >> > michal
>>> >> >
>>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>> wrote:
>>> >> >
>>> >> > Start VM fails on:
>>> >> >
>>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm]
>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive
'vda' path:
>>> >> >
>>> >> > 'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>> 9a78-bdd13a843c62/images/6cdabfe5-
>>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
>>> ->
>>> >> >
>>> >> > u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>>> >> > f35a01928211' (storagexml:334)
>>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api]
START
>>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>>> >> > options=None)
>>> >> > from=::ffff:10.35.161.127,53512,
>>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>>> >> > 2758 (api:46)
>>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root]
>>> >> > /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net:
rc=2
>>> >> > err=vm
>>> >> > net allocation hook: [unexpected error]: Traceback (most
recent
>>> call
>>> >> > last):
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 105, in <module>
>>> >> > main()
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 93, in main
>>> >> > allocate_random_network(device_xml)
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 62, in allocate_random_network
>>> >> > net = _get_random_network()
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 50, in _get_random_network
>>> >> > available_nets = _parse_nets()
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 46, in _parse_nets
>>> >> > return [net for net in os.environ[AVAIL_NETS_KEY].split()]
>>> >> > File "/usr/lib64/python2.7/UserDict.py", line 23, in
__getitem__
>>> >> > raise KeyError(key)
>>> >> > KeyError: 'equivnets'
>>> >> >
>>> >> >
>>> >> > (hooks:110)
>>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm]
>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm
start process
>>> >> > failed
>>> >> > (vm:943)
>>> >> > Traceback (most recent call last):
>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> 872,
>>> in
>>> >> > _startUnderlyingVm
>>> >> > self._run()
>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> 2861,
>>> in
>>> >> > _run
>>> >> > domxml = hooks.before_vm_start(self._buildDomainXML(),
>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> 2254,
>>> in
>>> >> > _buildDomainXML
>>> >> > dom, self.id, self._custom['custom'])
>>> >> > File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>>> preprocess.py",
>>> >> > line 240, in replace_device_xml_with_hooks_xml
>>> >> > dev_custom)
>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>> line
>>> 134,
>>> >> > in
>>> >> > before_device_create
>>> >> > params=customProperties)
>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>> line
>>> 120,
>>> >> > in
>>> >> > _runHooksDir
>>> >> > raise exception.HookError(err)
>>> >> > HookError: Hook Error: ('vm net allocation hook:
[unexpected
>>> error]:
>>> >> > Traceback (most recent call last):\n File
>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>> line
>>> >> > 105, in
>>> >> > <module>\n main()\n
>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>> t",
>>> >> > line
>>> >> > 93, in main\n allocate_random_network(device_xml)\n File
>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>> line
>>> 62,
>>> >> > i
>>> >> > n allocate_random_network\n net = _get_random_network()\n
File
>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>> line
>>> 50,
>>> >> > in
>>> >> > _get_random_network\n available_nets = _parse_nets()\n
File
>>> "/us
>>> >> >
r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>>> 46,
>>> in
>>> >> > _parse_nets\n return [net for net in
>>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
>>> >> > "/usr/lib64/python2.7/UserDict.py", line 23, in
__getit
>>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>>> >> >
>>> >> >
>>> >> >
>>> >> > Hence, the success rate was 28% against 100% running with d/s
>>> (d/s).
>>> If
>>> >> > needed, I'll compare against the latest master, but I think
you get
>>> the
>>> >> > picture with d/s.
>>> >> >
>>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>>> >> > kernel 3.10.0-862.el7.x86_64
>>> >> > rhel7.5
>>> >> >
>>> >> >
>>> >> > Logs attached
>>> >> >
>>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <
>>> ebenahar(a)redhat.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> nvm, found gluster 3.12 repo, managed to install vdsm
>>> >> >>
>>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <
>>> ebenahar(a)redhat.com
>>> >
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> No, vdsm requires it:
>>> >> >>>
>>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> >> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>> >> >>> Requires: glusterfs-fuse >= 3.12
>>> >> >>> Installed:
glusterfs-fuse-3.8.4-54.8.el7.x86_64
>>> (@rhv-4.2.3)
>>> >> >>>
>>> >> >>> Therefore, vdsm package installation is skipped upon
force
>>> install.
>>> >> >>>
>>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek
>>> >> >>> <michal.skrivanek(a)redhat.com> wrote:
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>> wrote:
>>> >> >>>>
>>> >> >>>> Hi guys,
>>> >> >>>>
>>> >> >>>> The vdsm build from the patch requires
glusterfs-fuse > 3.12.
>>> This
>>> is
>>> >> >>>> while the latest 4.2.3-5 d/s build requires 3.8.4
>>> (3.4.0.59rhs-1.el7)
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> because it is still oVirt, not a downstream build.
We can’t
>>> really
>>> do
>>> >> >>>> downstream builds with unmerged changes:/
>>> >> >>>>
>>> >> >>>> Trying to get this gluster-fuse build, so far no
luck.
>>> >> >>>> Is this requirement intentional?
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> it should work regardless, I guess you can force
install it
>>> without
>>> >> >>>> the
>>> >> >>>> dependency
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek
>>> >> >>>> <michal.skrivanek(a)redhat.com> wrote:
>>> >> >>>>>
>>> >> >>>>> Hi Elad,
>>> >> >>>>> to make it easier to compare, Martin backported
the change to
>>> 4.2
>>> so
>>> >> >>>>> it
>>> >> >>>>> is actually comparable with a run without that
patch. Would you
>>> >> >>>>> please try
>>> >> >>>>> that out?
>>> >> >>>>> It would be best to have 4.2 upstream and
this[1] run to really
>>> >> >>>>> minimize the noise.
>>> >> >>>>>
>>> >> >>>>> Thanks,
>>> >> >>>>> michal
>>> >> >>>>>
>>> >> >>>>> [1]
>>> >> >>>>>
>>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>>> demand-el7-x86_64/28/
>>> >> >>>>>
>>> >> >>>>> On 27 Apr 2018, at 09:23, Martin Polednik <
>>> mpolednik(a)redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben Aharon
wrote:
>>> >> >>>>>
>>> >> >>>>> I will update with the results of the next
tier1 execution on
>>> latest
>>> >> >>>>> 4.2.3
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> That isn't master but old branch though.
Could you run it
>>> against
>>> >> >>>>> *current* VDSM master?
>>> >> >>>>>
>>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin
Polednik
>>> >> >>>>> <mpolednik(a)redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon
wrote:
>>> >> >>>>>
>>> >> >>>>> Hi, I've triggered another execution [1]
due to some issues I
>>> saw
>>> in
>>> >> >>>>> the
>>> >> >>>>> first which are not related to the patch.
>>> >> >>>>>
>>> >> >>>>> The success rate is 78% which is low comparing
to tier1
>>> executions
>>> >> >>>>> with
>>> >> >>>>> code from downstream builds (95-100% success
rates) [2].
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Could you run the current master (without the
dynamic_ownership
>>> >> >>>>> patch)
>>> >> >>>>> so that we have viable comparision?
>>> >> >>>>>
>>> >> >>>>> From what I could see so far, there is an issue
with move and
>>> copy
>>> >> >>>>>
>>> >> >>>>> operations to and from Gluster domains. For
example [3].
>>> >> >>>>>
>>> >> >>>>> The logs are attached.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> [1]
>>> >> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>> >> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> [2]
>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>> >> >>>>>
>>> >> >>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> [3]
>>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7)
[vdsm.api]
>>> FINISH
>>> >> >>>>> deleteImage error=Image does not exist in
domain:
>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> >> >>>>> from=:
>>> >> >>>>> :ffff:10.35.161.182,40936,
>>> >> >>>>> flow_id=disks_syncAction_ba6b2630-5976-4935,
>>> >> >>>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42
(api:51)
>>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7)
>>> >> >>>>> [storage.TaskManager.Task]
>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>>> >> >>>>> (task:875)
>>> >> >>>>> Traceback (most recent call last):
>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
>>> line
>>> >> >>>>> 882,
>>> >> >>>>> in
>>> >> >>>>> _run
>>> >> >>>>> return fn(*args, **kargs)
>>> >> >>>>> File "<string>", line 2, in
deleteImage
>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>>> line
>>> 49,
>>> >> >>>>> in
>>> >> >>>>> method
>>> >> >>>>> ret = func(*args, **kwargs)
>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>>> line
>>> >> >>>>> 1503,
>>> >> >>>>> in
>>> >> >>>>> deleteImage
>>> >> >>>>> raise se.ImageDoesNotExistInSD(imgUUID,
sdUUID)
>>> >> >>>>> ImageDoesNotExistInSD: Image does not exist in
domain:
>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> >> >>>>>
>>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7)
>>> >> >>>>> [storage.TaskManager.Task]
>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task
>>> is
>>> >> >>>>> aborted:
>>> >> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835
>>> -
>>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" -
>>> code
>>> >> >>>>> 268
>>> >> >>>>> (task:1181)
>>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7)
>>> [storage.Dispatcher]
>>> >> >>>>> FINISH
>>> >> >>>>> deleteImage error=Image does not exist in
domain:
>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
>>> >> >>>>> -ca40ff054d
>>> >> >>>>> d4' (dispatcher:82)
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben
Aharon
>>> >> >>>>> <ebenahar(a)redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> Triggered a sanity tier1 execution [1] using
[2], which covers
>>> all
>>> >> >>>>> the
>>> >> >>>>>
>>> >> >>>>> requested areas, on iSCSI, NFS and Gluster.
>>> >> >>>>> I'll update with the results.
>>> >> >>>>>
>>> >> >>>>> [1]
>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>> >> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>> >> >>>>>
>>> >> >>>>> [2]
>>> >> >>>>> https://gerrit.ovirt.org/#/c/89830/
>>> >> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin
Polednik
>>> >> >>>>> <mpolednik(a)redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon
wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Hi Martin,
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> I see [1] requires a rebase, can you please
take care?
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Should be rebased.
>>> >> >>>>>
>>> >> >>>>> At the moment, our automation is stable only on
iSCSI, NFS,
>>> Gluster
>>> >> >>>>> and
>>> >> >>>>>
>>> >> >>>>> FC.
>>> >> >>>>> Ceph is not supported and Cinder will be
stabilized soon,
>>> AFAIR,
>>> >> >>>>> it's
>>> >> >>>>> not
>>> >> >>>>> stable enough at the moment.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> That is still pretty good.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Thanks
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin
Polednik
>>> >> >>>>> <mpolednik(a)redhat.com
>>> >> >>>>> >
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon
wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Hi, sorry if I misunderstood, I waited for more
input regarding
>>> what
>>> >> >>>>>
>>> >> >>>>> areas
>>> >> >>>>> have to be tested here.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> I'd say that you have quite a bit of
freedom in this regard.
>>> >> >>>>>
>>> >> >>>>> GlusterFS
>>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with some
>>> >> >>>>> suite
>>> >> >>>>> that covers basic operations (start & stop
VM, migrate it),
>>> >> >>>>> snapshots
>>> >> >>>>> and merging them, and whatever else would be
important for
>>> storage
>>> >> >>>>> sanity.
>>> >> >>>>>
>>> >> >>>>> mpolednik
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin
Polednik <
>>> >> >>>>> mpolednik(a)redhat.com
>>> >> >>>>> >
>>> >> >>>>>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon
wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> We can test this on iSCSI, NFS and GlusterFS.
As for ceph and
>>> >> >>>>> cinder,
>>> >> >>>>>
>>> >> >>>>> will
>>> >> >>>>>
>>> >> >>>>> have to check, since usually, we don't
execute our automation
>>> on
>>> >> >>>>> them.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Any update on this? I believe the gluster tests
were
>>> successful,
>>> >> >>>>> OST
>>> >> >>>>>
>>> >> >>>>> passes fine and unit tests pass fine, that
makes the storage
>>> >> >>>>> backends
>>> >> >>>>> test the last required piece.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com
>>> >
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> +Elad
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<
>>> danken(a)redhat.com
>>> >> >>>>>
>>> >> >>>>> >
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<
>>> nsoffer(a)redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>> >> >>>>>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Please make sure to run as much OST suites on
this patch as
>>> >> >>>>>
>>> >> >>>>> possible
>>> >> >>>>>
>>> >> >>>>> before merging ( using 'ci please
build' )
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> But note that OST is not a way to verify the
patch.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Such changes require testing with all storage
types we support.
>>> >> >>>>>
>>> >> >>>>> Nir
>>> >> >>>>>
>>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>> >> >>>>> mpolednik(a)redhat.com
>>> >> >>>>> >
>>> >> >>>>>
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Hey,
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> I've created a patch[0] that is finally
able to activate
>>> >> >>>>>
>>> >> >>>>> libvirt's
>>> >> >>>>> dynamic_ownership for VDSM while not negatively
affecting
>>> >> >>>>> functionality of our storage code.
>>> >> >>>>>
>>> >> >>>>> That of course comes with quite a bit of code
removal, mostly
>>> >> >>>>> in
>>> >> >>>>> the
>>> >> >>>>> area of host devices, hwrng and anything that
touches devices;
>>> >> >>>>> bunch
>>> >> >>>>> of test changes and one XML generation caveat
(storage is
>>> >> >>>>> handled
>>> >> >>>>> by
>>> >> >>>>> VDSM, therefore disk relabelling needs to be
disabled on the
>>> >> >>>>> VDSM
>>> >> >>>>> level).
>>> >> >>>>>
>>> >> >>>>> Because of the scope of the patch, I welcome
>>> >> >>>>> storage/virt/network
>>> >> >>>>> people to review the code and consider the
implication this
>>> >> >>>>> change
>>> >> >>>>> has
>>> >> >>>>> on current/future features.
>>> >> >>>>>
>>> >> >>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> In particular: dynamic_ownership was set to 0
prehistorically
>>> >> >>>>> (as
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> part
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>> because
>>> >> >>>>>
>>> >> >>>>> libvirt,
>>> >> >>>>> running as root, was not able to play properly
with root-squash
>>> >> >>>>> nfs
>>> >> >>>>> mounts.
>>> >> >>>>>
>>> >> >>>>> Have you attempted this use case?
>>> >> >>>>>
>>> >> >>>>> I join to Nir's request to run this with
storage QE.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> --
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Raz Tamir
>>> >> >>>>> Manager, RHV QE
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
_______________________________________________
>>> >> >>>>> Devel mailing list
>>> >> >>>>> Devel(a)ovirt.org
>>> >> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>
>>> >> >>
>>> >> >
>>> >> > <logs.tar.gz>
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>>
5:03 a.m.
New subject: dynamic ownership changes
On 31/05/18 12:47 +0300, Elad Ben Aharon wrote:
Execution is done, 59/65 cases passed. Latest 4.2.4 execution ended
with
100% so failures were caused probably due to the changes done in the patch.
Failures are mainly on preview snapshots.
Execution info provided to Martin separately.
I'm currently investigating the snapshot breakage, thanks Elad!
>On Wed, May 30, 2018 at 5:44 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>wrote:
>
>> Triggered a sanity automation execution using [1], which covers all the
>> requested areas, on iSCSI, NFS and Gluster.
>> I'll update with the results.
>>
>> [1]
>> *https://gerrit.ovirt.org/#/c/90906/
<https://gerrit.ovirt.org/#/c/90906/>*
>> vdsm-4.20.28-6.gitc23aef6.el7.x86_64
>>
>>
>> On Tue, May 29, 2018 at 4:26 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> Can you please create a cerry pick patch that is based on 4.2?
>>>>
>>>
>>> See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
>>> (storage needs real env).
>>>
>>> mpolednik
>>>
>>>
>>>
>>>> Thanks
>>>>
>>>> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>> wrote:
>>>>
>>>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>>>> wrote:
>>>>> > Hi Dan,
>>>>> >
>>>>> > In the last execution, the success rate was very low due to a
large
>>>>> number
>>>>> > of failures on start VM caused, according to Michal, by the
>>>>> > vdsm-hook-allocate_net that was installed on the host.
>>>>> >
>>>>> > This is the latest status here, would you like me to
re-execute?
>>>>>
>>>>> yes, of course. but you should rebase Polednik's code on top of
>>>>> *current* ovirt-4.2.3 branch.
>>>>>
>>>>> > If so, with
>>>>> > or W/O vdsm-hook-allocate_net installed?
>>>>>
>>>>> There was NO reason to have that installed. Please keep it (and any
>>>>> other needless code) out of the test environment.
>>>>>
>>>>> >
>>>>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>>> wrote:
>>>>> >>
>>>>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>>>>> >> <michal.skrivanek(a)redhat.com> wrote:
>>>>> >> > Hi Elad,
>>>>> >> > why did you install vdsm-hook-allocate_net?
>>>>> >> >
>>>>> >> > adding Dan as I think the hook is not supposed to fail
this badly
>>>>> in
>>>>> any
>>>>> >> > case
>>>>> >>
>>>>> >> yep, this looks bad and deserves a little bug report.
Installing this
>>>>> >> little hook should not block vm startup.
>>>>> >>
>>>>> >> But more importantly - what is the conclusion of this
thread? Do we
>>>>> >> have a green light from QE to take this in?
>>>>> >>
>>>>> >>
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> > michal
>>>>> >> >
>>>>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>>>> wrote:
>>>>> >> >
>>>>> >> > Start VM fails on:
>>>>> >> >
>>>>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce)
[virt.vm]
>>>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27')
drive 'vda' path:
>>>>> >> >
>>>>> >> >
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>>>> 9a78-bdd13a843c62/images/6cdabfe5-
>>>>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
>>>>> ->
>>>>> >> >
>>>>> >> >
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>>>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>>>>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>>>>> >> > f35a01928211' (storagexml:334)
>>>>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1)
[vdsm.api] START
>>>>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>>>>> >> > options=None)
>>>>> >> > from=::ffff:10.35.161.127,53512,
>>>>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>>>>> >> > 2758 (api:46)
>>>>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce)
[root]
>>>>> >> >
/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=2
>>>>> >> > err=vm
>>>>> >> > net allocation hook: [unexpected error]: Traceback
(most recent
>>>>> call
>>>>> >> > last):
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 105, in <module>
>>>>> >> > main()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 93, in main
>>>>> >> > allocate_random_network(device_xml)
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 62, in allocate_random_network
>>>>> >> > net = _get_random_network()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 50, in _get_random_network
>>>>> >> > available_nets = _parse_nets()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 46, in _parse_nets
>>>>> >> > return [net for net in
os.environ[AVAIL_NETS_KEY].split()]
>>>>> >> > File "/usr/lib64/python2.7/UserDict.py",
line 23, in __getitem__
>>>>> >> > raise KeyError(key)
>>>>> >> > KeyError: 'equivnets'
>>>>> >> >
>>>>> >> >
>>>>> >> > (hooks:110)
>>>>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce)
[virt.vm]
>>>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27')
The vm start process
>>>>> >> > failed
>>>>> >> > (vm:943)
>>>>> >> > Traceback (most recent call last):
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 872,
>>>>> in
>>>>> >> > _startUnderlyingVm
>>>>> >> > self._run()
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 2861,
>>>>> in
>>>>> >> > _run
>>>>> >> > domxml =
hooks.before_vm_start(self._buildDomainXML(),
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 2254,
>>>>> in
>>>>> >> > _buildDomainXML
>>>>> >> > dom, self.id, self._custom['custom'])
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>>>>> preprocess.py",
>>>>> >> > line 240, in replace_device_xml_with_hooks_xml
>>>>> >> > dev_custom)
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>>>> line
>>>>> 134,
>>>>> >> > in
>>>>> >> > before_device_create
>>>>> >> > params=customProperties)
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>>>> line
>>>>> 120,
>>>>> >> > in
>>>>> >> > _runHooksDir
>>>>> >> > raise exception.HookError(err)
>>>>> >> > HookError: Hook Error: ('vm net allocation hook:
[unexpected
>>>>> error]:
>>>>> >> > Traceback (most recent call last):\n File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> >> > 105, in
>>>>> >> > <module>\n main()\n
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 93, in main\n allocate_random_network(device_xml)\n
File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> 62,
>>>>> >> > i
>>>>> >> > n allocate_random_network\n net =
_get_random_network()\n File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> 50,
>>>>> >> > in
>>>>> >> > _get_random_network\n available_nets =
_parse_nets()\n File
>>>>> "/us
>>>>> >> >
r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>>>>> 46,
>>>>> in
>>>>> >> > _parse_nets\n return [net for net in
>>>>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
>>>>> >> > "/usr/lib64/python2.7/UserDict.py", line 23,
in __getit
>>>>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Hence, the success rate was 28% against 100% running
with d/s
>>>>> (d/s).
>>>>> If
>>>>> >> > needed, I'll compare against the latest master, but
I think you get
>>>>> the
>>>>> >> > picture with d/s.
>>>>> >> >
>>>>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>>>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>>>>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>>>>> >> > kernel 3.10.0-862.el7.x86_64
>>>>> >> > rhel7.5
>>>>> >> >
>>>>> >> >
>>>>> >> > Logs attached
>>>>> >> >
>>>>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <
>>>>> ebenahar(a)redhat.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> nvm, found gluster 3.12 repo, managed to install
vdsm
>>>>> >> >>
>>>>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon
<
>>>>> ebenahar(a)redhat.com
>>>>> >
>>>>> >> >> wrote:
>>>>> >> >>>
>>>>> >> >>> No, vdsm requires it:
>>>>> >> >>>
>>>>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>>>> >> >>> (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>>>> >> >>> Requires: glusterfs-fuse >= 3.12
>>>>> >> >>> Installed:
glusterfs-fuse-3.8.4-54.8.el7.x86_64
>>>>> (@rhv-4.2.3)
>>>>> >> >>>
>>>>> >> >>> Therefore, vdsm package installation is skipped
upon force
>>>>> install.
>>>>> >> >>>
>>>>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal
Skrivanek
>>>>> >> >>> <michal.skrivanek(a)redhat.com> wrote:
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>>>> wrote:
>>>>> >> >>>>
>>>>> >> >>>> Hi guys,
>>>>> >> >>>>
>>>>> >> >>>> The vdsm build from the patch requires
glusterfs-fuse > 3.12.
>>>>> This
>>>>> is
>>>>> >> >>>> while the latest 4.2.3-5 d/s build requires
3.8.4
>>>>> (3.4.0.59rhs-1.el7)
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> because it is still oVirt, not a downstream
build. We can’t
>>>>> really
>>>>> do
>>>>> >> >>>> downstream builds with unmerged changes:/
>>>>> >> >>>>
>>>>> >> >>>> Trying to get this gluster-fuse build, so
far no luck.
>>>>> >> >>>> Is this requirement intentional?
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> it should work regardless, I guess you can
force install it
>>>>> without
>>>>> >> >>>> the
>>>>> >> >>>> dependency
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal
Skrivanek
>>>>> >> >>>> <michal.skrivanek(a)redhat.com> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Hi Elad,
>>>>> >> >>>>> to make it easier to compare, Martin
backported the change to
>>>>> 4.2
>>>>> so
>>>>> >> >>>>> it
>>>>> >> >>>>> is actually comparable with a run
without that patch. Would you
>>>>> >> >>>>> please try
>>>>> >> >>>>> that out?
>>>>> >> >>>>> It would be best to have 4.2 upstream
and this[1] run to really
>>>>> >> >>>>> minimize the noise.
>>>>> >> >>>>>
>>>>> >> >>>>> Thanks,
>>>>> >> >>>>> michal
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>>
>>>>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>>>>> demand-el7-x86_64/28/
>>>>> >> >>>>>
>>>>> >> >>>>> On 27 Apr 2018, at 09:23, Martin
Polednik <
>>>>> mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> I will update with the results of the
next tier1 execution on
>>>>> latest
>>>>> >> >>>>> 4.2.3
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> That isn't master but old branch
though. Could you run it
>>>>> against
>>>>> >> >>>>> *current* VDSM master?
>>>>> >> >>>>>
>>>>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin
Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Hi, I've triggered another
execution [1] due to some issues I
>>>>> saw
>>>>> in
>>>>> >> >>>>> the
>>>>> >> >>>>> first which are not related to the
patch.
>>>>> >> >>>>>
>>>>> >> >>>>> The success rate is 78% which is low
comparing to tier1
>>>>> executions
>>>>> >> >>>>> with
>>>>> >> >>>>> code from downstream builds (95-100%
success rates) [2].
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Could you run the current master
(without the dynamic_ownership
>>>>> >> >>>>> patch)
>>>>> >> >>>>> so that we have viable comparision?
>>>>> >> >>>>>
>>>>> >> >>>>> From what I could see so far, there is
an issue with move and
>>>>> copy
>>>>> >> >>>>>
>>>>> >> >>>>> operations to and from Gluster domains.
For example [3].
>>>>> >> >>>>>
>>>>> >> >>>>> The logs are attached.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>>>> >> >>>>>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [2]
>>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>>> >> >>>>>
>>>>> >> >>>>>
rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [3]
>>>>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO
(jsonrpc/7) [vdsm.api]
>>>>> FINISH
>>>>> >> >>>>> deleteImage error=Image does not exist
in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> >> >>>>> from=:
>>>>> >> >>>>> :ffff:10.35.161.182,40936,
>>>>> >> >>>>>
flow_id=disks_syncAction_ba6b2630-5976-4935,
>>>>> >> >>>>>
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR
(jsonrpc/7)
>>>>> >> >>>>> [storage.TaskManager.Task]
>>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>>>>> >> >>>>> (task:875)
>>>>> >> >>>>> Traceback (most recent call last):
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
>>>>> line
>>>>> >> >>>>> 882,
>>>>> >> >>>>> in
>>>>> >> >>>>> _run
>>>>> >> >>>>> return fn(*args, **kargs)
>>>>> >> >>>>> File "<string>", line
2, in deleteImage
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>>>>> line
>>>>> 49,
>>>>> >> >>>>> in
>>>>> >> >>>>> method
>>>>> >> >>>>> ret = func(*args, **kwargs)
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>>>>> line
>>>>> >> >>>>> 1503,
>>>>> >> >>>>> in
>>>>> >> >>>>> deleteImage
>>>>> >> >>>>> raise
se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>>>> >> >>>>> ImageDoesNotExistInSD: Image does not
exist in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> >> >>>>>
>>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO
(jsonrpc/7)
>>>>> >> >>>>> [storage.TaskManager.Task]
>>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task
>>>>> is
>>>>> >> >>>>> aborted:
>>>>> >> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835
>>>>> -
>>>>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" -
>>>>> code
>>>>> >> >>>>> 268
>>>>> >> >>>>> (task:1181)
>>>>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR
(jsonrpc/7)
>>>>> [storage.Dispatcher]
>>>>> >> >>>>> FINISH
>>>>> >> >>>>> deleteImage error=Image does not exist
in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
>>>>> >> >>>>> -ca40ff054d
>>>>> >> >>>>> d4' (dispatcher:82)
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad
Ben Aharon
>>>>> >> >>>>> <ebenahar(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Triggered a sanity tier1 execution [1]
using [2], which covers
>>>>> all
>>>>> >> >>>>> the
>>>>> >> >>>>>
>>>>> >> >>>>> requested areas, on iSCSI, NFS and
Gluster.
>>>>> >> >>>>> I'll update with the results.
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>>> >> >>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>> >> >>>>>
>>>>> >> >>>>> [2]
>>>>> >> >>>>> https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin
Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hi Martin,
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I see [1] requires a rebase, can you
please take care?
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Should be rebased.
>>>>> >> >>>>>
>>>>> >> >>>>> At the moment, our automation is stable
only on iSCSI, NFS,
>>>>> Gluster
>>>>> >> >>>>> and
>>>>> >> >>>>>
>>>>> >> >>>>> FC.
>>>>> >> >>>>> Ceph is not supported and Cinder will
be stabilized soon,
>>>>> AFAIR,
>>>>> >> >>>>> it's
>>>>> >> >>>>> not
>>>>> >> >>>>> stable enough at the moment.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> That is still pretty good.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Thanks
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin
Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hi, sorry if I misunderstood, I waited
for more input regarding
>>>>> what
>>>>> >> >>>>>
>>>>> >> >>>>> areas
>>>>> >> >>>>> have to be tested here.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I'd say that you have quite a bit
of freedom in this regard.
>>>>> >> >>>>>
>>>>> >> >>>>> GlusterFS
>>>>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with some
>>>>> >> >>>>> suite
>>>>> >> >>>>> that covers basic operations (start
& stop VM, migrate it),
>>>>> >> >>>>> snapshots
>>>>> >> >>>>> and merging them, and whatever else
would be important for
>>>>> storage
>>>>> >> >>>>> sanity.
>>>>> >> >>>>>
>>>>> >> >>>>> mpolednik
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM,
Martin Polednik <
>>>>> >> >>>>> mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> We can test this on iSCSI, NFS and
GlusterFS. As for ceph and
>>>>> >> >>>>> cinder,
>>>>> >> >>>>>
>>>>> >> >>>>> will
>>>>> >> >>>>>
>>>>> >> >>>>> have to check, since usually, we
don't execute our automation
>>>>> on
>>>>> >> >>>>> them.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Any update on this? I believe the
gluster tests were
>>>>> successful,
>>>>> >> >>>>> OST
>>>>> >> >>>>>
>>>>> >> >>>>> passes fine and unit tests pass fine,
that makes the storage
>>>>> >> >>>>> backends
>>>>> >> >>>>> test the last required piece.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz
Tamir <ratamir(a)redhat.com
>>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> +Elad
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan
Kenigsberg <
>>>>> danken(a)redhat.com
>>>>> >> >>>>>
>>>>> >> >>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir
Soffer <
>>>>> nsoffer(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal
Edri <eedri(a)redhat.com>
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Please make sure to run as much OST
suites on this patch as
>>>>> >> >>>>>
>>>>> >> >>>>> possible
>>>>> >> >>>>>
>>>>> >> >>>>> before merging ( using 'ci please
build' )
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> But note that OST is not a way to
verify the patch.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Such changes require testing with all
storage types we support.
>>>>> >> >>>>>
>>>>> >> >>>>> Nir
>>>>> >> >>>>>
>>>>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>>>> >> >>>>> mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hey,
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I've created a patch[0] that is
finally able to activate
>>>>> >> >>>>>
>>>>> >> >>>>> libvirt's
>>>>> >> >>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>> >> >>>>> functionality of our storage code.
>>>>> >> >>>>>
>>>>> >> >>>>> That of course comes with quite a bit
of code removal, mostly
>>>>> >> >>>>> in
>>>>> >> >>>>> the
>>>>> >> >>>>> area of host devices, hwrng and
anything that touches devices;
>>>>> >> >>>>> bunch
>>>>> >> >>>>> of test changes and one XML generation
caveat (storage is
>>>>> >> >>>>> handled
>>>>> >> >>>>> by
>>>>> >> >>>>> VDSM, therefore disk relabelling needs
to be disabled on the
>>>>> >> >>>>> VDSM
>>>>> >> >>>>> level).
>>>>> >> >>>>>
>>>>> >> >>>>> Because of the scope of the patch, I
welcome
>>>>> >> >>>>> storage/virt/network
>>>>> >> >>>>> people to review the code and consider
the implication this
>>>>> >> >>>>> change
>>>>> >> >>>>> has
>>>>> >> >>>>> on current/future features.
>>>>> >> >>>>>
>>>>> >> >>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> In particular: dynamic_ownership was
set to 0 prehistorically
>>>>> >> >>>>> (as
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> part
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>>>> because
>>>>> >> >>>>>
>>>>> >> >>>>> libvirt,
>>>>> >> >>>>> running as root, was not able to play
properly with root-squash
>>>>> >> >>>>> nfs
>>>>> >> >>>>> mounts.
>>>>> >> >>>>>
>>>>> >> >>>>> Have you attempted this use case?
>>>>> >> >>>>>
>>>>> >> >>>>> I join to Nir's request to run this
with storage QE.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> --
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Raz Tamir
>>>>> >> >>>>> Manager, RHV QE
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
_______________________________________________
>>>>> >> >>>>> Devel mailing list
>>>>> >> >>>>> Devel(a)ovirt.org
>>>>> >> >>>>>
http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>
>>>>> >> >>
>>>>> >> >
>>>>> >> > <logs.tar.gz>
>>>>> >> >
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>
5:19 a.m.
New subject: dynamic ownership changes
On Thu, May 31, 2018 at 1:05 PM Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 31/05/18 12:47 +0300, Elad Ben Aharon wrote:
>Execution is done, 59/65 cases passed. Latest 4.2.4 execution ended with
>100% so failures were caused probably due to the changes done in the
patch.
>Failures are mainly on preview snapshots.
Can we run the same job on the patch before Martin patch? maybe
the issue are already in master, caused by other patches?
>
>Execution info provided to Martin separately.
I'm currently investigating the snapshot breakage, thanks Elad!
>On Wed, May 30, 2018 at 5:44 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>wrote:
>
>> Triggered a sanity automation execution using [1], which covers all the
>> requested areas, on iSCSI, NFS and Gluster.
>> I'll update with the results.
>>
>> [1]
>> *https://gerrit.ovirt.org/#/c/90906/ <
https://gerrit.ovirt.org/#/c/90906/>*
>> vdsm-4.20.28-6.gitc23aef6.el7.x86_64
>>
>>
>> On Tue, May 29, 2018 at 4:26 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> Can you please create a cerry pick patch that is based on 4.2?
>>>>
>>>
>>> See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
>>> (storage needs real env).
>>>
>>> mpolednik
>>>
>>>
>>>
>>>> Thanks
>>>>
>>>> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>> wrote:
>>>>
>>>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon
<ebenahar(a)redhat.com
>
>>>>> wrote:
>>>>> > Hi Dan,
>>>>> >
>>>>> > In the last execution, the success rate was very low due to a
large
>>>>> number
>>>>> > of failures on start VM caused, according to Michal, by the
>>>>> > vdsm-hook-allocate_net that was installed on the host.
>>>>> >
>>>>> > This is the latest status here, would you like me to
re-execute?
>>>>>
>>>>> yes, of course. but you should rebase Polednik's code on top of
>>>>> *current* ovirt-4.2.3 branch.
>>>>>
>>>>> > If so, with
>>>>> > or W/O vdsm-hook-allocate_net installed?
>>>>>
>>>>> There was NO reason to have that installed. Please keep it (and any
>>>>> other needless code) out of the test environment.
>>>>>
>>>>> >
>>>>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg
<danken(a)redhat.com
>
>>>>> wrote:
>>>>> >>
>>>>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
>>>>> >> <michal.skrivanek(a)redhat.com> wrote:
>>>>> >> > Hi Elad,
>>>>> >> > why did you install vdsm-hook-allocate_net?
>>>>> >> >
>>>>> >> > adding Dan as I think the hook is not supposed to fail
this
badly
>>>>> in
>>>>> any
>>>>> >> > case
>>>>> >>
>>>>> >> yep, this looks bad and deserves a little bug report.
Installing
this
>>>>> >> little hook should not block vm startup.
>>>>> >>
>>>>> >> But more importantly - what is the conclusion of this
thread? Do
we
>>>>> >> have a green light from QE to take this in?
>>>>> >>
>>>>> >>
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> > michal
>>>>> >> >
>>>>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>>>> wrote:
>>>>> >> >
>>>>> >> > Start VM fails on:
>>>>> >> >
>>>>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce)
[virt.vm]
>>>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27')
drive 'vda' path:
>>>>> >> >
>>>>> >> >
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>>>> 9a78-bdd13a843c62/images/6cdabfe5-
>>>>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211'
>>>>> ->
>>>>> >> >
>>>>> >> >
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>>>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>>>>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>>>>> >> > f35a01928211' (storagexml:334)
>>>>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1)
[vdsm.api] START
>>>>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>>>>> >> > options=None)
>>>>> >> > from=::ffff:10.35.161.127,53512,
>>>>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>>>>> >> > 2758 (api:46)
>>>>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce)
[root]
>>>>> >> >
/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net:
rc=2
>>>>> >> > err=vm
>>>>> >> > net allocation hook: [unexpected error]: Traceback
(most recent
>>>>> call
>>>>> >> > last):
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 105, in <module>
>>>>> >> > main()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 93, in main
>>>>> >> > allocate_random_network(device_xml)
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 62, in allocate_random_network
>>>>> >> > net = _get_random_network()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 50, in _get_random_network
>>>>> >> > available_nets = _parse_nets()
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 46, in _parse_nets
>>>>> >> > return [net for net in
os.environ[AVAIL_NETS_KEY].split()]
>>>>> >> > File "/usr/lib64/python2.7/UserDict.py",
line 23, in
__getitem__
>>>>> >> > raise KeyError(key)
>>>>> >> > KeyError: 'equivnets'
>>>>> >> >
>>>>> >> >
>>>>> >> > (hooks:110)
>>>>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce)
[virt.vm]
>>>>> >> > (vmId='e6ce66ce-852f-48c5-9997-5d2959432a27')
The vm start
process
>>>>> >> > failed
>>>>> >> > (vm:943)
>>>>> >> > Traceback (most recent call last):
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 872,
>>>>> in
>>>>> >> > _startUnderlyingVm
>>>>> >> > self._run()
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 2861,
>>>>> in
>>>>> >> > _run
>>>>> >> > domxml =
hooks.before_vm_start(self._buildDomainXML(),
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>> 2254,
>>>>> in
>>>>> >> > _buildDomainXML
>>>>> >> > dom, self.id, self._custom['custom'])
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>>>>> preprocess.py",
>>>>> >> > line 240, in replace_device_xml_with_hooks_xml
>>>>> >> > dev_custom)
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>>>> line
>>>>> 134,
>>>>> >> > in
>>>>> >> > before_device_create
>>>>> >> > params=customProperties)
>>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>>>> line
>>>>> 120,
>>>>> >> > in
>>>>> >> > _runHooksDir
>>>>> >> > raise exception.HookError(err)
>>>>> >> > HookError: Hook Error: ('vm net allocation hook:
[unexpected
>>>>> error]:
>>>>> >> > Traceback (most recent call last):\n File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> >> > 105, in
>>>>> >> > <module>\n main()\n
>>>>> >> > File
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_ne
>>>>> t",
>>>>> >> > line
>>>>> >> > 93, in main\n allocate_random_network(device_xml)\n
File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> 62,
>>>>> >> > i
>>>>> >> > n allocate_random_network\n net =
_get_random_network()\n
File
>>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>>>> line
>>>>> 50,
>>>>> >> > in
>>>>> >> > _get_random_network\n available_nets =
_parse_nets()\n File
>>>>> "/us
>>>>> >> >
r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line
>>>>> 46,
>>>>> in
>>>>> >> > _parse_nets\n return [net for net in
>>>>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
>>>>> >> > "/usr/lib64/python2.7/UserDict.py", line 23,
in __getit
>>>>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Hence, the success rate was 28% against 100% running
with d/s
>>>>> (d/s).
>>>>> If
>>>>> >> > needed, I'll compare against the latest master,
but I think you
get
>>>>> the
>>>>> >> > picture with d/s.
>>>>> >> >
>>>>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>>>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>>>>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>>>>> >> > kernel 3.10.0-862.el7.x86_64
>>>>> >> > rhel7.5
>>>>> >> >
>>>>> >> >
>>>>> >> > Logs attached
>>>>> >> >
>>>>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <
>>>>> ebenahar(a)redhat.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> nvm, found gluster 3.12 repo, managed to install
vdsm
>>>>> >> >>
>>>>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon
<
>>>>> ebenahar(a)redhat.com
>>>>> >
>>>>> >> >> wrote:
>>>>> >> >>>
>>>>> >> >>> No, vdsm requires it:
>>>>> >> >>>
>>>>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>>>> >> >>>
(/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>>>> >> >>> Requires: glusterfs-fuse >= 3.12
>>>>> >> >>> Installed:
glusterfs-fuse-3.8.4-54.8.el7.x86_64
>>>>> (@rhv-4.2.3)
>>>>> >> >>>
>>>>> >> >>> Therefore, vdsm package installation is
skipped upon force
>>>>> install.
>>>>> >> >>>
>>>>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal
Skrivanek
>>>>> >> >>> <michal.skrivanek(a)redhat.com> wrote:
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On 5 May 2018, at 00:38, Elad Ben Aharon
<
ebenahar(a)redhat.com>
>>>>> wrote:
>>>>> >> >>>>
>>>>> >> >>>> Hi guys,
>>>>> >> >>>>
>>>>> >> >>>> The vdsm build from the patch requires
glusterfs-fuse > 3.12.
>>>>> This
>>>>> is
>>>>> >> >>>> while the latest 4.2.3-5 d/s build
requires 3.8.4
>>>>> (3.4.0.59rhs-1.el7)
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> because it is still oVirt, not a
downstream build. We can’t
>>>>> really
>>>>> do
>>>>> >> >>>> downstream builds with unmerged changes:/
>>>>> >> >>>>
>>>>> >> >>>> Trying to get this gluster-fuse build, so
far no luck.
>>>>> >> >>>> Is this requirement intentional?
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> it should work regardless, I guess you can
force install it
>>>>> without
>>>>> >> >>>> the
>>>>> >> >>>> dependency
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal
Skrivanek
>>>>> >> >>>> <michal.skrivanek(a)redhat.com>
wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Hi Elad,
>>>>> >> >>>>> to make it easier to compare, Martin
backported the change
to
>>>>> 4.2
>>>>> so
>>>>> >> >>>>> it
>>>>> >> >>>>> is actually comparable with a run
without that patch. Would
you
>>>>> >> >>>>> please try
>>>>> >> >>>>> that out?
>>>>> >> >>>>> It would be best to have 4.2 upstream
and this[1] run to
really
>>>>> >> >>>>> minimize the noise.
>>>>> >> >>>>>
>>>>> >> >>>>> Thanks,
>>>>> >> >>>>> michal
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>>
>>>>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>>>>> demand-el7-x86_64/28/
>>>>> >> >>>>>
>>>>> >> >>>>> On 27 Apr 2018, at 09:23, Martin
Polednik <
>>>>> mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> I will update with the results of the
next tier1 execution
on
>>>>> latest
>>>>> >> >>>>> 4.2.3
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> That isn't master but old branch
though. Could you run it
>>>>> against
>>>>> >> >>>>> *current* VDSM master?
>>>>> >> >>>>>
>>>>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM,
Martin Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Hi, I've triggered another
execution [1] due to some issues
I
>>>>> saw
>>>>> in
>>>>> >> >>>>> the
>>>>> >> >>>>> first which are not related to the
patch.
>>>>> >> >>>>>
>>>>> >> >>>>> The success rate is 78% which is low
comparing to tier1
>>>>> executions
>>>>> >> >>>>> with
>>>>> >> >>>>> code from downstream builds (95-100%
success rates) [2].
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Could you run the current master
(without the
dynamic_ownership
>>>>> >> >>>>> patch)
>>>>> >> >>>>> so that we have viable comparision?
>>>>> >> >>>>>
>>>>> >> >>>>> From what I could see so far, there is
an issue with move
and
>>>>> copy
>>>>> >> >>>>>
>>>>> >> >>>>> operations to and from Gluster
domains. For example [3].
>>>>> >> >>>>>
>>>>> >> >>>>> The logs are attached.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>> *
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>>>> >> >>>>> <
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [2]
>>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>>> >> >>>>>
>>>>> >> >>>>>
rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [3]
>>>>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO
(jsonrpc/7) [vdsm.api]
>>>>> FINISH
>>>>> >> >>>>> deleteImage error=Image does not exist
in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> >> >>>>> from=:
>>>>> >> >>>>> :ffff:10.35.161.182,40936,
>>>>> >> >>>>>
flow_id=disks_syncAction_ba6b2630-5976-4935,
>>>>> >> >>>>>
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR
(jsonrpc/7)
>>>>> >> >>>>> [storage.TaskManager.Task]
>>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
error
>>>>> >> >>>>> (task:875)
>>>>> >> >>>>> Traceback (most recent call last):
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py",
>>>>> line
>>>>> >> >>>>> 882,
>>>>> >> >>>>> in
>>>>> >> >>>>> _run
>>>>> >> >>>>> return fn(*args, **kargs)
>>>>> >> >>>>> File "<string>", line
2, in deleteImage
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>>>>> line
>>>>> 49,
>>>>> >> >>>>> in
>>>>> >> >>>>> method
>>>>> >> >>>>> ret = func(*args, **kwargs)
>>>>> >> >>>>> File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>>>>> line
>>>>> >> >>>>> 1503,
>>>>> >> >>>>> in
>>>>> >> >>>>> deleteImage
>>>>> >> >>>>> raise
se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>>>> >> >>>>> ImageDoesNotExistInSD: Image does not
exist in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>>> >> >>>>>
>>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO
(jsonrpc/7)
>>>>> >> >>>>> [storage.TaskManager.Task]
>>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task
>>>>> is
>>>>> >> >>>>> aborted:
>>>>> >> >>>>> "Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835
>>>>> -
>>>>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'"
-
>>>>> code
>>>>> >> >>>>> 268
>>>>> >> >>>>> (task:1181)
>>>>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR
(jsonrpc/7)
>>>>> [storage.Dispatcher]
>>>>> >> >>>>> FINISH
>>>>> >> >>>>> deleteImage error=Image does not exist
in domain:
>>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>>>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
>>>>> >> >>>>> -ca40ff054d
>>>>> >> >>>>> d4' (dispatcher:82)
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad
Ben Aharon
>>>>> >> >>>>> <ebenahar(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> Triggered a sanity tier1 execution [1]
using [2], which
covers
>>>>> all
>>>>> >> >>>>> the
>>>>> >> >>>>>
>>>>> >> >>>>> requested areas, on iSCSI, NFS and
Gluster.
>>>>> >> >>>>> I'll update with the results.
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
>>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>>> >> >>>>>
_dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>> >> >>>>>
>>>>> >> >>>>> [2]
>>>>> >> >>>>> https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM,
Martin Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hi Martin,
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I see [1] requires a rebase, can you
please take care?
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Should be rebased.
>>>>> >> >>>>>
>>>>> >> >>>>> At the moment, our automation is
stable only on iSCSI, NFS,
>>>>> Gluster
>>>>> >> >>>>> and
>>>>> >> >>>>>
>>>>> >> >>>>> FC.
>>>>> >> >>>>> Ceph is not supported and Cinder will
be stabilized soon,
>>>>> AFAIR,
>>>>> >> >>>>> it's
>>>>> >> >>>>> not
>>>>> >> >>>>> stable enough at the moment.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> That is still pretty good.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> [1]
https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Thanks
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM,
Martin Polednik
>>>>> >> >>>>> <mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hi, sorry if I misunderstood, I waited
for more input
regarding
>>>>> what
>>>>> >> >>>>>
>>>>> >> >>>>> areas
>>>>> >> >>>>> have to be tested here.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I'd say that you have quite a bit
of freedom in this regard.
>>>>> >> >>>>>
>>>>> >> >>>>> GlusterFS
>>>>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with
some
>>>>> >> >>>>> suite
>>>>> >> >>>>> that covers basic operations (start
& stop VM, migrate it),
>>>>> >> >>>>> snapshots
>>>>> >> >>>>> and merging them, and whatever else
would be important for
>>>>> storage
>>>>> >> >>>>> sanity.
>>>>> >> >>>>>
>>>>> >> >>>>> mpolednik
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM,
Martin Polednik <
>>>>> >> >>>>> mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben
Aharon wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> We can test this on iSCSI, NFS and
GlusterFS. As for ceph
and
>>>>> >> >>>>> cinder,
>>>>> >> >>>>>
>>>>> >> >>>>> will
>>>>> >> >>>>>
>>>>> >> >>>>> have to check, since usually, we
don't execute our
automation
>>>>> on
>>>>> >> >>>>> them.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Any update on this? I believe the
gluster tests were
>>>>> successful,
>>>>> >> >>>>> OST
>>>>> >> >>>>>
>>>>> >> >>>>> passes fine and unit tests pass fine,
that makes the storage
>>>>> >> >>>>> backends
>>>>> >> >>>>> test the last required piece.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz
Tamir <
ratamir(a)redhat.com
>>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> +Elad
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan
Kenigsberg <
>>>>> danken(a)redhat.com
>>>>> >> >>>>>
>>>>> >> >>>>> >
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir
Soffer <
>>>>> nsoffer(a)redhat.com>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal
Edri <
eedri(a)redhat.com>
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Please make sure to run as much OST
suites on this patch as
>>>>> >> >>>>>
>>>>> >> >>>>> possible
>>>>> >> >>>>>
>>>>> >> >>>>> before merging ( using 'ci please
build' )
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> But note that OST is not a way to
verify the patch.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Such changes require testing with all
storage types we
support.
>>>>> >> >>>>>
>>>>> >> >>>>> Nir
>>>>> >> >>>>>
>>>>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM,
Martin Polednik <
>>>>> >> >>>>> mpolednik(a)redhat.com
>>>>> >> >>>>> >
>>>>> >> >>>>>
>>>>> >> >>>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Hey,
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> I've created a patch[0] that is
finally able to activate
>>>>> >> >>>>>
>>>>> >> >>>>> libvirt's
>>>>> >> >>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>> >> >>>>> functionality of our storage code.
>>>>> >> >>>>>
>>>>> >> >>>>> That of course comes with quite a bit
of code removal,
mostly
>>>>> >> >>>>> in
>>>>> >> >>>>> the
>>>>> >> >>>>> area of host devices, hwrng and
anything that touches
devices;
>>>>> >> >>>>> bunch
>>>>> >> >>>>> of test changes and one XML generation
caveat (storage is
>>>>> >> >>>>> handled
>>>>> >> >>>>> by
>>>>> >> >>>>> VDSM, therefore disk relabelling needs
to be disabled on the
>>>>> >> >>>>> VDSM
>>>>> >> >>>>> level).
>>>>> >> >>>>>
>>>>> >> >>>>> Because of the scope of the patch, I
welcome
>>>>> >> >>>>> storage/virt/network
>>>>> >> >>>>> people to review the code and consider
the implication this
>>>>> >> >>>>> change
>>>>> >> >>>>> has
>>>>> >> >>>>> on current/future features.
>>>>> >> >>>>>
>>>>> >> >>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> In particular: dynamic_ownership was
set to 0
prehistorically
>>>>> >> >>>>> (as
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> part
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>>>> because
>>>>> >> >>>>>
>>>>> >> >>>>> libvirt,
>>>>> >> >>>>> running as root, was not able to play
properly with
root-squash
>>>>> >> >>>>> nfs
>>>>> >> >>>>> mounts.
>>>>> >> >>>>>
>>>>> >> >>>>> Have you attempted this use case?
>>>>> >> >>>>>
>>>>> >> >>>>> I join to Nir's request to run
this with storage QE.
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> --
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>> Raz Tamir
>>>>> >> >>>>> Manager, RHV QE
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
_______________________________________________
>>>>> >> >>>>> Devel mailing list
>>>>> >> >>>>> Devel(a)ovirt.org
>>>>> >> >>>>>
http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>
>>>>> >> >>
>>>>> >> >
>>>>> >> > <logs.tar.gz>
>>>>> >> >
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WGMUK5T7PDY...
6:58 a.m.
New subject: dynamic ownership changes
Martin, please update, if you think the failures are not related to your
patch I'll test with the master as Nir suggested.
Thanks
On Thu, May 31, 2018 at 1:19 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Thu, May 31, 2018 at 1:05 PM Martin Polednik
<mpolednik(a)redhat.com>
wrote:
> On 31/05/18 12:47 +0300, Elad Ben Aharon wrote:
> >Execution is done, 59/65 cases passed. Latest 4.2.4 execution ended with
> >100% so failures were caused probably due to the changes done in the
> patch.
> >Failures are mainly on preview snapshots.
>
Can we run the same job on the patch before Martin patch? maybe
the issue are already in master, caused by other patches?
> >
> >Execution info provided to Martin separately.
>
> I'm currently investigating the snapshot breakage, thanks Elad!
>
> >On Wed, May 30, 2018 at 5:44 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
> >wrote:
> >
> >> Triggered a sanity automation execution using [1], which covers all the
> >> requested areas, on iSCSI, NFS and Gluster.
> >> I'll update with the results.
> >>
> >> [1]
> >> *https://gerrit.ovirt.org/#/c/90906/ <https://gerrit.ovirt.org/#/c/
> 90906/>*
> >> vdsm-4.20.28-6.gitc23aef6.el7.x86_64
> >>
> >>
> >> On Tue, May 29, 2018 at 4:26 PM, Martin Polednik <mpolednik(a)redhat.com
> >
> >> wrote:
> >>
> >>> On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
> >>>
> >>>> Hi Martin,
> >>>>
> >>>> Can you please create a cerry pick patch that is based on 4.2?
> >>>>
> >>>
> >>> See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn unrelated
> >>> (storage needs real env).
> >>>
> >>> mpolednik
> >>>
> >>>
> >>>
> >>>> Thanks
> >>>>
> >>>> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg
<danken(a)redhat.com>
> >>>> wrote:
> >>>>
> >>>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <
> ebenahar(a)redhat.com>
> >>>>> wrote:
> >>>>> > Hi Dan,
> >>>>> >
> >>>>> > In the last execution, the success rate was very low due to
a
> large
> >>>>> number
> >>>>> > of failures on start VM caused, according to Michal, by
the
> >>>>> > vdsm-hook-allocate_net that was installed on the host.
> >>>>> >
> >>>>> > This is the latest status here, would you like me to
re-execute?
> >>>>>
> >>>>> yes, of course. but you should rebase Polednik's code on top
of
> >>>>> *current* ovirt-4.2.3 branch.
> >>>>>
> >>>>> > If so, with
> >>>>> > or W/O vdsm-hook-allocate_net installed?
> >>>>>
> >>>>> There was NO reason to have that installed. Please keep it (and
any
> >>>>> other needless code) out of the test environment.
> >>>>>
> >>>>> >
> >>>>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg <
> danken(a)redhat.com>
> >>>>> wrote:
> >>>>> >>
> >>>>> >> On Mon, May 7, 2018 at 3:53 PM, Michal Skrivanek
> >>>>> >> <michal.skrivanek(a)redhat.com> wrote:
> >>>>> >> > Hi Elad,
> >>>>> >> > why did you install vdsm-hook-allocate_net?
> >>>>> >> >
> >>>>> >> > adding Dan as I think the hook is not supposed to
fail this
> badly
> >>>>> in
> >>>>> any
> >>>>> >> > case
> >>>>> >>
> >>>>> >> yep, this looks bad and deserves a little bug report.
Installing
> this
> >>>>> >> little hook should not block vm startup.
> >>>>> >>
> >>>>> >> But more importantly - what is the conclusion of this
thread? Do
> we
> >>>>> >> have a green light from QE to take this in?
> >>>>> >>
> >>>>> >>
> >>>>> >> >
> >>>>> >> > Thanks,
> >>>>> >> > michal
> >>>>> >> >
> >>>>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
> >>>>> wrote:
> >>>>> >> >
> >>>>> >> > Start VM fails on:
> >>>>> >> >
> >>>>> >> > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce)
[virt.vm]
> >>>>> >> >
(vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda'
> path:
> >>>>> >> >
> >>>>> >> >
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
> >>>>> 9a78-bdd13a843c62/images/6cdabfe5-
> >>>>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a0192
> 8211'
> >>>>> ->
> >>>>> >> >
> >>>>> >> >
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
> >>>>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
> >>>>> 9834f235d1c8/7ef97445-30e6-4435-8425-
> >>>>> >> > f35a01928211' (storagexml:334)
> >>>>> >> > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1)
[vdsm.api] START
> >>>>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
> >>>>> >> > options=None)
> >>>>> >> > from=::ffff:10.35.161.127,53512,
> >>>>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
> >>>>> >> > 2758 (api:46)
> >>>>> >> > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce)
[root]
> >>>>> >> >
/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net:
> rc=2
> >>>>> >> > err=vm
> >>>>> >> > net allocation hook: [unexpected error]: Traceback
(most recent
> >>>>> call
> >>>>> >> > last):
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 105, in <module>
> >>>>> >> > main()
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 93, in main
> >>>>> >> > allocate_random_network(device_xml)
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 62, in allocate_random_network
> >>>>> >> > net = _get_random_network()
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 50, in _get_random_network
> >>>>> >> > available_nets = _parse_nets()
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 46, in _parse_nets
> >>>>> >> > return [net for net in
os.environ[AVAIL_NETS_KEY].split()]
> >>>>> >> > File
"/usr/lib64/python2.7/UserDict.py", line 23, in
> __getitem__
> >>>>> >> > raise KeyError(key)
> >>>>> >> > KeyError: 'equivnets'
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > (hooks:110)
> >>>>> >> > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce)
[virt.vm]
> >>>>> >> >
(vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start
> process
> >>>>> >> > failed
> >>>>> >> > (vm:943)
> >>>>> >> > Traceback (most recent call last):
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> >>>>> 872,
> >>>>> in
> >>>>> >> > _startUnderlyingVm
> >>>>> >> > self._run()
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> >>>>> 2861,
> >>>>> in
> >>>>> >> > _run
> >>>>> >> > domxml =
hooks.before_vm_start(self._buildDomainXML(),
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> >>>>> 2254,
> >>>>> in
> >>>>> >> > _buildDomainXML
> >>>>> >> > dom, self.id, self._custom['custom'])
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
> >>>>> preprocess.py",
> >>>>> >> > line 240, in replace_device_xml_with_hooks_xml
> >>>>> >> > dev_custom)
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
> >>>>> line
> >>>>> 134,
> >>>>> >> > in
> >>>>> >> > before_device_create
> >>>>> >> > params=customProperties)
> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
> >>>>> line
> >>>>> 120,
> >>>>> >> > in
> >>>>> >> > _runHooksDir
> >>>>> >> > raise exception.HookError(err)
> >>>>> >> > HookError: Hook Error: ('vm net allocation
hook: [unexpected
> >>>>> error]:
> >>>>> >> > Traceback (most recent call last):\n File
> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
> net",
> >>>>> line
> >>>>> >> > 105, in
> >>>>> >> > <module>\n main()\n
> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
> e_device_create/10_allocate_ne
> >>>>> t",
> >>>>> >> > line
> >>>>> >> > 93, in main\n
allocate_random_network(device_xml)\n File
> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
> net",
> >>>>> line
> >>>>> 62,
> >>>>> >> > i
> >>>>> >> > n allocate_random_network\n net =
_get_random_network()\n
> File
> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
> net",
> >>>>> line
> >>>>> 50,
> >>>>> >> > in
> >>>>> >> > _get_random_network\n available_nets =
_parse_nets()\n File
> >>>>> "/us
> >>>>> >> >
r/libexec/vdsm/hooks/before_device_create/10_allocate_net",
> line
> >>>>> 46,
> >>>>> in
> >>>>> >> > _parse_nets\n return [net for net in
> >>>>> >> > os.environ[AVAIL_NETS_KEY].split()]\n File
> >>>>> >> > "/usr/lib64/python2.7/UserDict.py", line
23, in __getit
> >>>>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Hence, the success rate was 28% against 100%
running with d/s
> >>>>> (d/s).
> >>>>> If
> >>>>> >> > needed, I'll compare against the latest
master, but I think
> you get
> >>>>> the
> >>>>> >> > picture with d/s.
> >>>>> >> >
> >>>>> >> > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
> >>>>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
> >>>>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
> >>>>> >> > kernel 3.10.0-862.el7.x86_64
> >>>>> >> > rhel7.5
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Logs attached
> >>>>> >> >
> >>>>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon
<
> >>>>> ebenahar(a)redhat.com>
> >>>>> >> > wrote:
> >>>>> >> >>
> >>>>> >> >> nvm, found gluster 3.12 repo, managed to
install vdsm
> >>>>> >> >>
> >>>>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad Ben
Aharon <
> >>>>> ebenahar(a)redhat.com
> >>>>> >
> >>>>> >> >> wrote:
> >>>>> >> >>>
> >>>>> >> >>> No, vdsm requires it:
> >>>>> >> >>>
> >>>>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
> >>>>> >> >>>
(/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
> >>>>> >> >>> Requires: glusterfs-fuse >=
3.12
> >>>>> >> >>> Installed:
glusterfs-fuse-3.8.4-54.8.el7.x86_64
> >>>>> (@rhv-4.2.3)
> >>>>> >> >>>
> >>>>> >> >>> Therefore, vdsm package installation is
skipped upon force
> >>>>> install.
> >>>>> >> >>>
> >>>>> >> >>> On Sat, May 5, 2018 at 11:42 AM, Michal
Skrivanek
> >>>>> >> >>> <michal.skrivanek(a)redhat.com>
wrote:
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>> On 5 May 2018, at 00:38, Elad Ben
Aharon <
> ebenahar(a)redhat.com>
> >>>>> wrote:
> >>>>> >> >>>>
> >>>>> >> >>>> Hi guys,
> >>>>> >> >>>>
> >>>>> >> >>>> The vdsm build from the patch requires
glusterfs-fuse >
> 3.12.
> >>>>> This
> >>>>> is
> >>>>> >> >>>> while the latest 4.2.3-5 d/s build
requires 3.8.4
> >>>>> (3.4.0.59rhs-1.el7)
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>> because it is still oVirt, not a
downstream build. We can’t
> >>>>> really
> >>>>> do
> >>>>> >> >>>> downstream builds with unmerged
changes:/
> >>>>> >> >>>>
> >>>>> >> >>>> Trying to get this gluster-fuse build,
so far no luck.
> >>>>> >> >>>> Is this requirement intentional?
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>> it should work regardless, I guess you
can force install it
> >>>>> without
> >>>>> >> >>>> the
> >>>>> >> >>>> dependency
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>> On Fri, May 4, 2018 at 2:38 PM, Michal
Skrivanek
> >>>>> >> >>>> <michal.skrivanek(a)redhat.com>
wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> Hi Elad,
> >>>>> >> >>>>> to make it easier to compare,
Martin backported the change
> to
> >>>>> 4.2
> >>>>> so
> >>>>> >> >>>>> it
> >>>>> >> >>>>> is actually comparable with a run
without that patch.
> Would you
> >>>>> >> >>>>> please try
> >>>>> >> >>>>> that out?
> >>>>> >> >>>>> It would be best to have 4.2
upstream and this[1] run to
> really
> >>>>> >> >>>>> minimize the noise.
> >>>>> >> >>>>>
> >>>>> >> >>>>> Thanks,
> >>>>> >> >>>>> michal
> >>>>> >> >>>>>
> >>>>> >> >>>>> [1]
> >>>>> >> >>>>>
> >>>>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
> >>>>> demand-el7-x86_64/28/
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 27 Apr 2018, at 09:23, Martin
Polednik <
> >>>>> mpolednik(a)redhat.com>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 24/04/18 00:37 +0300, Elad Ben
Aharon wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> I will update with the results of
the next tier1 execution
> on
> >>>>> latest
> >>>>> >> >>>>> 4.2.3
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> That isn't master but old
branch though. Could you run it
> >>>>> against
> >>>>> >> >>>>> *current* VDSM master?
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Mon, Apr 23, 2018 at 3:56 PM,
Martin Polednik
> >>>>> >> >>>>> <mpolednik(a)redhat.com>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 23/04/18 01:23 +0300, Elad Ben
Aharon wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> Hi, I've triggered another
execution [1] due to some
> issues I
> >>>>> saw
> >>>>> in
> >>>>> >> >>>>> the
> >>>>> >> >>>>> first which are not related to the
patch.
> >>>>> >> >>>>>
> >>>>> >> >>>>> The success rate is 78% which is
low comparing to tier1
> >>>>> executions
> >>>>> >> >>>>> with
> >>>>> >> >>>>> code from downstream builds
(95-100% success rates) [2].
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Could you run the current master
(without the
> dynamic_ownership
> >>>>> >> >>>>> patch)
> >>>>> >> >>>>> so that we have viable
comparision?
> >>>>> >> >>>>>
> >>>>> >> >>>>> From what I could see so far,
there is an issue with move
> and
> >>>>> copy
> >>>>> >> >>>>>
> >>>>> >> >>>>> operations to and from Gluster
domains. For example [3].
> >>>>> >> >>>>>
> >>>>> >> >>>>> The logs are attached.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> [1]
> >>>>> >> >>>>> *https://rhv-jenkins.rhev-ci-v
> ms.eng.rdu2.redhat.com/job/rhv
> >>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/
> >>>>> >> >>>>> <https://rhv-jenkins.rhev-ci-v
> ms.eng.rdu2.redhat.com/job/rhv
> >>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> [2]
> >>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
> >>>>> >> >>>>>
> >>>>> >> >>>>>
rhv-4.2-ge-runner-tier1-after-upgrade/7/
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> [3]
> >>>>> >> >>>>> 2018-04-22 13:06:28,316+0300 INFO
(jsonrpc/7) [vdsm.api]
> >>>>> FINISH
> >>>>> >> >>>>> deleteImage error=Image does not
exist in domain:
> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> >>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
> >>>>> >> >>>>> from=:
> >>>>> >> >>>>> :ffff:10.35.161.182,40936,
> >>>>> >> >>>>>
flow_id=disks_syncAction_ba6b2630-5976-4935,
> >>>>> >> >>>>>
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
> >>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 ERROR
(jsonrpc/7)
> >>>>> >> >>>>> [storage.TaskManager.Task]
> >>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
> error
> >>>>> >> >>>>> (task:875)
> >>>>> >> >>>>> Traceback (most recent call
last):
> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
> ges/vdsm/storage/task.py",
> >>>>> line
> >>>>> >> >>>>> 882,
> >>>>> >> >>>>> in
> >>>>> >> >>>>> _run
> >>>>> >> >>>>> return fn(*args, **kargs)
> >>>>> >> >>>>> File "<string>",
line 2, in deleteImage
> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
> ges/vdsm/common/api.py",
> >>>>> line
> >>>>> 49,
> >>>>> >> >>>>> in
> >>>>> >> >>>>> method
> >>>>> >> >>>>> ret = func(*args, **kwargs)
> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
> ges/vdsm/storage/hsm.py",
> >>>>> line
> >>>>> >> >>>>> 1503,
> >>>>> >> >>>>> in
> >>>>> >> >>>>> deleteImage
> >>>>> >> >>>>> raise
se.ImageDoesNotExistInSD(imgUUID, sdUUID)
> >>>>> >> >>>>> ImageDoesNotExistInSD: Image does
not exist in domain:
> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> >>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
> >>>>> >> >>>>>
> >>>>> >> >>>>> 2018-04-22 13:06:28,317+0300 INFO
(jsonrpc/7)
> >>>>> >> >>>>> [storage.TaskManager.Task]
> >>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting:
> Task
> >>>>> is
> >>>>> >> >>>>> aborted:
> >>>>> >> >>>>> "Image does not exist in
domain:
> 'image=cabb8846-7a4b-4244-9835
> >>>>> -
> >>>>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'"
> -
> >>>>> code
> >>>>> >> >>>>> 268
> >>>>> >> >>>>> (task:1181)
> >>>>> >> >>>>> 2018-04-22 13:06:28,318+0300 ERROR
(jsonrpc/7)
> >>>>> [storage.Dispatcher]
> >>>>> >> >>>>> FINISH
> >>>>> >> >>>>> deleteImage error=Image does not
exist in domain:
> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> >>>>> >> >>>>> domain=e5fd29c8-52ba-467e-be09
> >>>>> >> >>>>> -ca40ff054d
> >>>>> >> >>>>> d4' (dispatcher:82)
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Thu, Apr 19, 2018 at 5:34 PM,
Elad Ben Aharon
> >>>>> >> >>>>> <ebenahar(a)redhat.com>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> Triggered a sanity tier1 execution
[1] using [2], which
> covers
> >>>>> all
> >>>>> >> >>>>> the
> >>>>> >> >>>>>
> >>>>> >> >>>>> requested areas, on iSCSI, NFS and
Gluster.
> >>>>> >> >>>>> I'll update with the results.
> >>>>> >> >>>>>
> >>>>> >> >>>>> [1]
> >>>>> >> >>>>> https://rhv-jenkins.rhev-ci-vm
> s.eng.rdu2.redhat.com/view/4.2
> >>>>> >> >>>>>
_dev/job/rhv-4.2-ge-flow-storage/1161/
> >>>>> >> >>>>>
> >>>>> >> >>>>> [2]
> >>>>> >> >>>>>
https://gerrit.ovirt.org/#/c/89830/
> >>>>> >> >>>>>
vdsm-4.30.0-291.git77aef9a.el7.x86_64
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Thu, Apr 19, 2018 at 3:07 PM,
Martin Polednik
> >>>>> >> >>>>> <mpolednik(a)redhat.com>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 19/04/18 14:54 +0300, Elad Ben
Aharon wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Hi Martin,
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> I see [1] requires a rebase, can
you please take care?
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Should be rebased.
> >>>>> >> >>>>>
> >>>>> >> >>>>> At the moment, our automation is
stable only on iSCSI, NFS,
> >>>>> Gluster
> >>>>> >> >>>>> and
> >>>>> >> >>>>>
> >>>>> >> >>>>> FC.
> >>>>> >> >>>>> Ceph is not supported and Cinder
will be stabilized soon,
> >>>>> AFAIR,
> >>>>> >> >>>>> it's
> >>>>> >> >>>>> not
> >>>>> >> >>>>> stable enough at the moment.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> That is still pretty good.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> [1]
https://gerrit.ovirt.org/#/c/89830/
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Thanks
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 18, 2018 at 2:17 PM,
Martin Polednik
> >>>>> >> >>>>> <mpolednik(a)redhat.com
> >>>>> >> >>>>> >
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 18/04/18 11:37 +0300, Elad Ben
Aharon wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Hi, sorry if I misunderstood, I
waited for more input
> regarding
> >>>>> what
> >>>>> >> >>>>>
> >>>>> >> >>>>> areas
> >>>>> >> >>>>> have to be tested here.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> I'd say that you have quite a
bit of freedom in this
> regard.
> >>>>> >> >>>>>
> >>>>> >> >>>>> GlusterFS
> >>>>> >> >>>>> should be covered by Dennis, so
iSCSI/NFS/ceph/cinder with
> some
> >>>>> >> >>>>> suite
> >>>>> >> >>>>> that covers basic operations
(start & stop VM, migrate it),
> >>>>> >> >>>>> snapshots
> >>>>> >> >>>>> and merging them, and whatever
else would be important for
> >>>>> storage
> >>>>> >> >>>>> sanity.
> >>>>> >> >>>>>
> >>>>> >> >>>>> mpolednik
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 18, 2018 at 11:16 AM,
Martin Polednik <
> >>>>> >> >>>>> mpolednik(a)redhat.com
> >>>>> >> >>>>> >
> >>>>> >> >>>>>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On 11/04/18 16:52 +0300, Elad Ben
Aharon wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> We can test this on iSCSI, NFS and
GlusterFS. As for ceph
> and
> >>>>> >> >>>>> cinder,
> >>>>> >> >>>>>
> >>>>> >> >>>>> will
> >>>>> >> >>>>>
> >>>>> >> >>>>> have to check, since usually, we
don't execute our
> automation
> >>>>> on
> >>>>> >> >>>>> them.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Any update on this? I believe the
gluster tests were
> >>>>> successful,
> >>>>> >> >>>>> OST
> >>>>> >> >>>>>
> >>>>> >> >>>>> passes fine and unit tests pass
fine, that makes the
> storage
> >>>>> >> >>>>> backends
> >>>>> >> >>>>> test the last required piece.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:38 PM,
Raz Tamir <
> ratamir(a)redhat.com
> >>>>> >
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> +Elad
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 11, 2018 at 4:28 PM,
Dan Kenigsberg <
> >>>>> danken(a)redhat.com
> >>>>> >> >>>>>
> >>>>> >> >>>>> >
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:34 PM,
Nir Soffer <
> >>>>> nsoffer(a)redhat.com>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Wed, Apr 11, 2018 at 12:31 PM
Eyal Edri <
> eedri(a)redhat.com>
> >>>>> >> >>>>>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Please make sure to run as much
OST suites on this patch as
> >>>>> >> >>>>>
> >>>>> >> >>>>> possible
> >>>>> >> >>>>>
> >>>>> >> >>>>> before merging ( using 'ci
please build' )
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> But note that OST is not a way to
verify the patch.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Such changes require testing with
all storage types we
> support.
> >>>>> >> >>>>>
> >>>>> >> >>>>> Nir
> >>>>> >> >>>>>
> >>>>> >> >>>>> On Tue, Apr 10, 2018 at 4:09 PM,
Martin Polednik <
> >>>>> >> >>>>> mpolednik(a)redhat.com
> >>>>> >> >>>>> >
> >>>>> >> >>>>>
> >>>>> >> >>>>> wrote:
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Hey,
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> I've created a patch[0] that
is finally able to activate
> >>>>> >> >>>>>
> >>>>> >> >>>>> libvirt's
> >>>>> >> >>>>> dynamic_ownership for VDSM while
not negatively affecting
> >>>>> >> >>>>> functionality of our storage
code.
> >>>>> >> >>>>>
> >>>>> >> >>>>> That of course comes with quite a
bit of code removal,
> mostly
> >>>>> >> >>>>> in
> >>>>> >> >>>>> the
> >>>>> >> >>>>> area of host devices, hwrng and
anything that touches
> devices;
> >>>>> >> >>>>> bunch
> >>>>> >> >>>>> of test changes and one XML
generation caveat (storage is
> >>>>> >> >>>>> handled
> >>>>> >> >>>>> by
> >>>>> >> >>>>> VDSM, therefore disk relabelling
needs to be disabled on
> the
> >>>>> >> >>>>> VDSM
> >>>>> >> >>>>> level).
> >>>>> >> >>>>>
> >>>>> >> >>>>> Because of the scope of the patch,
I welcome
> >>>>> >> >>>>> storage/virt/network
> >>>>> >> >>>>> people to review the code and
consider the implication this
> >>>>> >> >>>>> change
> >>>>> >> >>>>> has
> >>>>> >> >>>>> on current/future features.
> >>>>> >> >>>>>
> >>>>> >> >>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> In particular: dynamic_ownership
was set to 0
> prehistorically
> >>>>> >> >>>>> (as
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> part
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
> >>>>> because
> >>>>> >> >>>>>
> >>>>> >> >>>>> libvirt,
> >>>>> >> >>>>> running as root, was not able to
play properly with
> root-squash
> >>>>> >> >>>>> nfs
> >>>>> >> >>>>> mounts.
> >>>>> >> >>>>>
> >>>>> >> >>>>> Have you attempted this use case?
> >>>>> >> >>>>>
> >>>>> >> >>>>> I join to Nir's request to run
this with storage QE.
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> --
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>> Raz Tamir
> >>>>> >> >>>>> Manager, RHV QE
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
_______________________________________________
> >>>>> >> >>>>> Devel mailing list
> >>>>> >> >>>>> Devel(a)ovirt.org
> >>>>> >> >>>>>
http://lists.ovirt.org/mailman/listinfo/devel
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>>
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>
> >>>>> >> >>
> >>>>> >> >
> >>>>> >> > <logs.tar.gz>
> >>>>> >> >
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/communit
> y/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archiv
> es/list/devel(a)ovirt.org/message/WGMUK5T7PDYSIBIUZE3AJIYQAWJOILTC/
>
7:56 a.m.
New subject: dynamic ownership changes
On 31/05/18 14:58 +0300, Elad Ben Aharon wrote:
Martin, please update, if you think the failures are not related to
your
patch I'll test with the master as Nir suggested.
Thanks
I believe the failures are related to the patch, and more specifically
to the way libvirt handles seclabel for snapshots.
Opened https://bugzilla.redhat.com/show_bug.cgi?id=1584682.
>On Thu, May 31, 2018 at 1:19 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Thu, May 31, 2018 at 1:05 PM Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> On 31/05/18 12:47 +0300, Elad Ben Aharon wrote:
>>> >Execution is done, 59/65 cases passed. Latest 4.2.4 execution ended with
>>> >100% so failures were caused probably due to the changes done in the
>>> patch.
>>> >Failures are mainly on preview snapshots.
>>>
>>
>> Can we run the same job on the patch before Martin patch? maybe
>> the issue are already in master, caused by other patches?
>>
>>
>>> >
>>> >Execution info provided to Martin separately.
>>>
>>> I'm currently investigating the snapshot breakage, thanks Elad!
>>>
>>> >On Wed, May 30, 2018 at 5:44 PM, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>> >wrote:
>>> >
>>> >> Triggered a sanity automation execution using [1], which covers all
the
>>> >> requested areas, on iSCSI, NFS and Gluster.
>>> >> I'll update with the results.
>>> >>
>>> >> [1]
>>> >> *https://gerrit.ovirt.org/#/c/90906/
<https://gerrit.ovirt.org/#/c/
>>> 90906/>*
>>> >> vdsm-4.20.28-6.gitc23aef6.el7.x86_64
>>> >>
>>> >>
>>> >> On Tue, May 29, 2018 at 4:26 PM, Martin Polednik
<mpolednik(a)redhat.com
>>> >
>>> >> wrote:
>>> >>
>>> >>> On 29/05/18 15:30 +0300, Elad Ben Aharon wrote:
>>> >>>
>>> >>>> Hi Martin,
>>> >>>>
>>> >>>> Can you please create a cerry pick patch that is based on
4.2?
>>> >>>>
>>> >>>
>>> >>> See https://gerrit.ovirt.org/#/c/90906/. The CI failure isn
unrelated
>>> >>> (storage needs real env).
>>> >>>
>>> >>> mpolednik
>>> >>>
>>> >>>
>>> >>>
>>> >>>> Thanks
>>> >>>>
>>> >>>> On Tue, May 29, 2018 at 1:34 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> On Tue, May 29, 2018 at 1:21 PM, Elad Ben Aharon <
>>> ebenahar(a)redhat.com>
>>> >>>>> wrote:
>>> >>>>> > Hi Dan,
>>> >>>>> >
>>> >>>>> > In the last execution, the success rate was very
low due to a
>>> large
>>> >>>>> number
>>> >>>>> > of failures on start VM caused, according to
Michal, by the
>>> >>>>> > vdsm-hook-allocate_net that was installed on the
host.
>>> >>>>> >
>>> >>>>> > This is the latest status here, would you like me
to re-execute?
>>> >>>>>
>>> >>>>> yes, of course. but you should rebase Polednik's
code on top of
>>> >>>>> *current* ovirt-4.2.3 branch.
>>> >>>>>
>>> >>>>> > If so, with
>>> >>>>> > or W/O vdsm-hook-allocate_net installed?
>>> >>>>>
>>> >>>>> There was NO reason to have that installed. Please keep
it (and any
>>> >>>>> other needless code) out of the test environment.
>>> >>>>>
>>> >>>>> >
>>> >>>>> > On Tue, May 29, 2018 at 1:14 PM, Dan Kenigsberg
<
>>> danken(a)redhat.com>
>>> >>>>> wrote:
>>> >>>>> >>
>>> >>>>> >> On Mon, May 7, 2018 at 3:53 PM, Michal
Skrivanek
>>> >>>>> >> <michal.skrivanek(a)redhat.com> wrote:
>>> >>>>> >> > Hi Elad,
>>> >>>>> >> > why did you install
vdsm-hook-allocate_net?
>>> >>>>> >> >
>>> >>>>> >> > adding Dan as I think the hook is not
supposed to fail this
>>> badly
>>> >>>>> in
>>> >>>>> any
>>> >>>>> >> > case
>>> >>>>> >>
>>> >>>>> >> yep, this looks bad and deserves a little bug
report. Installing
>>> this
>>> >>>>> >> little hook should not block vm startup.
>>> >>>>> >>
>>> >>>>> >> But more importantly - what is the conclusion
of this thread? Do
>>> we
>>> >>>>> >> have a green light from QE to take this in?
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> >
>>> >>>>> >> > Thanks,
>>> >>>>> >> > michal
>>> >>>>> >> >
>>> >>>>> >> > On 5 May 2018, at 19:22, Elad Ben Aharon
<ebenahar(a)redhat.com>
>>> >>>>> wrote:
>>> >>>>> >> >
>>> >>>>> >> > Start VM fails on:
>>> >>>>> >> >
>>> >>>>> >> > 2018-05-05 17:53:27,399+0300 INFO
(vm/e6ce66ce) [virt.vm]
>>> >>>>> >> >
(vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda'
>>> path:
>>> >>>>> >> >
>>> >>>>> >> >
'dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>> >>>>> 9a78-bdd13a843c62/images/6cdabfe5-
>>> >>>>> >> >
d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a0192
>>> 8211'
>>> >>>>> ->
>>> >>>>> >> >
>>> >>>>> >> >
u'*dev=/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-
>>> >>>>> 9a78-bdd13a843c62/images/6cdabfe5-d1ca-40af-ae63-
>>> >>>>> 9834f235d1c8/7ef97445-30e6-4435-8425-
>>> >>>>> >> > f35a01928211' (storagexml:334)
>>> >>>>> >> > 2018-05-05 17:53:27,888+0300 INFO
(jsonrpc/1) [vdsm.api] START
>>> >>>>> >> >
getSpmStatus(spUUID='940fe6f3-b0c6-4d0c-a921-198e7819c1cc',
>>> >>>>> >> > options=None)
>>> >>>>> >> > from=::ffff:10.35.161.127,53512,
>>> >>>>> >> > task_id=c70ace39-dbfe-4f5c-ae49-a1e3a82c
>>> >>>>> >> > 2758 (api:46)
>>> >>>>> >> > 2018-05-05 17:53:27,909+0300 INFO
(vm/e6ce66ce) [root]
>>> >>>>> >> >
/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net:
>>> rc=2
>>> >>>>> >> > err=vm
>>> >>>>> >> > net allocation hook: [unexpected error]:
Traceback (most recent
>>> >>>>> call
>>> >>>>> >> > last):
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 105, in <module>
>>> >>>>> >> > main()
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 93, in main
>>> >>>>> >> > allocate_random_network(device_xml)
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 62, in allocate_random_network
>>> >>>>> >> > net = _get_random_network()
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 50, in _get_random_network
>>> >>>>> >> > available_nets = _parse_nets()
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 46, in _parse_nets
>>> >>>>> >> > return [net for net in
os.environ[AVAIL_NETS_KEY].split()]
>>> >>>>> >> > File
"/usr/lib64/python2.7/UserDict.py", line 23, in
>>> __getitem__
>>> >>>>> >> > raise KeyError(key)
>>> >>>>> >> > KeyError: 'equivnets'
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> > (hooks:110)
>>> >>>>> >> > 2018-05-05 17:53:27,915+0300 ERROR
(vm/e6ce66ce) [virt.vm]
>>> >>>>> >> >
(vmId='e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start
>>> process
>>> >>>>> >> > failed
>>> >>>>> >> > (vm:943)
>>> >>>>> >> > Traceback (most recent call last):
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> >>>>> 872,
>>> >>>>> in
>>> >>>>> >> > _startUnderlyingVm
>>> >>>>> >> > self._run()
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> >>>>> 2861,
>>> >>>>> in
>>> >>>>> >> > _run
>>> >>>>> >> > domxml =
hooks.before_vm_start(self._buildDomainXML(),
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>> >>>>> 2254,
>>> >>>>> in
>>> >>>>> >> > _buildDomainXML
>>> >>>>> >> > dom, self.id,
self._custom['custom'])
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/virt/domxml_
>>> >>>>> preprocess.py",
>>> >>>>> >> > line 240, in
replace_device_xml_with_hooks_xml
>>> >>>>> >> > dev_custom)
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>> >>>>> line
>>> >>>>> 134,
>>> >>>>> >> > in
>>> >>>>> >> > before_device_create
>>> >>>>> >> > params=customProperties)
>>> >>>>> >> > File
"/usr/lib/python2.7/site-packages/vdsm/common/hooks.py",
>>> >>>>> line
>>> >>>>> 120,
>>> >>>>> >> > in
>>> >>>>> >> > _runHooksDir
>>> >>>>> >> > raise exception.HookError(err)
>>> >>>>> >> > HookError: Hook Error: ('vm net
allocation hook: [unexpected
>>> >>>>> error]:
>>> >>>>> >> > Traceback (most recent call last):\n
File
>>> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>>> net",
>>> >>>>> line
>>> >>>>> >> > 105, in
>>> >>>>> >> > <module>\n main()\n
>>> >>>>> >> > File "/usr/libexec/vdsm/hooks/befor
>>> e_device_create/10_allocate_ne
>>> >>>>> t",
>>> >>>>> >> > line
>>> >>>>> >> > 93, in main\n
allocate_random_network(device_xml)\n File
>>> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>>> net",
>>> >>>>> line
>>> >>>>> 62,
>>> >>>>> >> > i
>>> >>>>> >> > n allocate_random_network\n net =
_get_random_network()\n
>>> File
>>> >>>>> >> >
"/usr/libexec/vdsm/hooks/before_device_create/10_allocate_
>>> net",
>>> >>>>> line
>>> >>>>> 50,
>>> >>>>> >> > in
>>> >>>>> >> > _get_random_network\n available_nets =
_parse_nets()\n File
>>> >>>>> "/us
>>> >>>>> >> >
r/libexec/vdsm/hooks/before_device_create/10_allocate_net",
>>> line
>>> >>>>> 46,
>>> >>>>> in
>>> >>>>> >> > _parse_nets\n return [net for net in
>>> >>>>> >> > os.environ[AVAIL_NETS_KEY].split()]\n
File
>>> >>>>> >> >
"/usr/lib64/python2.7/UserDict.py", line 23, in __getit
>>> >>>>> >> > em__\n raise KeyError(key)\nKeyError:
\'equivnets\'\n\n\n',)
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> > Hence, the success rate was 28% against
100% running with d/s
>>> >>>>> (d/s).
>>> >>>>> If
>>> >>>>> >> > needed, I'll compare against the
latest master, but I think
>>> you get
>>> >>>>> the
>>> >>>>> >> > picture with d/s.
>>> >>>>> >> >
>>> >>>>> >> >
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> >>>>> >> > libvirt-3.9.0-14.el7_5.3.x86_64
>>> >>>>> >> > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64
>>> >>>>> >> > kernel 3.10.0-862.el7.x86_64
>>> >>>>> >> > rhel7.5
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> > Logs attached
>>> >>>>> >> >
>>> >>>>> >> > On Sat, May 5, 2018 at 1:26 PM, Elad Ben
Aharon <
>>> >>>>> ebenahar(a)redhat.com>
>>> >>>>> >> > wrote:
>>> >>>>> >> >>
>>> >>>>> >> >> nvm, found gluster 3.12 repo, managed
to install vdsm
>>> >>>>> >> >>
>>> >>>>> >> >> On Sat, May 5, 2018 at 1:12 PM, Elad
Ben Aharon <
>>> >>>>> ebenahar(a)redhat.com
>>> >>>>> >
>>> >>>>> >> >> wrote:
>>> >>>>> >> >>>
>>> >>>>> >> >>> No, vdsm requires it:
>>> >>>>> >> >>>
>>> >>>>> >> >>> Error: Package:
vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64
>>> >>>>> >> >>>
(/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)
>>> >>>>> >> >>> Requires: glusterfs-fuse
>= 3.12
>>> >>>>> >> >>> Installed:
glusterfs-fuse-3.8.4-54.8.el7.x86_64
>>> >>>>> (@rhv-4.2.3)
>>> >>>>> >> >>>
>>> >>>>> >> >>> Therefore, vdsm package
installation is skipped upon force
>>> >>>>> install.
>>> >>>>> >> >>>
>>> >>>>> >> >>> On Sat, May 5, 2018 at 11:42 AM,
Michal Skrivanek
>>> >>>>> >> >>>
<michal.skrivanek(a)redhat.com> wrote:
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> On 5 May 2018, at 00:38, Elad
Ben Aharon <
>>> ebenahar(a)redhat.com>
>>> >>>>> wrote:
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> Hi guys,
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> The vdsm build from the patch
requires glusterfs-fuse >
>>> 3.12.
>>> >>>>> This
>>> >>>>> is
>>> >>>>> >> >>>> while the latest 4.2.3-5 d/s
build requires 3.8.4
>>> >>>>> (3.4.0.59rhs-1.el7)
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> because it is still oVirt, not
a downstream build. We can’t
>>> >>>>> really
>>> >>>>> do
>>> >>>>> >> >>>> downstream builds with
unmerged changes:/
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> Trying to get this
gluster-fuse build, so far no luck.
>>> >>>>> >> >>>> Is this requirement
intentional?
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> it should work regardless, I
guess you can force install it
>>> >>>>> without
>>> >>>>> >> >>>> the
>>> >>>>> >> >>>> dependency
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>> On Fri, May 4, 2018 at 2:38
PM, Michal Skrivanek
>>> >>>>> >> >>>>
<michal.skrivanek(a)redhat.com> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Hi Elad,
>>> >>>>> >> >>>>> to make it easier to
compare, Martin backported the change
>>> to
>>> >>>>> 4.2
>>> >>>>> so
>>> >>>>> >> >>>>> it
>>> >>>>> >> >>>>> is actually comparable
with a run without that patch.
>>> Would you
>>> >>>>> >> >>>>> please try
>>> >>>>> >> >>>>> that out?
>>> >>>>> >> >>>>> It would be best to have
4.2 upstream and this[1] run to
>>> really
>>> >>>>> >> >>>>> minimize the noise.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Thanks,
>>> >>>>> >> >>>>> michal
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [1]
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-
>>> >>>>> demand-el7-x86_64/28/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 27 Apr 2018, at 09:23,
Martin Polednik <
>>> >>>>> mpolednik(a)redhat.com>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 24/04/18 00:37 +0300,
Elad Ben Aharon wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> I will update with the
results of the next tier1 execution
>>> on
>>> >>>>> latest
>>> >>>>> >> >>>>> 4.2.3
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> That isn't master but
old branch though. Could you run it
>>> >>>>> against
>>> >>>>> >> >>>>> *current* VDSM master?
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Mon, Apr 23, 2018 at
3:56 PM, Martin Polednik
>>> >>>>> >> >>>>>
<mpolednik(a)redhat.com>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 23/04/18 01:23 +0300,
Elad Ben Aharon wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Hi, I've triggered
another execution [1] due to some
>>> issues I
>>> >>>>> saw
>>> >>>>> in
>>> >>>>> >> >>>>> the
>>> >>>>> >> >>>>> first which are not
related to the patch.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> The success rate is 78%
which is low comparing to tier1
>>> >>>>> executions
>>> >>>>> >> >>>>> with
>>> >>>>> >> >>>>> code from downstream
builds (95-100% success rates) [2].
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Could you run the current
master (without the
>>> dynamic_ownership
>>> >>>>> >> >>>>> patch)
>>> >>>>> >> >>>>> so that we have viable
comparision?
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> From what I could see so
far, there is an issue with move
>>> and
>>> >>>>> copy
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> operations to and from
Gluster domains. For example [3].
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> The logs are attached.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [1]
>>> >>>>> >> >>>>>
*https://rhv-jenkins.rhev-ci-v
>>> ms.eng.rdu2.redhat.com/job/rhv
>>> >>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>> >>>>> >> >>>>>
<https://rhv-jenkins.rhev-ci-v
>>> ms.eng.rdu2.redhat.com/job/rhv
>>> >>>>> >> >>>>>
-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [2]
>>> >>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [3]
>>> >>>>> >> >>>>> 2018-04-22
13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api]
>>> >>>>> FINISH
>>> >>>>> >> >>>>> deleteImage error=Image
does not exist in domain:
>>> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> >>>>> >> >>>>> from=:
>>> >>>>> >> >>>>>
:ffff:10.35.161.182,40936,
>>> >>>>> >> >>>>>
flow_id=disks_syncAction_ba6b2630-5976-4935,
>>> >>>>> >> >>>>>
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>> >>>>> >> >>>>> 2018-04-22
13:06:28,317+0300 ERROR (jsonrpc/7)
>>> >>>>> >> >>>>>
[storage.TaskManager.Task]
>>> >>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
>>> error
>>> >>>>> >> >>>>> (task:875)
>>> >>>>> >> >>>>> Traceback (most recent
call last):
>>> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
>>> ges/vdsm/storage/task.py",
>>> >>>>> line
>>> >>>>> >> >>>>> 882,
>>> >>>>> >> >>>>> in
>>> >>>>> >> >>>>> _run
>>> >>>>> >> >>>>> return fn(*args,
**kargs)
>>> >>>>> >> >>>>> File
"<string>", line 2, in deleteImage
>>> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
>>> ges/vdsm/common/api.py",
>>> >>>>> line
>>> >>>>> 49,
>>> >>>>> >> >>>>> in
>>> >>>>> >> >>>>> method
>>> >>>>> >> >>>>> ret = func(*args,
**kwargs)
>>> >>>>> >> >>>>> File
"/usr/lib/python2.7/site-packa
>>> ges/vdsm/storage/hsm.py",
>>> >>>>> line
>>> >>>>> >> >>>>> 1503,
>>> >>>>> >> >>>>> in
>>> >>>>> >> >>>>> deleteImage
>>> >>>>> >> >>>>> raise
se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>> >>>>> >> >>>>> ImageDoesNotExistInSD:
Image does not exist in domain:
>>> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> 2018-04-22
13:06:28,317+0300 INFO (jsonrpc/7)
>>> >>>>> >> >>>>>
[storage.TaskManager.Task]
>>> >>>>> >> >>>>>
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting:
>>> Task
>>> >>>>> is
>>> >>>>> >> >>>>> aborted:
>>> >>>>> >> >>>>> "Image does not exist
in domain:
>>> 'image=cabb8846-7a4b-4244-9835
>>> >>>>> -
>>> >>>>> >> >>>>> 5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'"
>>> -
>>> >>>>> code
>>> >>>>> >> >>>>> 268
>>> >>>>> >> >>>>> (task:1181)
>>> >>>>> >> >>>>> 2018-04-22
13:06:28,318+0300 ERROR (jsonrpc/7)
>>> >>>>> [storage.Dispatcher]
>>> >>>>> >> >>>>> FINISH
>>> >>>>> >> >>>>> deleteImage error=Image
does not exist in domain:
>>> >>>>> >> >>>>>
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> >>>>> >> >>>>>
domain=e5fd29c8-52ba-467e-be09
>>> >>>>> >> >>>>> -ca40ff054d
>>> >>>>> >> >>>>> d4' (dispatcher:82)
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Thu, Apr 19, 2018 at
5:34 PM, Elad Ben Aharon
>>> >>>>> >> >>>>>
<ebenahar(a)redhat.com>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Triggered a sanity tier1
execution [1] using [2], which
>>> covers
>>> >>>>> all
>>> >>>>> >> >>>>> the
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> requested areas, on iSCSI,
NFS and Gluster.
>>> >>>>> >> >>>>> I'll update with the
results.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [1]
>>> >>>>> >> >>>>>
https://rhv-jenkins.rhev-ci-vm
>>> s.eng.rdu2.redhat.com/view/4.2
>>> >>>>> >> >>>>>
_dev/job/rhv-4.2-ge-flow-storage/1161/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [2]
>>> >>>>> >> >>>>>
https://gerrit.ovirt.org/#/c/89830/
>>> >>>>> >> >>>>>
vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Thu, Apr 19, 2018 at
3:07 PM, Martin Polednik
>>> >>>>> >> >>>>>
<mpolednik(a)redhat.com>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 19/04/18 14:54 +0300,
Elad Ben Aharon wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Hi Martin,
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> I see [1] requires a
rebase, can you please take care?
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Should be rebased.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> At the moment, our
automation is stable only on iSCSI, NFS,
>>> >>>>> Gluster
>>> >>>>> >> >>>>> and
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> FC.
>>> >>>>> >> >>>>> Ceph is not supported and
Cinder will be stabilized soon,
>>> >>>>> AFAIR,
>>> >>>>> >> >>>>> it's
>>> >>>>> >> >>>>> not
>>> >>>>> >> >>>>> stable enough at the
moment.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> That is still pretty
good.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [1]
https://gerrit.ovirt.org/#/c/89830/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Thanks
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 18, 2018 at
2:17 PM, Martin Polednik
>>> >>>>> >> >>>>> <mpolednik(a)redhat.com
>>> >>>>> >> >>>>> >
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 18/04/18 11:37 +0300,
Elad Ben Aharon wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Hi, sorry if I
misunderstood, I waited for more input
>>> regarding
>>> >>>>> what
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> areas
>>> >>>>> >> >>>>> have to be tested here.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> I'd say that you have
quite a bit of freedom in this
>>> regard.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> GlusterFS
>>> >>>>> >> >>>>> should be covered by
Dennis, so iSCSI/NFS/ceph/cinder with
>>> some
>>> >>>>> >> >>>>> suite
>>> >>>>> >> >>>>> that covers basic
operations (start & stop VM, migrate it),
>>> >>>>> >> >>>>> snapshots
>>> >>>>> >> >>>>> and merging them, and
whatever else would be important for
>>> >>>>> storage
>>> >>>>> >> >>>>> sanity.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> mpolednik
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 18, 2018 at
11:16 AM, Martin Polednik <
>>> >>>>> >> >>>>> mpolednik(a)redhat.com
>>> >>>>> >> >>>>> >
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On 11/04/18 16:52 +0300,
Elad Ben Aharon wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> We can test this on iSCSI,
NFS and GlusterFS. As for ceph
>>> and
>>> >>>>> >> >>>>> cinder,
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> will
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> have to check, since
usually, we don't execute our
>>> automation
>>> >>>>> on
>>> >>>>> >> >>>>> them.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Any update on this? I
believe the gluster tests were
>>> >>>>> successful,
>>> >>>>> >> >>>>> OST
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> passes fine and unit tests
pass fine, that makes the
>>> storage
>>> >>>>> >> >>>>> backends
>>> >>>>> >> >>>>> test the last required
piece.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 11, 2018 at
4:38 PM, Raz Tamir <
>>> ratamir(a)redhat.com
>>> >>>>> >
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> +Elad
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 11, 2018 at
4:28 PM, Dan Kenigsberg <
>>> >>>>> danken(a)redhat.com
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> >
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 11, 2018 at
12:34 PM, Nir Soffer <
>>> >>>>> nsoffer(a)redhat.com>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Wed, Apr 11, 2018 at
12:31 PM Eyal Edri <
>>> eedri(a)redhat.com>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Please make sure to run as
much OST suites on this patch as
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> possible
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> before merging ( using
'ci please build' )
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> But note that OST is not a
way to verify the patch.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Such changes require
testing with all storage types we
>>> support.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Nir
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> On Tue, Apr 10, 2018 at
4:09 PM, Martin Polednik <
>>> >>>>> >> >>>>> mpolednik(a)redhat.com
>>> >>>>> >> >>>>> >
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> wrote:
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Hey,
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> I've created a
patch[0] that is finally able to activate
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> libvirt's
>>> >>>>> >> >>>>> dynamic_ownership for VDSM
while not negatively affecting
>>> >>>>> >> >>>>> functionality of our
storage code.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> That of course comes with
quite a bit of code removal,
>>> mostly
>>> >>>>> >> >>>>> in
>>> >>>>> >> >>>>> the
>>> >>>>> >> >>>>> area of host devices,
hwrng and anything that touches
>>> devices;
>>> >>>>> >> >>>>> bunch
>>> >>>>> >> >>>>> of test changes and one
XML generation caveat (storage is
>>> >>>>> >> >>>>> handled
>>> >>>>> >> >>>>> by
>>> >>>>> >> >>>>> VDSM, therefore disk
relabelling needs to be disabled on
>>> the
>>> >>>>> >> >>>>> VDSM
>>> >>>>> >> >>>>> level).
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Because of the scope of
the patch, I welcome
>>> >>>>> >> >>>>> storage/virt/network
>>> >>>>> >> >>>>> people to review the code
and consider the implication this
>>> >>>>> >> >>>>> change
>>> >>>>> >> >>>>> has
>>> >>>>> >> >>>>> on current/future
features.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> In particular:
dynamic_ownership was set to 0
>>> prehistorically
>>> >>>>> >> >>>>> (as
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> part
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>> >>>>> because
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> libvirt,
>>> >>>>> >> >>>>> running as root, was not
able to play properly with
>>> root-squash
>>> >>>>> >> >>>>> nfs
>>> >>>>> >> >>>>> mounts.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Have you attempted this
use case?
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> I join to Nir's
request to run this with storage QE.
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> --
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>> Raz Tamir
>>> >>>>> >> >>>>> Manager, RHV QE
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
_______________________________________________
>>> >>>>> >> >>>>> Devel mailing list
>>> >>>>> >> >>>>> Devel(a)ovirt.org
>>> >>>>> >> >>>>>
http://lists.ovirt.org/mailman/listinfo/devel
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>>
>>> >>>>> >> >>>
>>> >>>>> >> >>
>>> >>>>> >> >
>>> >>>>> >> > <logs.tar.gz>
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>>
>>> >>>>>
>>> >>
>>> _______________________________________________
>>> Devel mailing list -- devel(a)ovirt.org
>>> To unsubscribe send an email to devel-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct: https://www.ovirt.org/communit
>>> y/about/community-guidelines/
>>> List Archives: https://lists.ovirt.org/archiv
>>> es/list/devel(a)ovirt.org/message/WGMUK5T7PDYSIBIUZE3AJIYQAWJOILTC/
>>>
>>
2163
days inactive
2187
days old
11 comments
5 participants
participants (5)
-
Dan Kenigsberg -
Elad Ben Aharon -
Martin Polednik -
Michal Skrivanek -
Nir Soffer