[ovirt-devel] dynamic ownership changes

Wed Apr 25 07:12:29 UTC 2018

Here it is - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
rhv-4.2-ge-runner-tier1/122/

This was executed over iscsi, nfs, gluster and fcp. It ended up with 98.43%
success rate.

2 storage related test cases failed and their failures, from first glance,
don't seem to be bugs.

On Tue, Apr 24, 2018 at 12:37 AM, Elad Ben Aharon <ebenahar at redhat.com>
wrote:

> I will update with the results of the next tier1 execution on latest
> 4.2.3
>
> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik <mpolednik at redhat.com>
> wrote:
>
>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>>
>>> Hi, I've triggered another execution [1] due to some issues I saw in the
>>> first which are not related to the patch.
>>>
>>> The success rate is 78% which is low comparing to tier1 executions with
>>> code from downstream builds (95-100% success rates) [2].
>>>
>>
>> Could you run the current master (without the dynamic_ownership patch)
>> so that we have viable comparision?
>>
>> From what I could see so far, there is an issue with move and copy
>>> operations to and from Gluster domains. For example [3].
>>>
>>> The logs are attached.
>>>
>>>
>>> [1]
>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>
>>>
>>>
>>> [2]
>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>
>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>
>>>
>>>
>>> [3]
>>> 2018-04-22 13:06:28,316+0300 INFO  (jsonrpc/7) [vdsm.api] FINISH
>>> deleteImage error=Image does not exist in domain:
>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> from=:
>>> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
>>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>>> (task:875)
>>> Traceback (most recent call last):
>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
>>> in
>>> _run
>>>   return fn(*args, **kargs)
>>> File "<string>", line 2, in deleteImage
>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
>>> method
>>>   ret = func(*args, **kwargs)
>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503,
>>> in
>>> deleteImage
>>>   raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>> ImageDoesNotExistInSD: Image does not exist in domain:
>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>
>>> 2018-04-22 13:06:28,317+0300 INFO  (jsonrpc/7) [storage.TaskManager.Task]
>>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
>>> "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
>>> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
>>> (task:1181)
>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher]
>>> FINISH
>>> deleteImage error=Image does not exist in domain:
>>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=e5fd29c8-52ba-467e-be09
>>> -ca40ff054d
>>> d4' (dispatcher:82)
>>>
>>>
>>>
>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar at redhat.com>
>>> wrote:
>>>
>>> Triggered a sanity tier1 execution [1] using [2], which covers all the
>>>> requested areas, on iSCSI, NFS and Gluster.
>>>> I'll update with the results.
>>>>
>>>> [1]
>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>
>>>> [2]
>>>> https://gerrit.ovirt.org/#/c/89830/
>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>
>>>>
>>>>
>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik at redhat.com>
>>>> wrote:
>>>>
>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> Hi Martin,
>>>>>>
>>>>>> I see [1] requires a rebase, can you please take care?
>>>>>>
>>>>>>
>>>>> Should be rebased.
>>>>>
>>>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>>>>
>>>>>> FC.
>>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's
>>>>>> not
>>>>>> stable enough at the moment.
>>>>>>
>>>>>>
>>>>> That is still pretty good.
>>>>>
>>>>>
>>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <
>>>>>> mpolednik at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>>>>
>>>>>>>> areas
>>>>>>>> have to be tested here.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'd say that you have quite a bit of freedom in this regard.
>>>>>>> GlusterFS
>>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
>>>>>>> that covers basic operations (start & stop VM, migrate it), snapshots
>>>>>>> and merging them, and whatever else would be important for storage
>>>>>>> sanity.
>>>>>>>
>>>>>>> mpolednik
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>>>> mpolednik at redhat.com
>>>>>>> >
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
>>>>>>>>> cinder,
>>>>>>>>>
>>>>>>>>> will
>>>>>>>>>> have to check, since usually, we don't execute our automation on
>>>>>>>>>> them.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any update on this? I believe the gluster tests were successful,
>>>>>>>>>> OST
>>>>>>>>>>
>>>>>>>>> passes fine and unit tests pass fine, that makes the storage
>>>>>>>>> backends
>>>>>>>>> test the last required piece.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +Elad
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <
>>>>>>>>>>> danken at redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer at redhat.com
>>>>>>>>>>> >
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri at redhat.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please make sure to run as much OST suites on this patch as
>>>>>>>>>>>>> possible
>>>>>>>>>>>>>
>>>>>>>>>>>>> before merging ( using 'ci please build' )
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But note that OST is not a way to verify the patch.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Such changes require testing with all storage types we support.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nir
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>>>>>>>>>> mpolednik at redhat.com
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've created a patch[0] that is finally able to activate
>>>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That of course comes with quite a bit of code removal,
>>>>>>>>>>>>>>> mostly in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> area of host devices, hwrng and anything that touches
>>>>>>>>>>>>>>> devices;
>>>>>>>>>>>>>>> bunch
>>>>>>>>>>>>>>> of test changes and one XML generation caveat (storage is
>>>>>>>>>>>>>>> handled
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> VDSM, therefore disk relabelling needs to be disabled on the
>>>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>>>> level).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Because of the scope of the patch, I welcome
>>>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>>>> people to review the code and consider the implication this
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In particular:  dynamic_ownership was set to 0
>>>>>>>>>>>>>>> prehistorically
>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>>>>>>>>>>> because
>>>>>>>>>>>> libvirt,
>>>>>>>>>>>> running as root, was not able to play properly with root-squash
>>>>>>>>>>>> nfs
>>>>>>>>>>>> mounts.
>>>>>>>>>>>>
>>>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>>>
>>>>>>>>>>>> I join to Nir's request to run this with storage QE.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Raz Tamir
>>>>>>>>>>> Manager, RHV QE
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180425/1f2ec3a4/attachment-0001.html>