9:09 a.m.
Hey,
I've created a patch[0] that is finally able to activate libvirt's
dynamic_ownership for VDSM while not negatively affecting
functionality of our storage code.
That of course comes with quite a bit of code removal, mostly in the
area of host devices, hwrng and anything that touches devices; bunch
of test changes and one XML generation caveat (storage is handled by
VDSM, therefore disk relabelling needs to be disabled on the VDSM
level).
Because of the scope of the patch, I welcome storage/virt/network
people to review the code and consider the implication this change has
on current/future features.
[0] https://gerrit.ovirt.org/#/c/89830/
mpolednik
5:30 a.m.
Please make sure to run as much OST suites on this patch as possible before
merging ( using 'ci please build' )
On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
Hey,
I've created a patch[0] that is finally able to activate libvirt's
dynamic_ownership for VDSM while not negatively affecting
functionality of our storage code.
That of course comes with quite a bit of code removal, mostly in the
area of host devices, hwrng and anything that touches devices; bunch
of test changes and one XML generation caveat (storage is handled by
VDSM, therefore disk relabelling needs to be disabled on the VDSM
level).
Because of the scope of the patch, I welcome storage/virt/network
people to review the code and consider the implication this change has
on current/future features.
[0] https://gerrit.ovirt.org/#/c/89830/
mpolednik
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
5:34 a.m.
On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
Please make sure to run as much OST suites on this patch as possible
before merging ( using 'ci please build' )
But note that OST is not a way to verify the patch.
Such changes require testing with all storage types we support.
Nir
On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
> Hey,
>
> I've created a patch[0] that is finally able to activate libvirt's
> dynamic_ownership for VDSM while not negatively affecting
> functionality of our storage code.
>
> That of course comes with quite a bit of code removal, mostly in the
> area of host devices, hwrng and anything that touches devices; bunch
> of test changes and one XML generation caveat (storage is handled by
> VDSM, therefore disk relabelling needs to be disabled on the VDSM
> level).
>
> Because of the scope of the patch, I welcome storage/virt/network
> people to review the code and consider the implication this change has
> on current/future features.
>
> [0] https://gerrit.ovirt.org/#/c/89830/
>
> mpolednik
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018 <+972%209-769-2018>
irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
5:37 a.m.
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
wrote:
> Please make sure to run as much OST suites on this patch as possible
> before merging ( using 'ci please build' )
>
But note that OST is not a way to verify the patch.
Such changes require testing with all storage types we support.
Well, we already have HE suite that runs on ISCSI, so at least we have
NFS+ISCSI on nested,
for real storage testing, you'll have to do it manually
Nir
On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
>> Hey,
>>
>> I've created a patch[0] that is finally able to activate libvirt's
>> dynamic_ownership for VDSM while not negatively affecting
>> functionality of our storage code.
>>
>> That of course comes with quite a bit of code removal, mostly in the
>> area of host devices, hwrng and anything that touches devices; bunch
>> of test changes and one XML generation caveat (storage is handled by
>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>> level).
>>
>> Because of the scope of the patch, I welcome storage/virt/network
>> people to review the code and consider the implication this change has
>> on current/future features.
>>
>> [0] https://gerrit.ovirt.org/#/c/89830/
>>
>> mpolednik
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
<https://redhat.com/trusted>
> phone: +972-9-7692018 <+972%209-769-2018>
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
8:27 a.m.
On Wed, Apr 11, 2018 at 12:38 PM Eyal Edri <eedri(a)redhat.com> wrote:
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>
>> Please make sure to run as much OST suites on this patch as possible
>> before merging ( using 'ci please build' )
>>
>
> But note that OST is not a way to verify the patch.
>
> Such changes require testing with all storage types we support.
>
Well, we already have HE suite that runs on ISCSI, so at least we have
NFS+ISCSI on nested,
for real storage testing, you'll have to do it manually
We need glusterfs (both native and fuse based), and cinder/ceph storage.
But we cannot practically test all flows with all types of storage for
every patch.
Nir
>
> Nir
>
> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> Hey,
>>>
>>> I've created a patch[0] that is finally able to activate libvirt's
>>> dynamic_ownership for VDSM while not negatively affecting
>>> functionality of our storage code.
>>>
>>> That of course comes with quite a bit of code removal, mostly in the
>>> area of host devices, hwrng and anything that touches devices; bunch
>>> of test changes and one XML generation caveat (storage is handled by
>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>> level).
>>>
>>> Because of the scope of the patch, I welcome storage/virt/network
>>> people to review the code and consider the implication this change has
>>> on current/future features.
>>>
>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>
>>> mpolednik
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV DevOps
>>
>> EMEA VIRTUALIZATION R&D
>>
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>> <https://redhat.com/trusted>
>> phone: +972-9-7692018 <+972%209-769-2018>
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018 <+972%209-769-2018>
irc: eedri (on #tlv #rhev-dev #rhev-integ)
8:30 a.m.
On 11/04/18 12:27 +0000, Nir Soffer wrote:
On Wed, Apr 11, 2018 at 12:38 PM Eyal Edri <eedri(a)redhat.com>
wrote:
> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>
>>> Please make sure to run as much OST suites on this patch as possible
>>> before merging ( using 'ci please build' )
>>>
>>
>> But note that OST is not a way to verify the patch.
>>
>> Such changes require testing with all storage types we support.
>>
>
> Well, we already have HE suite that runs on ISCSI, so at least we have
> NFS+ISCSI on nested,
> for real storage testing, you'll have to do it manually
>
We need glusterfs (both native and fuse based), and cinder/ceph storage.
But we cannot practically test all flows with all types of storage for
every patch.
That leads to a question - how do I go around verifying such patch
without sufficient environment? Is there someone from storage QA that
could assist with this?
Nir
>
>
>>
>> Nir
>>
>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>>> wrote:
>>>
>>>> Hey,
>>>>
>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>> dynamic_ownership for VDSM while not negatively affecting
>>>> functionality of our storage code.
>>>>
>>>> That of course comes with quite a bit of code removal, mostly in the
>>>> area of host devices, hwrng and anything that touches devices; bunch
>>>> of test changes and one XML generation caveat (storage is handled by
>>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>>> level).
>>>>
>>>> Because of the scope of the patch, I welcome storage/virt/network
>>>> people to review the code and consider the implication this change has
>>>> on current/future features.
>>>>
>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>> mpolednik
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA <https://www.redhat.com/>
>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>> <https://redhat.com/trusted>
>>> phone: +972-9-7692018 <+972%209-769-2018>
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
<https://redhat.com/trusted>
> phone: +972-9-7692018 <+972%209-769-2018>
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
8:35 a.m.
On Wed, Apr 11, 2018 at 3:30 PM Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 11/04/18 12:27 +0000, Nir Soffer wrote:
>On Wed, Apr 11, 2018 at 12:38 PM Eyal Edri <eedri(a)redhat.com> wrote:
>
>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com>
wrote:
>>
>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>>> Please make sure to run as much OST suites on this patch as possible
>>>> before merging ( using 'ci please build' )
>>>>
>>>
>>> But note that OST is not a way to verify the patch.
>>>
>>> Such changes require testing with all storage types we support.
>>>
>>
>> Well, we already have HE suite that runs on ISCSI, so at least we have
>> NFS+ISCSI on nested,
>> for real storage testing, you'll have to do it manually
>>
>
>We need glusterfs (both native and fuse based), and cinder/ceph storage.
>
>But we cannot practically test all flows with all types of storage for
>every patch.
That leads to a question - how do I go around verifying such patch
without sufficient environment? Is there someone from storage QA that
could assist with this?
Good question!
I hope Denis can help with verifying the glusterfs changes.
With cinder/ceph, maybe Elad can provide a setup for testing, or run some
automation tests on the patch?
Elad also have other automated tests for NFS/iSCSI that are worth running
before we merge such changes.
Nir
>Nir
>
>
>>
>>
>>>
>>> Nir
>>>
>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com
>
>>>> wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>> functionality of our storage code.
>>>>>
>>>>> That of course comes with quite a bit of code removal, mostly in
the
>>>>> area of host devices, hwrng and anything that touches devices;
bunch
>>>>> of test changes and one XML generation caveat (storage is handled
by
>>>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>>>> level).
>>>>>
>>>>> Because of the scope of the patch, I welcome storage/virt/network
>>>>> people to review the code and consider the implication this change
has
>>>>> on current/future features.
>>>>>
>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>> mpolednik
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Eyal edri
>>>>
>>>>
>>>> MANAGER
>>>>
>>>> RHV DevOps
>>>>
>>>> EMEA VIRTUALIZATION R&D
>>>>
>>>>
>>>> Red Hat EMEA <https://www.redhat.com/>
>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>> <https://redhat.com/trusted>
>>>> phone: +972-9-7692018 <+972%209-769-2018>
<+972%209-769-2018>
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>>
>>
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV DevOps
>>
>> EMEA VIRTUALIZATION R&D
>>
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <
https://redhat.com/trusted>
>> phone: +972-9-7692018 <+972%209-769-2018> <+972%209-769-2018>
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
9:12 a.m.
On Wed, Apr 11, 2018 at 3:27 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Apr 11, 2018 at 12:38 PM Eyal Edri <eedri(a)redhat.com>
wrote:
> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>
>>> Please make sure to run as much OST suites on this patch as possible
>>> before merging ( using 'ci please build' )
>>>
>>
>> But note that OST is not a way to verify the patch.
>>
>> Such changes require testing with all storage types we support.
>>
>
> Well, we already have HE suite that runs on ISCSI, so at least we have
> NFS+ISCSI on nested,
> for real storage testing, you'll have to do it manually
>
We need glusterfs (both native and fuse based), and cinder/ceph storage.
We have Gluster in o-s-t as well, as part of the HC suite. It doesn't use
Fuse though.
But we cannot practically test all flows with all types of storage for
every patch.
Indeed. But we could add easily do some, and we should at least execute the
minimal set that we are able to easily via o-s-t.
Y.
Nir
>
>
>>
>> Nir
>>
>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>>> wrote:
>>>
>>>> Hey,
>>>>
>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>> dynamic_ownership for VDSM while not negatively affecting
>>>> functionality of our storage code.
>>>>
>>>> That of course comes with quite a bit of code removal, mostly in the
>>>> area of host devices, hwrng and anything that touches devices; bunch
>>>> of test changes and one XML generation caveat (storage is handled by
>>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>>> level).
>>>>
>>>> Because of the scope of the patch, I welcome storage/virt/network
>>>> people to review the code and consider the implication this change has
>>>> on current/future features.
>>>>
>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>> mpolednik
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA <https://www.redhat.com/>
>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>> <https://redhat.com/trusted>
>>> phone: +972-9-7692018 <+972%209-769-2018>
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
<https://redhat.com/trusted>
> phone: +972-9-7692018 <+972%209-769-2018>
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
9:28 a.m.
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
wrote:
> Please make sure to run as much OST suites on this patch as possible
> before merging ( using 'ci please build' )
>
But note that OST is not a way to verify the patch.
Such changes require testing with all storage types we support.
Nir
On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
>> Hey,
>>
>> I've created a patch[0] that is finally able to activate libvirt's
>> dynamic_ownership for VDSM while not negatively affecting
>> functionality of our storage code.
>>
>> That of course comes with quite a bit of code removal, mostly in the
>> area of host devices, hwrng and anything that touches devices; bunch
>> of test changes and one XML generation caveat (storage is handled by
>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>> level).
>>
>> Because of the scope of the patch, I welcome storage/virt/network
>> people to review the code and consider the implication this change has
>> on current/future features.
>>
>> [0] https://gerrit.ovirt.org/#/c/89830/
>>
>
In particular: dynamic_ownership was set to 0 prehistorically (as part of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because libvirt,
running as root, was not able to play properly with root-squash nfs mounts.
Have you attempted this use case?
I join to Nir's request to run this with storage QE.
9:38 a.m.
+Elad
On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>
>> Please make sure to run as much OST suites on this patch as possible
>> before merging ( using 'ci please build' )
>>
>
> But note that OST is not a way to verify the patch.
>
> Such changes require testing with all storage types we support.
>
> Nir
>
> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> Hey,
>>>
>>> I've created a patch[0] that is finally able to activate libvirt's
>>> dynamic_ownership for VDSM while not negatively affecting
>>> functionality of our storage code.
>>>
>>> That of course comes with quite a bit of code removal, mostly in the
>>> area of host devices, hwrng and anything that touches devices; bunch
>>> of test changes and one XML generation caveat (storage is handled by
>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>> level).
>>>
>>> Because of the scope of the patch, I welcome storage/virt/network
>>> people to review the code and consider the implication this change has
>>> on current/future features.
>>>
>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>
>>
In particular: dynamic_ownership was set to 0 prehistorically (as part of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because libvirt,
running as root, was not able to play properly with root-squash nfs mounts.
Have you attempted this use case?
I join to Nir's request to run this with storage QE.
--
Raz Tamir
Manager, RHV QE
9:52 a.m.
We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder, will
have to check, since usually, we don't execute our automation on them.
On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com> wrote:
+Elad
On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>
>>> Please make sure to run as much OST suites on this patch as possible
>>> before merging ( using 'ci please build' )
>>>
>>
>> But note that OST is not a way to verify the patch.
>>
>> Such changes require testing with all storage types we support.
>>
>> Nir
>>
>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>>> wrote:
>>>
>>>> Hey,
>>>>
>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>> dynamic_ownership for VDSM while not negatively affecting
>>>> functionality of our storage code.
>>>>
>>>> That of course comes with quite a bit of code removal, mostly in the
>>>> area of host devices, hwrng and anything that touches devices; bunch
>>>> of test changes and one XML generation caveat (storage is handled by
>>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>>> level).
>>>>
>>>> Because of the scope of the patch, I welcome storage/virt/network
>>>> people to review the code and consider the implication this change has
>>>> on current/future features.
>>>>
>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>
> In particular: dynamic_ownership was set to 0 prehistorically (as part
> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because libvirt,
> running as root, was not able to play properly with root-squash nfs mounts.
>
> Have you attempted this use case?
>
> I join to Nir's request to run this with storage QE.
>
--
Raz Tamir
Manager, RHV QE
4:16 a.m.
On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder,
will
have to check, since usually, we don't execute our automation on them.
Any update on this? I believe the gluster tests were successful, OST
passes fine and unit tests pass fine, that makes the storage backends
test the last required piece.
On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com>
wrote:
> +Elad
>
> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
>
>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>>> Please make sure to run as much OST suites on this patch as possible
>>>> before merging ( using 'ci please build' )
>>>>
>>>
>>> But note that OST is not a way to verify the patch.
>>>
>>> Such changes require testing with all storage types we support.
>>>
>>> Nir
>>>
>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>>> wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>> functionality of our storage code.
>>>>>
>>>>> That of course comes with quite a bit of code removal, mostly in the
>>>>> area of host devices, hwrng and anything that touches devices; bunch
>>>>> of test changes and one XML generation caveat (storage is handled by
>>>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>>>> level).
>>>>>
>>>>> Because of the scope of the patch, I welcome storage/virt/network
>>>>> people to review the code and consider the implication this change
has
>>>>> on current/future features.
>>>>>
>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>
>>>>
>> In particular: dynamic_ownership was set to 0 prehistorically (as part
>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because libvirt,
>> running as root, was not able to play properly with root-squash nfs mounts.
>>
>> Have you attempted this use case?
>>
>> I join to Nir's request to run this with storage QE.
>>
>
>
>
> --
>
>
> Raz Tamir
> Manager, RHV QE
>
4:37 a.m.
Hi, sorry if I misunderstood, I waited for more input regarding what areas
have to be tested here.
On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
> We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder, will
> have to check, since usually, we don't execute our automation on them.
>
Any update on this? I believe the gluster tests were successful, OST
passes fine and unit tests pass fine, that makes the storage backends
test the last required piece.
On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com> wrote:
>
> +Elad
>>
>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>>
>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>>
>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>>
>>>> Please make sure to run as much OST suites on this patch as possible
>>>>> before merging ( using 'ci please build' )
>>>>>
>>>>>
>>>> But note that OST is not a way to verify the patch.
>>>>
>>>> Such changes require testing with all storage types we support.
>>>>
>>>> Nir
>>>>
>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>> >
>>>>
>>>>> wrote:
>>>>>
>>>>> Hey,
>>>>>>
>>>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>>> functionality of our storage code.
>>>>>>
>>>>>> That of course comes with quite a bit of code removal, mostly in
the
>>>>>> area of host devices, hwrng and anything that touches devices;
bunch
>>>>>> of test changes and one XML generation caveat (storage is handled
by
>>>>>> VDSM, therefore disk relabelling needs to be disabled on the
VDSM
>>>>>> level).
>>>>>>
>>>>>> Because of the scope of the patch, I welcome
storage/virt/network
>>>>>> people to review the code and consider the implication this
change
>>>>>> has
>>>>>> on current/future features.
>>>>>>
>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>
>>>>>>
>>>>> In particular: dynamic_ownership was set to 0 prehistorically (as
>>> part
>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>> libvirt,
>>> running as root, was not able to play properly with root-squash nfs
>>> mounts.
>>>
>>> Have you attempted this use case?
>>>
>>> I join to Nir's request to run this with storage QE.
>>>
>>>
>>
>>
>> --
>>
>>
>> Raz Tamir
>> Manager, RHV QE
>>
>>
7:17 a.m.
On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
Hi, sorry if I misunderstood, I waited for more input regarding what
areas
have to be tested here.
I'd say that you have quite a bit of freedom in this regard. GlusterFS
should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
that covers basic operations (start & stop VM, migrate it), snapshots
and merging them, and whatever else would be important for storage
sanity.
mpolednik
On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik
<mpolednik(a)redhat.com>
wrote:
> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>
>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder, will
>> have to check, since usually, we don't execute our automation on them.
>>
>
> Any update on this? I believe the gluster tests were successful, OST
> passes fine and unit tests pass fine, that makes the storage backends
> test the last required piece.
>
>
> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com> wrote:
>>
>> +Elad
>>>
>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <danken(a)redhat.com>
>>> wrote:
>>>
>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com>
wrote:
>>>>
>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
wrote:
>>>>>
>>>>> Please make sure to run as much OST suites on this patch as possible
>>>>>> before merging ( using 'ci please build' )
>>>>>>
>>>>>>
>>>>> But note that OST is not a way to verify the patch.
>>>>>
>>>>> Such changes require testing with all storage types we support.
>>>>>
>>>>> Nir
>>>>>
>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Hey,
>>>>>>>
>>>>>>> I've created a patch[0] that is finally able to activate
libvirt's
>>>>>>> dynamic_ownership for VDSM while not negatively affecting
>>>>>>> functionality of our storage code.
>>>>>>>
>>>>>>> That of course comes with quite a bit of code removal, mostly
in the
>>>>>>> area of host devices, hwrng and anything that touches
devices; bunch
>>>>>>> of test changes and one XML generation caveat (storage is
handled by
>>>>>>> VDSM, therefore disk relabelling needs to be disabled on the
VDSM
>>>>>>> level).
>>>>>>>
>>>>>>> Because of the scope of the patch, I welcome
storage/virt/network
>>>>>>> people to review the code and consider the implication this
change
>>>>>>> has
>>>>>>> on current/future features.
>>>>>>>
>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>
>>>>>>>
>>>>>> In particular: dynamic_ownership was set to 0 prehistorically
(as
>>>> part
>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>> libvirt,
>>>> running as root, was not able to play properly with root-squash nfs
>>>> mounts.
>>>>
>>>> Have you attempted this use case?
>>>>
>>>> I join to Nir's request to run this with storage QE.
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>> Raz Tamir
>>> Manager, RHV QE
>>>
>>>
7:54 a.m.
Hi Martin,
I see [1] requires a rebase, can you please take care?
At the moment, our automation is stable only on iSCSI, NFS, Gluster and FC.
Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's not
stable enough at the moment.
[1] https://gerrit.ovirt.org/#/c/89830/
Thanks
On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
> Hi, sorry if I misunderstood, I waited for more input regarding what areas
> have to be tested here.
>
I'd say that you have quite a bit of freedom in this regard. GlusterFS
should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
that covers basic operations (start & stop VM, migrate it), snapshots
and merging them, and whatever else would be important for storage
sanity.
mpolednik
On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>
>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder,
>>> will
>>> have to check, since usually, we don't execute our automation on them.
>>>
>>>
>> Any update on this? I believe the gluster tests were successful, OST
>> passes fine and unit tests pass fine, that makes the storage backends
>> test the last required piece.
>>
>>
>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com> wrote:
>>
>>>
>>> +Elad
>>>
>>>>
>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>> wrote:
>>>>
>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <nsoffer(a)redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com>
wrote:
>>>>>
>>>>>>
>>>>>> Please make sure to run as much OST suites on this patch as
possible
>>>>>>
>>>>>>> before merging ( using 'ci please build' )
>>>>>>>
>>>>>>>
>>>>>>> But note that OST is not a way to verify the patch.
>>>>>>
>>>>>> Such changes require testing with all storage types we support.
>>>>>>
>>>>>> Nir
>>>>>>
>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>>> mpolednik(a)redhat.com
>>>>>> >
>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hey,
>>>>>>>
>>>>>>>>
>>>>>>>> I've created a patch[0] that is finally able to
activate libvirt's
>>>>>>>> dynamic_ownership for VDSM while not negatively
affecting
>>>>>>>> functionality of our storage code.
>>>>>>>>
>>>>>>>> That of course comes with quite a bit of code removal,
mostly in
>>>>>>>> the
>>>>>>>> area of host devices, hwrng and anything that touches
devices;
>>>>>>>> bunch
>>>>>>>> of test changes and one XML generation caveat (storage is
handled
>>>>>>>> by
>>>>>>>> VDSM, therefore disk relabelling needs to be disabled on
the VDSM
>>>>>>>> level).
>>>>>>>>
>>>>>>>> Because of the scope of the patch, I welcome
storage/virt/network
>>>>>>>> people to review the code and consider the implication
this change
>>>>>>>> has
>>>>>>>> on current/future features.
>>>>>>>>
>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>
>>>>>>>>
>>>>>>>> In particular: dynamic_ownership was set to 0
prehistorically (as
>>>>>>>
>>>>>> part
>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>> libvirt,
>>>>> running as root, was not able to play properly with root-squash nfs
>>>>> mounts.
>>>>>
>>>>> Have you attempted this use case?
>>>>>
>>>>> I join to Nir's request to run this with storage QE.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Raz Tamir
>>>> Manager, RHV QE
>>>>
>>>>
>>>>
8:07 a.m.
On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
Hi Martin,
I see [1] requires a rebase, can you please take care?
Should be rebased.
At the moment, our automation is stable only on iSCSI, NFS, Gluster
and FC.
Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's not
stable enough at the moment.
That is still pretty good.
[1] https://gerrit.ovirt.org/#/c/89830/
Thanks
On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>
>> Hi, sorry if I misunderstood, I waited for more input regarding what areas
>> have to be tested here.
>>
>
> I'd say that you have quite a bit of freedom in this regard. GlusterFS
> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
> that covers basic operations (start & stop VM, migrate it), snapshots
> and merging them, and whatever else would be important for storage
> sanity.
>
> mpolednik
>
>
> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>
>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder,
>>>> will
>>>> have to check, since usually, we don't execute our automation on
them.
>>>>
>>>>
>>> Any update on this? I believe the gluster tests were successful, OST
>>> passes fine and unit tests pass fine, that makes the storage backends
>>> test the last required piece.
>>>
>>>
>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com> wrote:
>>>
>>>>
>>>> +Elad
>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> Please make sure to run as much OST suites on this patch as
possible
>>>>>>>
>>>>>>>> before merging ( using 'ci please build' )
>>>>>>>>
>>>>>>>>
>>>>>>>> But note that OST is not a way to verify the patch.
>>>>>>>
>>>>>>> Such changes require testing with all storage types we
support.
>>>>>>>
>>>>>>> Nir
>>>>>>>
>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>>>> mpolednik(a)redhat.com
>>>>>>> >
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hey,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've created a patch[0] that is finally able to
activate libvirt's
>>>>>>>>> dynamic_ownership for VDSM while not negatively
affecting
>>>>>>>>> functionality of our storage code.
>>>>>>>>>
>>>>>>>>> That of course comes with quite a bit of code
removal, mostly in
>>>>>>>>> the
>>>>>>>>> area of host devices, hwrng and anything that touches
devices;
>>>>>>>>> bunch
>>>>>>>>> of test changes and one XML generation caveat
(storage is handled
>>>>>>>>> by
>>>>>>>>> VDSM, therefore disk relabelling needs to be disabled
on the VDSM
>>>>>>>>> level).
>>>>>>>>>
>>>>>>>>> Because of the scope of the patch, I welcome
storage/virt/network
>>>>>>>>> people to review the code and consider the
implication this change
>>>>>>>>> has
>>>>>>>>> on current/future features.
>>>>>>>>>
>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In particular: dynamic_ownership was set to 0
prehistorically (as
>>>>>>>>
>>>>>>> part
>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>> libvirt,
>>>>>> running as root, was not able to play properly with root-squash
nfs
>>>>>> mounts.
>>>>>>
>>>>>> Have you attempted this use case?
>>>>>>
>>>>>> I join to Nir's request to run this with storage QE.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> Raz Tamir
>>>>> Manager, RHV QE
>>>>>
>>>>>
>>>>>
10:34 a.m.
Triggered a sanity tier1 execution [1] using [2], which covers all the
requested areas, on iSCSI, NFS and Gluster.
I'll update with the results.
[1]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.
2_dev/job/rhv-4.2-ge-flow-storage/1161/
[2]
https://gerrit.ovirt.org/#/c/89830/
vdsm-4.30.0-291.git77aef9a.el7.x86_64
On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
> Hi Martin,
>
> I see [1] requires a rebase, can you please take care?
>
Should be rebased.
At the moment, our automation is stable only on iSCSI, NFS, Gluster and FC.
> Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's not
> stable enough at the moment.
>
That is still pretty good.
[1] https://gerrit.ovirt.org/#/c/89830/
>
>
> Thanks
>
> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>
>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>> areas
>>> have to be tested here.
>>>
>>>
>> I'd say that you have quite a bit of freedom in this regard. GlusterFS
>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
>> that covers basic operations (start & stop VM, migrate it), snapshots
>> and merging them, and whatever else would be important for storage
>> sanity.
>>
>> mpolednik
>>
>>
>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <mpolednik(a)redhat.com>
>>
>>> wrote:
>>>
>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>
>>>>
>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and cinder,
>>>>
>>>>> will
>>>>> have to check, since usually, we don't execute our automation on
them.
>>>>>
>>>>>
>>>>> Any update on this? I believe the gluster tests were successful, OST
>>>> passes fine and unit tests pass fine, that makes the storage backends
>>>> test the last required piece.
>>>>
>>>>
>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir <ratamir(a)redhat.com>
wrote:
>>>>
>>>>
>>>>> +Elad
>>>>>
>>>>>
>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Please make sure to run as much OST suites on this patch
as
>>>>>>>> possible
>>>>>>>>
>>>>>>>> before merging ( using 'ci please build' )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But note that OST is not a way to verify the patch.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Such changes require testing with all storage types we
support.
>>>>>>>>
>>>>>>>> Nir
>>>>>>>>
>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <
>>>>>>>> mpolednik(a)redhat.com
>>>>>>>> >
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hey,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I've created a patch[0] that is finally able
to activate
>>>>>>>>>> libvirt's
>>>>>>>>>> dynamic_ownership for VDSM while not negatively
affecting
>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>
>>>>>>>>>> That of course comes with quite a bit of code
removal, mostly in
>>>>>>>>>> the
>>>>>>>>>> area of host devices, hwrng and anything that
touches devices;
>>>>>>>>>> bunch
>>>>>>>>>> of test changes and one XML generation caveat
(storage is handled
>>>>>>>>>> by
>>>>>>>>>> VDSM, therefore disk relabelling needs to be
disabled on the VDSM
>>>>>>>>>> level).
>>>>>>>>>>
>>>>>>>>>> Because of the scope of the patch, I welcome
storage/virt/network
>>>>>>>>>> people to review the code and consider the
implication this
>>>>>>>>>> change
>>>>>>>>>> has
>>>>>>>>>> on current/future features.
>>>>>>>>>>
>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In particular: dynamic_ownership was set to 0
prehistorically
>>>>>>>>>> (as
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> part
>>>>>>>>
>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
because
>>>>>>> libvirt,
>>>>>>> running as root, was not able to play properly with
root-squash nfs
>>>>>>> mounts.
>>>>>>>
>>>>>>> Have you attempted this use case?
>>>>>>>
>>>>>>> I join to Nir's request to run this with storage QE.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> Raz Tamir
>>>>>> Manager, RHV QE
>>>>>>
>>>>>>
>>>>>>
>>>>>>
6:23 p.m.
Hi, I've triggered another execution [1] due to some issues I saw in the
first which are not related to the patch.
The success rate is 78% which is low comparing to tier1 executions with
code from downstream builds (95-100% success rates) [2].
From what I could see so far, there is an issue with move and copy
operations to and from Gluster domains. For example [3].
The logs are attached.
[1]
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/testReport/
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
[2]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
rhv-4.2-ge-runner-tier1-after-upgrade/7/
[3]
2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
from=:
:ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in deleteImage
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503, in
deleteImage
raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
ImageDoesNotExistInSD: Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
"Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
(task:1181)
2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33, domain=e5fd29c8-52ba-467e-be09
-ca40ff054d
d4' (dispatcher:82)
On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
Triggered a sanity tier1 execution [1] using [2], which covers all
the
requested areas, on iSCSI, NFS and Gluster.
I'll update with the results.
[1]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
_dev/job/rhv-4.2-ge-flow-storage/1161/
[2]
https://gerrit.ovirt.org/#/c/89830/
vdsm-4.30.0-291.git77aef9a.el7.x86_64
On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>
>> Hi Martin,
>>
>> I see [1] requires a rebase, can you please take care?
>>
>
> Should be rebased.
>
> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>> FC.
>> Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's not
>> stable enough at the moment.
>>
>
> That is still pretty good.
>
>
> [1] https://gerrit.ovirt.org/#/c/89830/
>>
>>
>> Thanks
>>
>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>
>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>> areas
>>>> have to be tested here.
>>>>
>>>>
>>> I'd say that you have quite a bit of freedom in this regard. GlusterFS
>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
>>> that covers basic operations (start & stop VM, migrate it), snapshots
>>> and merging them, and whatever else would be important for storage
>>> sanity.
>>>
>>> mpolednik
>>>
>>>
>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <mpolednik(a)redhat.com
>>> >
>>>
>>>> wrote:
>>>>
>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>
>>>>>
>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
cinder,
>>>>>
>>>>>> will
>>>>>> have to check, since usually, we don't execute our automation
on
>>>>>> them.
>>>>>>
>>>>>>
>>>>>> Any update on this? I believe the gluster tests were successful,
OST
>>>>> passes fine and unit tests pass fine, that makes the storage
backends
>>>>> test the last required piece.
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>> +Elad
>>>>>>
>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Please make sure to run as much OST suites on this
patch as
>>>>>>>>> possible
>>>>>>>>>
>>>>>>>>> before merging ( using 'ci please build' )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But note that OST is not a way to verify the
patch.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Such changes require testing with all storage types
we support.
>>>>>>>>>
>>>>>>>>> Nir
>>>>>>>>>
>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<
>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hey,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I've created a patch[0] that is finally
able to activate
>>>>>>>>>>> libvirt's
>>>>>>>>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>
>>>>>>>>>>> That of course comes with quite a bit of code
removal, mostly in
>>>>>>>>>>> the
>>>>>>>>>>> area of host devices, hwrng and anything that
touches devices;
>>>>>>>>>>> bunch
>>>>>>>>>>> of test changes and one XML generation caveat
(storage is
>>>>>>>>>>> handled
>>>>>>>>>>> by
>>>>>>>>>>> VDSM, therefore disk relabelling needs to be
disabled on the
>>>>>>>>>>> VDSM
>>>>>>>>>>> level).
>>>>>>>>>>>
>>>>>>>>>>> Because of the scope of the patch, I welcome
>>>>>>>>>>> storage/virt/network
>>>>>>>>>>> people to review the code and consider the
implication this
>>>>>>>>>>> change
>>>>>>>>>>> has
>>>>>>>>>>> on current/future features.
>>>>>>>>>>>
>>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In particular: dynamic_ownership was set to
0 prehistorically
>>>>>>>>>>> (as
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> part
>>>>>>>>>
>>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
because
>>>>>>>> libvirt,
>>>>>>>> running as root, was not able to play properly with
root-squash nfs
>>>>>>>> mounts.
>>>>>>>>
>>>>>>>> Have you attempted this use case?
>>>>>>>>
>>>>>>>> I join to Nir's request to run this with storage QE.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> Raz Tamir
>>>>>>> Manager, RHV QE
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
6:29 p.m.
Sorry, this is the new execution link:
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
On Mon, Apr 23, 2018 at 1:23 AM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
Hi, I've triggered another execution [1] due to some issues I saw
in the
first which are not related to the patch.
The success rate is 78% which is low comparing to tier1 executions with
code from downstream builds (95-100% success rates) [2].
From what I could see so far, there is an issue with move and copy
operations to and from Gluster domains. For example [3].
The logs are attached.
[1]
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/testReport/
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
[2]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-
4.2-ge-runner-tier1-after-upgrade/7/
[3]
2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4' from=:
:ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
in _run
return fn(*args, **kargs)
File "<string>", line 2, in deleteImage
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503,
in deleteImage
raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
ImageDoesNotExistInSD: Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
"Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
(task:1181)
2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054d
d4' (dispatcher:82)
On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
> Triggered a sanity tier1 execution [1] using [2], which covers all the
> requested areas, on iSCSI, NFS and Gluster.
> I'll update with the results.
>
> [1]
> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
> _dev/job/rhv-4.2-ge-flow-storage/1161/
>
> [2]
> https://gerrit.ovirt.org/#/c/89830/
> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>
>
>
> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>
>>> Hi Martin,
>>>
>>> I see [1] requires a rebase, can you please take care?
>>>
>>
>> Should be rebased.
>>
>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>> FC.
>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's
>>> not
>>> stable enough at the moment.
>>>
>>
>> That is still pretty good.
>>
>>
>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>
>>>
>>> Thanks
>>>
>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>> wrote:
>>>
>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>
>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>> areas
>>>>> have to be tested here.
>>>>>
>>>>>
>>>> I'd say that you have quite a bit of freedom in this regard.
GlusterFS
>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>> and merging them, and whatever else would be important for storage
>>>> sanity.
>>>>
>>>> mpolednik
>>>>
>>>>
>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>> mpolednik(a)redhat.com>
>>>>
>>>>> wrote:
>>>>>
>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>>
>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
cinder,
>>>>>>
>>>>>>> will
>>>>>>> have to check, since usually, we don't execute our
automation on
>>>>>>> them.
>>>>>>>
>>>>>>>
>>>>>>> Any update on this? I believe the gluster tests were
successful, OST
>>>>>> passes fine and unit tests pass fine, that makes the storage
backends
>>>>>> test the last required piece.
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>> +Elad
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>> possible
>>>>>>>>>>
>>>>>>>>>> before merging ( using 'ci please build'
)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But note that OST is not a way to verify the
patch.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Such changes require testing with all storage
types we support.
>>>>>>>>>>
>>>>>>>>>> Nir
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<
>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>> libvirt's
>>>>>>>>>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>>
>>>>>>>>>>>> That of course comes with quite a bit of
code removal, mostly
>>>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>> area of host devices, hwrng and anything
that touches devices;
>>>>>>>>>>>> bunch
>>>>>>>>>>>> of test changes and one XML generation
caveat (storage is
>>>>>>>>>>>> handled
>>>>>>>>>>>> by
>>>>>>>>>>>> VDSM, therefore disk relabelling needs to
be disabled on the
>>>>>>>>>>>> VDSM
>>>>>>>>>>>> level).
>>>>>>>>>>>>
>>>>>>>>>>>> Because of the scope of the patch, I
welcome
>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>> people to review the code and consider
the implication this
>>>>>>>>>>>> change
>>>>>>>>>>>> has
>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>
>>>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In particular: dynamic_ownership was set
to 0 prehistorically
>>>>>>>>>>>> (as
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> part
>>>>>>>>>>
>>>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961
) because
>>>>>>>>> libvirt,
>>>>>>>>> running as root, was not able to play properly with
root-squash
>>>>>>>>> nfs
>>>>>>>>> mounts.
>>>>>>>>>
>>>>>>>>> Have you attempted this use case?
>>>>>>>>>
>>>>>>>>> I join to Nir's request to run this with storage
QE.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> Raz Tamir
>>>>>>>> Manager, RHV QE
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
6:34 p.m.
Also, snapshot preview failed (2nd snapshot):
2018-04-22 18:01:06,253+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC
call Volume.create succeeded in 0.84 seconds (__init__:311)
2018-04-22 18:01:06,261+0300 INFO (tasks/6)
[storage.ThreadPool.WorkerThread] START task
6823d724-cb1b-4706-a58a-83428363cce5 (cmd=<bound method Task.commit of
<vdsm.storage.task.Task instance at 0x7f1aac54fc68>>, args=None)
(threadPool
:208)
2018-04-22 18:01:06,906+0300 WARN (check/loop) [storage.asyncutils] Call
<bound method DirectioChecker._check of <DirectioChecker
/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Storage__NFS_storage__l
ocal__ge2__nfs__0/46d2fd2b-bdd0-40f5-be4c-0aaf2a629f1b/dom_md/metadata
running next_check=4920812.91 at 0x7f1aac3ed790>> delayed by 0.51 seconds
(asyncutils:138)
2018-04-22 18:01:07,082+0300 WARN (tasks/6) [storage.ResourceManager]
Resource factory failed to create resource
'01_img_7df9d2b2-52b5-4ac2-a9f0-a1d1e93eb6d2.095ad9d6-3154-449c-868c-f975dcdcb729'.
Canceling request. (resourceManager:543
)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py",
line 539, in registerResource
obj = namespaceObj.factory.createResource(name, lockType)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py",
line 193, in createResource
lockType)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py",
line 122, in __getResourceCandidatesList
imgUUID=resourceName)
File "/usr/lib/python2.7/site-packages/vdsm/storage/image.py", line 198,
in getChain
uuidlist = volclass.getImageVolumes(sdUUID, imgUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1537,
in getImageVolumes
return cls.manifestClass.getImageVolumes(sdUUID, imgUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line
337, in getImageVolumes
if (sd.produceVolume(imgUUID, volid).getImage() == imgUUID):
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 438, in
produceVolume
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line
69, in __init__
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86,
in __init__
self.validate()
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112,
in validate
self.validateVolumePath()
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line
129, in validateVolumePath
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'a404bfc9-57ef-4dcc-9f1b-458dfb08ad74',)
2018-04-22 18:01:07,083+0300 WARN (tasks/6)
[storage.ResourceManager.Request]
(ResName='01_img_7df9d2b2-52b5-4ac2-a9f0-a1d1e93eb6d2.095ad9d6-3154-449c-868c-f975dcdcb729',
ReqID='79c96e70-7334-4402-a390-dc87f939b7d2') Tried to cancel a p
rocessed request (resourceManager:187)
2018-04-22 18:01:07,084+0300 ERROR (tasks/6) [storage.TaskManager.Task]
(Task='6823d724-cb1b-4706-a58a-83428363cce5') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in
run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
79, in wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1939, in
createVolume
with rm.acquireResource(img_ns, imgUUID, rm.EXCLUSIVE):
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py",
line 1025, in acquireResource
return _manager.acquireResource(namespace, name, lockType,
timeout=timeout)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py",
line 475, in acquireResource
raise se.ResourceAcqusitionFailed()
ResourceAcqusitionFailed: Could not acquire resource. Probably resource
factory threw an exception.: ()
2018-04-22 18:01:07,735+0300 INFO (tasks/6)
[storage.ThreadPool.WorkerThread] FINISH task
6823d724-cb1b-4706-a58a-83428363cce5 (threadPool:210)
*Steps from [1]:*
*17:54:41* 2018-04-22 17:54:41,574 INFO Test Setup 2: Creating
VM vm_TestCase11660_2217544157*17:54:55* 2018-04-22 17:54:55,593 INFO
049:
storage/rhevmtests.storage.storage_snapshots.test_live_snapshot.TestCase11660.test_live_snapshot[glusterfs]*17:54:55*
2018-04-22 17:54:55,593 INFO Create a snapshot while VM is
running*17:54:55* 2018-04-22 17:54:55,593 INFO STORAGE:
GLUSTERFS*17:58:04* 2018-04-22 17:58:04,761 INFO Test Step 3:
Start writing continuously on VM vm_TestCase11660_2217544157 via
dd*17:58:35* 2018-04-22 17:58:35,334 INFO Test Step 4:
Creating live snapshot on a VM vm_TestCase11660_2217544157*17:58:35*
2018-04-22 17:58:35,334 INFO Test Step 5: Adding new snapshot
to VM vm_TestCase11660_2217544157 with all disks*17:58:35* 2018-04-22
17:58:35,337 INFO Test Step 6: Add snapshot to VM
vm_TestCase11660_2217544157 with {'description':
'snap_TestCase11660_2217545559', 'wait': True}*17:59:26* 2018-04-22
17:59:26,179 INFO Test Step 7: Writing files to VM's
vm_TestCase11660_2217544157 disk*18:00:33* 2018-04-22 18:00:33,117
INFO Test Step 8: Shutdown vm vm_TestCase11660_2217544157 with
{'async': 'false'}*18:01:04* 2018-04-22 18:01:04,038 INFO Test
Step 9: Previewing snapshot snap_TestCase11660_2217545559 on VM
vm_TestCase11660_2217544157
[1]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
On Mon, Apr 23, 2018 at 1:29 AM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
Sorry, this is the new execution link:
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
rhv-4.2-ge-runner-storage/1048/testReport/
On Mon, Apr 23, 2018 at 1:23 AM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
> Hi, I've triggered another execution [1] due to some issues I saw in the
> first which are not related to the patch.
>
> The success rate is 78% which is low comparing to tier1 executions with
> code from downstream builds (95-100% success rates) [2].
>
> From what I could see so far, there is an issue with move and copy
> operations to and from Gluster domains. For example [3].
>
> The logs are attached.
>
>
> [1]
>
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/testReport/
>
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
>
>
>
> [2]
> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-
> 4.2-ge-runner-tier1-after-upgrade/7/
>
>
>
> [3]
> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
> deleteImage error=Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4' from=:
> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
> (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
> in _run
> return fn(*args, **kargs)
> File "<string>", line 2, in deleteImage
> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
> method
> ret = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503,
> in deleteImage
> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
> ImageDoesNotExistInSD: Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
> "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
> (task:1181)
> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher]
> FINISH deleteImage error=Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09-ca40ff054d
> d4' (dispatcher:82)
>
>
>
> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
> wrote:
>
>> Triggered a sanity tier1 execution [1] using [2], which covers all the
>> requested areas, on iSCSI, NFS and Gluster.
>> I'll update with the results.
>>
>> [1]
>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>
>> [2]
>> https://gerrit.ovirt.org/#/c/89830/
>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>
>>
>>
>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> I see [1] requires a rebase, can you please take care?
>>>>
>>>
>>> Should be rebased.
>>>
>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>>> FC.
>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>> not
>>>> stable enough at the moment.
>>>>
>>>
>>> That is still pretty good.
>>>
>>>
>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>> >
>>>> wrote:
>>>>
>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>>> areas
>>>>>> have to be tested here.
>>>>>>
>>>>>>
>>>>> I'd say that you have quite a bit of freedom in this regard.
GlusterFS
>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>> and merging them, and whatever else would be important for storage
>>>>> sanity.
>>>>>
>>>>> mpolednik
>>>>>
>>>>>
>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>> mpolednik(a)redhat.com>
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>
>>>>>>>
>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>>>>>>> cinder,
>>>>>>>
>>>>>>>> will
>>>>>>>> have to check, since usually, we don't execute our
automation on
>>>>>>>> them.
>>>>>>>>
>>>>>>>>
>>>>>>>> Any update on this? I believe the gluster tests were
successful,
>>>>>>> OST
>>>>>>> passes fine and unit tests pass fine, that makes the storage
>>>>>>> backends
>>>>>>> test the last required piece.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> +Elad
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <
>>>>>>>>> danken(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>>> possible
>>>>>>>>>>>
>>>>>>>>>>> before merging ( using 'ci please
build' )
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But note that OST is not a way to verify
the patch.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Such changes require testing with all storage
types we support.
>>>>>>>>>>>
>>>>>>>>>>> Nir
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hey,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That of course comes with quite a bit
of code removal, mostly
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> area of host devices, hwrng and
anything that touches devices;
>>>>>>>>>>>>> bunch
>>>>>>>>>>>>> of test changes and one XML
generation caveat (storage is
>>>>>>>>>>>>> handled
>>>>>>>>>>>>> by
>>>>>>>>>>>>> VDSM, therefore disk relabelling
needs to be disabled on the
>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>> level).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Because of the scope of the patch, I
welcome
>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>> people to review the code and
consider the implication this
>>>>>>>>>>>>> change
>>>>>>>>>>>>> has
>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In particular: dynamic_ownership was
set to 0
>>>>>>>>>>>>> prehistorically (as
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> part
>>>>>>>>>>>
>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>>>>>> libvirt,
>>>>>>>>>> running as root, was not able to play properly
with root-squash
>>>>>>>>>> nfs
>>>>>>>>>> mounts.
>>>>>>>>>>
>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>
>>>>>>>>>> I join to Nir's request to run this with
storage QE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raz Tamir
>>>>>>>>> Manager, RHV QE
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>
>
8:56 a.m.
On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
Hi, I've triggered another execution [1] due to some issues I saw
in the
first which are not related to the patch.
The success rate is 78% which is low comparing to tier1 executions with
code from downstream builds (95-100% success rates) [2].
Could you run the current master (without the dynamic_ownership patch)
so that we have viable comparision?
From what I could see so far, there is an issue with move and copy
operations to and from Gluster domains. For example [3].
The logs are attached.
[1]
*https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner-tier1-after-upgrade/7/testReport/
<https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-4.2-ge-runner...
[2]
https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
rhv-4.2-ge-runner-tier1-after-upgrade/7/
[3]
2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
from=:
:ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in deleteImage
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503, in
deleteImage
raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
ImageDoesNotExistInSD: Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33,
domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
(Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
"Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
(task:1181)
2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH
deleteImage error=Image does not exist in domain:
'image=cabb8846-7a4b-4244-9835-5f603e682f33, domain=e5fd29c8-52ba-467e-be09
-ca40ff054d
d4' (dispatcher:82)
On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
> Triggered a sanity tier1 execution [1] using [2], which covers all the
> requested areas, on iSCSI, NFS and Gluster.
> I'll update with the results.
>
> [1]
> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
> _dev/job/rhv-4.2-ge-flow-storage/1161/
>
> [2]
> https://gerrit.ovirt.org/#/c/89830/
> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>
>
>
> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>
>>> Hi Martin,
>>>
>>> I see [1] requires a rebase, can you please take care?
>>>
>>
>> Should be rebased.
>>
>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>> FC.
>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR, it's
not
>>> stable enough at the moment.
>>>
>>
>> That is still pretty good.
>>
>>
>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>
>>>
>>> Thanks
>>>
>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>> wrote:
>>>
>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>
>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>> areas
>>>>> have to be tested here.
>>>>>
>>>>>
>>>> I'd say that you have quite a bit of freedom in this regard.
GlusterFS
>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some suite
>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>> and merging them, and whatever else would be important for storage
>>>> sanity.
>>>>
>>>> mpolednik
>>>>
>>>>
>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik
<mpolednik(a)redhat.com
>>>> >
>>>>
>>>>> wrote:
>>>>>
>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>>
>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph and
cinder,
>>>>>>
>>>>>>> will
>>>>>>> have to check, since usually, we don't execute our
automation on
>>>>>>> them.
>>>>>>>
>>>>>>>
>>>>>>> Any update on this? I believe the gluster tests were
successful, OST
>>>>>> passes fine and unit tests pass fine, that makes the storage
backends
>>>>>> test the last required piece.
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>> +Elad
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>> possible
>>>>>>>>>>
>>>>>>>>>> before merging ( using 'ci please build'
)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But note that OST is not a way to verify the
patch.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Such changes require testing with all storage
types we support.
>>>>>>>>>>
>>>>>>>>>> Nir
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik
<
>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>> libvirt's
>>>>>>>>>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>>
>>>>>>>>>>>> That of course comes with quite a bit of
code removal, mostly in
>>>>>>>>>>>> the
>>>>>>>>>>>> area of host devices, hwrng and anything
that touches devices;
>>>>>>>>>>>> bunch
>>>>>>>>>>>> of test changes and one XML generation
caveat (storage is
>>>>>>>>>>>> handled
>>>>>>>>>>>> by
>>>>>>>>>>>> VDSM, therefore disk relabelling needs to
be disabled on the
>>>>>>>>>>>> VDSM
>>>>>>>>>>>> level).
>>>>>>>>>>>>
>>>>>>>>>>>> Because of the scope of the patch, I
welcome
>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>> people to review the code and consider
the implication this
>>>>>>>>>>>> change
>>>>>>>>>>>> has
>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>
>>>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In particular: dynamic_ownership was set
to 0 prehistorically
>>>>>>>>>>>> (as
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> part
>>>>>>>>>>
>>>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=554961
) because
>>>>>>>>> libvirt,
>>>>>>>>> running as root, was not able to play properly with
root-squash nfs
>>>>>>>>> mounts.
>>>>>>>>>
>>>>>>>>> Have you attempted this use case?
>>>>>>>>>
>>>>>>>>> I join to Nir's request to run this with storage
QE.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> Raz Tamir
>>>>>>>> Manager, RHV QE
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
5:37 p.m.
I will update with the results of the next tier1 execution on latest 4.2.3
On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
> Hi, I've triggered another execution [1] due to some issues I saw in the
> first which are not related to the patch.
>
> The success rate is 78% which is low comparing to tier1 executions with
> code from downstream builds (95-100% success rates) [2].
>
Could you run the current master (without the dynamic_ownership patch)
so that we have viable comparision?
From what I could see so far, there is an issue with move and copy
> operations to and from Gluster domains. For example [3].
>
> The logs are attached.
>
>
> [1]
> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>
>
>
> [2]
> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>
> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>
>
>
> [3]
> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
> deleteImage error=Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
> from=:
> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
> in
> _run
> return fn(*args, **kargs)
> File "<string>", line 2, in deleteImage
> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
> method
> ret = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1503,
> in
> deleteImage
> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
> ImageDoesNotExistInSD: Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>
> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
> "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
> (task:1181)
> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH
> deleteImage error=Image does not exist in domain:
> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
> domain=e5fd29c8-52ba-467e-be09
> -ca40ff054d
> d4' (dispatcher:82)
>
>
>
> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
> wrote:
>
> Triggered a sanity tier1 execution [1] using [2], which covers all the
>> requested areas, on iSCSI, NFS and Gluster.
>> I'll update with the results.
>>
>> [1]
>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>
>> [2]
>> https://gerrit.ovirt.org/#/c/89830/
>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>
>>
>>
>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>
>>> Hi Martin,
>>>>
>>>> I see [1] requires a rebase, can you please take care?
>>>>
>>>>
>>> Should be rebased.
>>>
>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>>
>>>> FC.
>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>> not
>>>> stable enough at the moment.
>>>>
>>>>
>>> That is still pretty good.
>>>
>>>
>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>> >
>>>> wrote:
>>>>
>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>
>>>>>
>>>>> Hi, sorry if I misunderstood, I waited for more input regarding what
>>>>>
>>>>>> areas
>>>>>> have to be tested here.
>>>>>>
>>>>>>
>>>>>> I'd say that you have quite a bit of freedom in this regard.
>>>>> GlusterFS
>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>> and merging them, and whatever else would be important for storage
>>>>> sanity.
>>>>>
>>>>> mpolednik
>>>>>
>>>>>
>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>> mpolednik(a)redhat.com
>>>>> >
>>>>>
>>>>> wrote:
>>>>>>
>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>
>>>>>>
>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>>>>>>> cinder,
>>>>>>>
>>>>>>> will
>>>>>>>> have to check, since usually, we don't execute our
automation on
>>>>>>>> them.
>>>>>>>>
>>>>>>>>
>>>>>>>> Any update on this? I believe the gluster tests were
successful,
>>>>>>>> OST
>>>>>>>>
>>>>>>> passes fine and unit tests pass fine, that makes the storage
>>>>>>> backends
>>>>>>> test the last required piece.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> +Elad
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>>>>>>>>> >
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>>> possible
>>>>>>>>>>>
>>>>>>>>>>> before merging ( using 'ci please
build' )
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But note that OST is not a way to verify
the patch.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Such changes require testing with all storage
types we support.
>>>>>>>>>>>
>>>>>>>>>>> Nir
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hey,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>> dynamic_ownership for VDSM while not
negatively affecting
>>>>>>>>>>>>> functionality of our storage code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That of course comes with quite a bit
of code removal, mostly
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> area of host devices, hwrng and
anything that touches devices;
>>>>>>>>>>>>> bunch
>>>>>>>>>>>>> of test changes and one XML
generation caveat (storage is
>>>>>>>>>>>>> handled
>>>>>>>>>>>>> by
>>>>>>>>>>>>> VDSM, therefore disk relabelling
needs to be disabled on the
>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>> level).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Because of the scope of the patch, I
welcome
>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>> people to review the code and
consider the implication this
>>>>>>>>>>>>> change
>>>>>>>>>>>>> has
>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In particular: dynamic_ownership was
set to 0 prehistorically
>>>>>>>>>>>>> (as
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> part
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>>>>>> libvirt,
>>>>>>>>>> running as root, was not able to play properly
with root-squash
>>>>>>>>>> nfs
>>>>>>>>>> mounts.
>>>>>>>>>>
>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>
>>>>>>>>>> I join to Nir's request to run this with
storage QE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raz Tamir
>>>>>>>>> Manager, RHV QE
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>
3:12 a.m.
Here it is - https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
rhv-4.2-ge-runner-tier1/122/
This was executed over iscsi, nfs, gluster and fcp. It ended up with 98.43%
success rate.
2 storage related test cases failed and their failures, from first glance,
don't seem to be bugs.
On Tue, Apr 24, 2018 at 12:37 AM, Elad Ben Aharon <ebenahar(a)redhat.com>
wrote:
I will update with the results of the next tier1 execution on latest
4.2.3
On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>
>> Hi, I've triggered another execution [1] due to some issues I saw in the
>> first which are not related to the patch.
>>
>> The success rate is 78% which is low comparing to tier1 executions with
>> code from downstream builds (95-100% success rates) [2].
>>
>
> Could you run the current master (without the dynamic_ownership patch)
> so that we have viable comparision?
>
> From what I could see so far, there is an issue with move and copy
>> operations to and from Gluster domains. For example [3].
>>
>> The logs are attached.
>>
>>
>> [1]
>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>
>>
>>
>> [2]
>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>
>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>
>>
>>
>> [3]
>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>> deleteImage error=Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> from=:
>> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
>> (task:875)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
882,
>> in
>> _run
>> return fn(*args, **kargs)
>> File "<string>", line 2, in deleteImage
>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49,
in
>> method
>> ret = func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
1503,
>> in
>> deleteImage
>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> ImageDoesNotExistInSD: Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>
>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
>> "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
>> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
>> (task:1181)
>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher]
>> FINISH
>> deleteImage error=Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09
>> -ca40ff054d
>> d4' (dispatcher:82)
>>
>>
>>
>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>>
>> Triggered a sanity tier1 execution [1] using [2], which covers all the
>>> requested areas, on iSCSI, NFS and Gluster.
>>> I'll update with the results.
>>>
>>> [1]
>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>
>>> [2]
>>> https://gerrit.ovirt.org/#/c/89830/
>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>
>>>
>>>
>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>> wrote:
>>>
>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>
>>>> Hi Martin,
>>>>>
>>>>> I see [1] requires a rebase, can you please take care?
>>>>>
>>>>>
>>>> Should be rebased.
>>>>
>>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>>>
>>>>> FC.
>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>>> not
>>>>> stable enough at the moment.
>>>>>
>>>>>
>>>> That is still pretty good.
>>>>
>>>>
>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik <
>>>>> mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>>
>>>>>> Hi, sorry if I misunderstood, I waited for more input regarding
what
>>>>>>
>>>>>>> areas
>>>>>>> have to be tested here.
>>>>>>>
>>>>>>>
>>>>>>> I'd say that you have quite a bit of freedom in this
regard.
>>>>>> GlusterFS
>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>>> and merging them, and whatever else would be important for
storage
>>>>>> sanity.
>>>>>>
>>>>>> mpolednik
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>>> mpolednik(a)redhat.com
>>>>>> >
>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>>
>>>>>>>
>>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>>>>>>>> cinder,
>>>>>>>>
>>>>>>>> will
>>>>>>>>> have to check, since usually, we don't execute
our automation on
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any update on this? I believe the gluster tests were
successful,
>>>>>>>>> OST
>>>>>>>>>
>>>>>>>> passes fine and unit tests pass fine, that makes the
storage
>>>>>>>> backends
>>>>>>>> test the last required piece.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> +Elad
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <
>>>>>>>>>> danken(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>>>> possible
>>>>>>>>>>>>
>>>>>>>>>>>> before merging ( using 'ci please
build' )
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> But note that OST is not a way to
verify the patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Such changes require testing with all
storage types we support.
>>>>>>>>>>>>
>>>>>>>>>>>> Nir
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>>> dynamic_ownership for VDSM while
not negatively affecting
>>>>>>>>>>>>>> functionality of our storage
code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That of course comes with quite a
bit of code removal,
>>>>>>>>>>>>>> mostly in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> area of host devices, hwrng and
anything that touches
>>>>>>>>>>>>>> devices;
>>>>>>>>>>>>>> bunch
>>>>>>>>>>>>>> of test changes and one XML
generation caveat (storage is
>>>>>>>>>>>>>> handled
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>> VDSM, therefore disk relabelling
needs to be disabled on the
>>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>>> level).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Because of the scope of the
patch, I welcome
>>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>>> people to review the code and
consider the implication this
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>> has
>>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In particular: dynamic_ownership
was set to 0
>>>>>>>>>>>>>> prehistorically
>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> part
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 )
>>>>>>>>>>> because
>>>>>>>>>>> libvirt,
>>>>>>>>>>> running as root, was not able to play
properly with root-squash
>>>>>>>>>>> nfs
>>>>>>>>>>> mounts.
>>>>>>>>>>>
>>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>>
>>>>>>>>>>> I join to Nir's request to run this with
storage QE.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Raz Tamir
>>>>>>>>>> Manager, RHV QE
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>
>
>
3:23 a.m.
On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
I will update with the results of the next tier1 execution on latest
4.2.3
That isn't master but old branch though. Could you run it against
*current* VDSM master?
On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik
<mpolednik(a)redhat.com>
wrote:
> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>
>> Hi, I've triggered another execution [1] due to some issues I saw in the
>> first which are not related to the patch.
>>
>> The success rate is 78% which is low comparing to tier1 executions with
>> code from downstream builds (95-100% success rates) [2].
>>
>
> Could you run the current master (without the dynamic_ownership patch)
> so that we have viable comparision?
>
> From what I could see so far, there is an issue with move and copy
>> operations to and from Gluster domains. For example [3].
>>
>> The logs are attached.
>>
>>
>> [1]
>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>
>>
>>
>> [2]
>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>
>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>
>>
>>
>> [3]
>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>> deleteImage error=Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>> from=:
>> :ffff:10.35.161.182,40936, flow_id=disks_syncAction_ba6b2630-5976-4935,
>> task_id=3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task]
>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error
(task:875)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
882,
>> in
>> _run
>> return fn(*args, **kargs)
>> File "<string>", line 2, in deleteImage
>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49,
in
>> method
>> ret = func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
1503,
>> in
>> deleteImage
>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>> ImageDoesNotExistInSD: Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>
>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) [storage.TaskManager.Task]
>> (Task='3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is aborted:
>> "Image does not exist in domain: 'image=cabb8846-7a4b-4244-9835-
>> 5f603e682f33, domain=e5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268
>> (task:1181)
>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH
>> deleteImage error=Image does not exist in domain:
>> 'image=cabb8846-7a4b-4244-9835-5f603e682f33,
>> domain=e5fd29c8-52ba-467e-be09
>> -ca40ff054d
>> d4' (dispatcher:82)
>>
>>
>>
>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon <ebenahar(a)redhat.com>
>> wrote:
>>
>> Triggered a sanity tier1 execution [1] using [2], which covers all the
>>> requested areas, on iSCSI, NFS and Gluster.
>>> I'll update with the results.
>>>
>>> [1]
>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>
>>> [2]
>>> https://gerrit.ovirt.org/#/c/89830/
>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>
>>>
>>>
>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik
<mpolednik(a)redhat.com>
>>> wrote:
>>>
>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>
>>>> Hi Martin,
>>>>>
>>>>> I see [1] requires a rebase, can you please take care?
>>>>>
>>>>>
>>>> Should be rebased.
>>>>
>>>> At the moment, our automation is stable only on iSCSI, NFS, Gluster and
>>>>
>>>>> FC.
>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
it's
>>>>> not
>>>>> stable enough at the moment.
>>>>>
>>>>>
>>>> That is still pretty good.
>>>>
>>>>
>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik
<mpolednik(a)redhat.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>
>>>>>>
>>>>>> Hi, sorry if I misunderstood, I waited for more input regarding
what
>>>>>>
>>>>>>> areas
>>>>>>> have to be tested here.
>>>>>>>
>>>>>>>
>>>>>>> I'd say that you have quite a bit of freedom in this
regard.
>>>>>> GlusterFS
>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with some
suite
>>>>>> that covers basic operations (start & stop VM, migrate it),
snapshots
>>>>>> and merging them, and whatever else would be important for
storage
>>>>>> sanity.
>>>>>>
>>>>>> mpolednik
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>>> mpolednik(a)redhat.com
>>>>>> >
>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>>
>>>>>>>
>>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph
and
>>>>>>>> cinder,
>>>>>>>>
>>>>>>>> will
>>>>>>>>> have to check, since usually, we don't execute
our automation on
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any update on this? I believe the gluster tests were
successful,
>>>>>>>>> OST
>>>>>>>>>
>>>>>>>> passes fine and unit tests pass fine, that makes the
storage
>>>>>>>> backends
>>>>>>>> test the last required piece.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir
<ratamir(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> +Elad
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg
<danken(a)redhat.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please make sure to run as much OST suites on
this patch as
>>>>>>>>>>>> possible
>>>>>>>>>>>>
>>>>>>>>>>>> before merging ( using 'ci please
build' )
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> But note that OST is not a way to
verify the patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Such changes require testing with all
storage types we support.
>>>>>>>>>>>>
>>>>>>>>>>>> Nir
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin
Polednik <
>>>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've created a patch[0] that is
finally able to activate
>>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>>> dynamic_ownership for VDSM while
not negatively affecting
>>>>>>>>>>>>>> functionality of our storage
code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That of course comes with quite a
bit of code removal, mostly
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> area of host devices, hwrng and
anything that touches devices;
>>>>>>>>>>>>>> bunch
>>>>>>>>>>>>>> of test changes and one XML
generation caveat (storage is
>>>>>>>>>>>>>> handled
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>> VDSM, therefore disk relabelling
needs to be disabled on the
>>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>>> level).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Because of the scope of the
patch, I welcome
>>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>>> people to review the code and
consider the implication this
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>> has
>>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In particular: dynamic_ownership
was set to 0 prehistorically
>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> part
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because
>>>>>>>>>>> libvirt,
>>>>>>>>>>> running as root, was not able to play
properly with root-squash
>>>>>>>>>>> nfs
>>>>>>>>>>> mounts.
>>>>>>>>>>>
>>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>>
>>>>>>>>>>> I join to Nir's request to run this with
storage QE.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Raz Tamir
>>>>>>>>>> Manager, RHV QE
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>
>
>
7:38 a.m.
--Apple-Mail=_807B5345-431A-48DA-A469-7819B1EAE0EF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Hi Elad,
to make it easier to compare, Martin backported the change to 4.2 so it =
is actually comparable with a run without that patch. Would you please =
try that out?=20
It would be best to have 4.2 upstream and this[1] run to really minimize =
the noise.
Thanks,
michal
[1] =
http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_64=
/28/ =
<http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_6=
4/28/>
On 27 Apr 2018, at 09:23, Martin Polednik
<mpolednik(a)redhat.com> =
wrote:
=20
On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:
> I will update with the results of the next tier1 execution on latest =
4.2.3
=20
That isn't master but old branch though. Could you run it against
*current* VDSM master?
=20
> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik =
<mpolednik(a)redhat.com>
> wrote:
>=20
>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote:
>>=20
>>> Hi, I've triggered another execution [1] due to some issues I saw =
in the
>>> first which are not related to the patch.
>>>=20
>>> The success rate is 78% which is low comparing to tier1 executions =
with
>>> code from downstream builds (95-100% success rates) [2].
>>>=20
>>=20
>> Could you run the current master (without the dynamic_ownership =
patch)
>> so that we have viable comparision?
>>=20
>> =46rom what I could see so far, there is an issue with move and copy
>>> operations to and from Gluster domains. For example [3].
>>>=20
>>> The logs are attached.
>>>=20
>>>=20
>>> [1]
>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/
>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv
>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>*
>>>=20
>>>=20
>>>=20
>>> [2]
>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/
>>>=20
>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/
>>>=20
>>>=20
>>>=20
>>> [3]
>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH
>>> deleteImage error=3DImage does not exist in domain:
>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>> from=3D:
>>> :ffff:10.35.161.182,40936, =
flow_id=3Ddisks_syncAction_ba6b2630-5976-4935,
>>> task_id=3D3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)
>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) =
[storage.TaskManager.Task]
>>> (Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42')
Unexpected error =
(task:875)
>>> Traceback (most recent call last):
>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
=
882,
>>> in
>>> _run
>>> return fn(*args, **kargs)
>>> File "<string>", line 2, in deleteImage
>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line =
49, in
>>> method
>>> ret =3D func(*args, **kwargs)
>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line =
1503,
>>> in
>>> deleteImage
>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
>>> ImageDoesNotExistInSD: Image does not exist in domain:
>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'
>>>=20
>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) =
[storage.TaskManager.Task]
>>> (Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42')
aborting: Task is =
aborted:
>>> "Image does not exist in domain:
'image=3Dcabb8846-7a4b-4244-9835-
>>> 5f603e682f33, domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'" - =
code 268
>>> (task:1181)
>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) [storage.Dispatcher] =
FINISH
>>> deleteImage error=3DImage does not exist in domain:
>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,
>>> domain=3De5fd29c8-52ba-467e-be09
>>> -ca40ff054d
>>> d4' (dispatcher:82)
>>>=20
>>>=20
>>>=20
>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon =
<ebenahar(a)redhat.com>
>>> wrote:
>>>=20
>>> Triggered a sanity tier1 execution [1] using [2], which covers all =
the
>>>> requested areas, on iSCSI, NFS and Gluster.
>>>> I'll update with the results.
>>>>=20
>>>> [1]
>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2
>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/
>>>>=20
>>>> [2]
>>>> https://gerrit.ovirt.org/#/c/89830/
>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64
>>>>=20
>>>>=20
>>>>=20
>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik =
<mpolednik(a)redhat.com>
>>>> wrote:
>>>>=20
>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote:
>>>>>=20
>>>>> Hi Martin,
>>>>>>=20
>>>>>> I see [1] requires a rebase, can you please take care?
>>>>>>=20
>>>>>>=20
>>>>> Should be rebased.
>>>>>=20
>>>>> At the moment, our automation is stable only on iSCSI, NFS, =
Gluster and
>>>>>=20
>>>>>> FC.
>>>>>> Ceph is not supported and Cinder will be stabilized soon, AFAIR,
=
it's
>>>>>> not
>>>>>> stable enough at the moment.
>>>>>>=20
>>>>>>=20
>>>>> That is still pretty good.
>>>>>=20
>>>>>=20
>>>>> [1] https://gerrit.ovirt.org/#/c/89830/
>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> Thanks
>>>>>>=20
>>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik =
<mpolednik(a)redhat.com
>>>>>> >
>>>>>> wrote:
>>>>>>=20
>>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote:
>>>>>>=20
>>>>>>>=20
>>>>>>> Hi, sorry if I misunderstood, I waited for more input
regarding =
what
>>>>>>>=20
>>>>>>>> areas
>>>>>>>> have to be tested here.
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> I'd say that you have quite a bit of freedom in this
regard.
>>>>>>> GlusterFS
>>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with
some =
suite
>>>>>>> that covers basic operations (start &
stop VM, migrate it), =
snapshots
>>>>>>> and merging them, and whatever else would
be important for =
storage
>>>>>>> sanity.
>>>>>>>=20
>>>>>>> mpolednik
>>>>>>>=20
>>>>>>>=20
>>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik <
>>>>>>> mpolednik(a)redhat.com
>>>>>>> >
>>>>>>>=20
>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for
ceph and
>>>>>>>>> cinder,
>>>>>>>>>=20
>>>>>>>>> will
>>>>>>>>>> have to check, since usually, we don't
execute our =
automation on
>>>>>>>>>> them.
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> Any update on this? I believe the gluster tests
were =
successful,
>>>>>>>>>> OST
>>>>>>>>>>=20
>>>>>>>>> passes fine and unit tests pass fine, that makes the
storage
>>>>>>>>> backends
>>>>>>>>> test the last required piece.
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir =
<ratamir(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> +Elad
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg =
<danken(a)redhat.com
>>>>>>>>>>> >
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
=
<nsoffer(a)redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri =
<eedri(a)redhat.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> Please make sure to run as much OST
suites on this patch =
as
>>>>>>>>>>>>> possible
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> before merging ( using 'ci please
build' )
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> But note that OST is not a way to
verify the patch.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Such changes require testing with all
storage types we =
support.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Nir
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM,
Martin Polednik <
>>>>>>>>>>>>> mpolednik(a)redhat.com
>>>>>>>>>>>>> >
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> I've created a patch[0] that
is finally able to activate
>>>>>>>>>>>>>>> libvirt's
>>>>>>>>>>>>>>> dynamic_ownership for VDSM
while not negatively =
affecting
>>>>>>>>>>>>>>>
functionality of our storage code.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> That of course comes with
quite a bit of code removal, =
mostly
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> area of host devices, hwrng
and anything that touches =
devices;
>>>>>>>>>>>>>>> bunch
>>>>>>>>>>>>>>> of test changes and one XML
generation caveat (storage =
is
>>>>>>>>>>>>>>> handled
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> VDSM, therefore disk
relabelling needs to be disabled =
on the
>>>>>>>>>>>>>>> VDSM
>>>>>>>>>>>>>>> level).
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Because of the scope of the
patch, I welcome
>>>>>>>>>>>>>>> storage/virt/network
>>>>>>>>>>>>>>> people to review the code and
consider the implication =
this
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>> on current/future features.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> [0]
https://gerrit.ovirt.org/#/c/89830/
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> In particular:
dynamic_ownership was set to 0 =
prehistorically
>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> of
https://bugzilla.redhat.com/show_bug.cgi?id=3D554961 ) =
because
>>>>>>>>>>>> libvirt,
>>>>>>>>>>>> running as root, was not able to play
properly with =
root-squash
>>>>>>>>>>>> nfs
>>>>>>>>>>>> mounts.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Have you attempted this use case?
>>>>>>>>>>>>=20
>>>>>>>>>>>> I join to Nir's request to run this
with storage QE.
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> --
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> Raz Tamir
>>>>>>>>>>> Manager, RHV QE
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>=20
>>=20
>>=20
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
=20
=20
--Apple-Mail=_807B5345-431A-48DA-A469-7819B1EAE0EF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hi
=
Elad,<div class=3D"">to make it easier to compare, Martin backported the
=
change to 4.2 so it is actually comparable with a run without that =
patch. Would you please try that out? </div><div
class=3D"">It =
would be best to have 4.2 upstream and this[1] run to really minimize =
the noise.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Thanks,</div><div
class=3D"">michal</div><div class=3D""><br =
class=3D""></div><div class=3D"">[1] <a =
href=3D"http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-dem...
7-x86_64/28/" =
class=3D"">http://jenkins.ovirt.org/job/vdsm_4.2_build-artif...
-el7-x86_64/28/</a></div><div class=3D""><br
class=3D""><div><blockquote =
type=3D"cite" class=3D""><div class=3D"">On 27 Apr
2018, at 09:23, =
Martin Polednik <<a href=3D"mailto:mpolednik@redhat.com" =
class=3D"">mpolednik(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
class=3D"">On =
24/04/18 00:37 +0300, Elad Ben Aharon wrote:<br class=3D""><blockquote
=
type=3D"cite" class=3D"">I will update with the results of the next
=
tier1 execution on latest 4.2.3<br class=3D""></blockquote><br =
class=3D"">That isn't master but old branch though. Could you run it =
against<br class=3D"">*current* VDSM master?<br
class=3D""><br =
class=3D""><blockquote type=3D"cite" class=3D"">On
Mon, Apr 23, 2018 at =
3:56 PM, Martin Polednik <<a href=3D"mailto:mpolednik@redhat.com" =
class=3D"">mpolednik(a)redhat.com</a>&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D"">On =
23/04/18 01:23 +0300, Elad Ben Aharon wrote:<br class=3D""><br =
class=3D""><blockquote type=3D"cite" class=3D"">Hi,
I've triggered =
another execution [1] due to some issues I saw in the<br class=3D"">first
=
which are not related to the patch.<br class=3D""><br
class=3D"">The =
success rate is 78% which is low comparing to tier1 executions with<br =
class=3D"">code from downstream builds (95-100% success rates) [2].<br =
class=3D""><br class=3D""></blockquote><br
class=3D"">Could you run the =
current master (without the dynamic_ownership patch)<br class=3D"">so =
that we have viable comparision?<br class=3D""><br
class=3D"">=46rom =
what I could see so far, there is an issue with move and copy<br =
class=3D""><blockquote type=3D"cite"
class=3D"">operations to and from =
Gluster domains. For example [3].<br class=3D""><br
class=3D"">The logs =
are attached.<br class=3D""><br class=3D""><br
class=3D"">[1]<br =
class=3D"">*<a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv&... =
class=3D"">https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.c...
<br class=3D"">-4.2-ge-runner-tier1-after-upgrade/7/testReport/<br =
class=3D""><<a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv&... =
class=3D"">https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.c...
<br
class=3D"">-4.2-ge-runner-tier1-after-upgrade/7/testReport/>*<br =
class=3D""><br class=3D""><br
class=3D""><br class=3D"">[2]<br =
class=3D""><a =
href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/" =
class=3D"">https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.c...
class=3D""><br
class=3D"">rhv-4.2-ge-runner-tier1-after-upgrade/7/<br =
class=3D""><br class=3D""><br
class=3D""><br class=3D"">[3]<br =
class=3D"">2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) =
[vdsm.api] FINISH<br class=3D"">deleteImage error=3DImage does not exist
=
in domain:<br
class=3D"">'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,<br=
class=3D"">domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'<br =
class=3D"">from=3D:<br class=3D"">:ffff:10.35.161.182,40936,
=
flow_id=3Ddisks_syncAction_ba6b2630-5976-4935,<br =
class=3D"">task_id=3D3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51)<br =
class=3D"">2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) =
[storage.TaskManager.Task]<br =
class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected
=
error (task:875)<br class=3D"">Traceback (most recent call last):<br =
class=3D"">File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", =
line 882,<br class=3D"">in<br class=3D"">_run<br
class=3D""> =
return fn(*args, **kargs)<br class=3D"">File
"<string>", =
line 2, in deleteImage<br class=3D"">File =
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in<br =
class=3D"">method<br class=3D""> ret =3D
func(*args, **kwargs)<br =
class=3D"">File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", =
line 1503,<br class=3D"">in<br
class=3D"">deleteImage<br class=3D""> =
raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)<br =
class=3D"">ImageDoesNotExistInSD: Image does not exist in domain:<br =
class=3D"">'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,<br =
class=3D"">domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'<br =
class=3D""><br class=3D"">2018-04-22 13:06:28,317+0300 INFO
=
(jsonrpc/7) [storage.TaskManager.Task]<br =
class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting:
=
Task is aborted:<br class=3D"">"Image does not exist in domain: =
'image=3Dcabb8846-7a4b-4244-9835-<br class=3D"">5f603e682f33, =
domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'" - code 268<br =
class=3D"">(task:1181)<br class=3D"">2018-04-22
13:06:28,318+0300 ERROR =
(jsonrpc/7) [storage.Dispatcher] FINISH<br class=3D"">deleteImage =
error=3DImage does not exist in domain:<br =
class=3D"">'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33,<br =
class=3D"">domain=3De5fd29c8-52ba-467e-be09<br
class=3D"">-ca40ff054d<br =
class=3D"">d4' (dispatcher:82)<br class=3D""><br
class=3D""><br =
class=3D""><br class=3D"">On Thu, Apr 19, 2018 at 5:34 PM,
Elad Ben =
Aharon &lt;ebenahar(a)redhat.com&gt;<br class=3D"">wrote:<br
class=3D""><br =
class=3D"">Triggered a sanity tier1 execution [1] using [2], which =
covers all the<br class=3D""><blockquote type=3D"cite" =
class=3D"">requested areas, on iSCSI, NFS and Gluster.<br
class=3D"">I'll =
update with the results.<br class=3D""><br
class=3D"">[1]<br =
class=3D"">https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.c...
class=3D"">_dev/job/rhv-4.2-ge-flow-storage/1161/<br
class=3D""><br =
class=3D"">[2]<br
class=3D"">https://gerrit.ovirt.org/#/c/89830/<br =
class=3D"">vdsm-4.30.0-291.git77aef9a.el7.x86_64<br
class=3D""><br =
class=3D""><br class=3D""><br class=3D"">On
Thu, Apr 19, 2018 at 3:07 =
PM, Martin Polednik &lt;mpolednik(a)redhat.com&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D"">On 19/04/18 14:54 +0300, Elad Ben
Aharon =
wrote:<br class=3D""><blockquote type=3D"cite"
class=3D""><br =
class=3D"">Hi Martin,<br class=3D""><blockquote
type=3D"cite" =
class=3D""><br class=3D"">I see [1] requires a rebase, can
you please =
take care?<br class=3D""><br class=3D""><br
class=3D""></blockquote>Should=
be rebased.<br class=3D""><br class=3D"">At the moment,
our automation =
is stable only on iSCSI, NFS, Gluster and<br class=3D""><br =
class=3D""><blockquote type=3D"cite"
class=3D"">FC.<br class=3D"">Ceph =
is not supported and Cinder will be stabilized soon, AFAIR, it's<br =
class=3D"">not<br class=3D"">stable enough at the
moment.<br =
class=3D""><br class=3D""><br
class=3D""></blockquote>That is still =
pretty good.<br class=3D""><br class=3D""><br
class=3D"">[1] =
https://gerrit.ovirt.org/#/c/89830/<br class=3D""><br =
class=3D""><blockquote type=3D"cite"
class=3D""><br class=3D""><br =
class=3D"">Thanks<br class=3D""><br
class=3D"">On Wed, Apr 18, 2018 at =
2:17 PM, Martin Polednik &lt;mpolednik(a)redhat.com<br
class=3D"">><br =
class=3D"">wrote:<br class=3D""><br
class=3D"">On 18/04/18 11:37 +0300, =
Elad Ben Aharon wrote:<br class=3D""><br
class=3D""><blockquote =
type=3D"cite" class=3D""><br class=3D"">Hi, sorry
if I misunderstood, I =
waited for more input regarding what<br class=3D""><br =
class=3D""><blockquote type=3D"cite"
class=3D"">areas<br class=3D"">have =
to be tested here.<br class=3D""><br class=3D""><br
class=3D"">I'd say =
that you have quite a bit of freedom in this regard.<br =
class=3D""></blockquote>GlusterFS<br class=3D"">should
be covered by =
Dennis, so iSCSI/NFS/ceph/cinder with some suite<br class=3D"">that =
covers basic operations (start & stop VM, migrate it), snapshots<br =
class=3D"">and merging them, and whatever else would be important for =
storage<br class=3D"">sanity.<br class=3D""><br
class=3D"">mpolednik<br =
class=3D""><br class=3D""><br class=3D"">On
Wed, Apr 18, 2018 at 11:16 =
AM, Martin Polednik <<br class=3D"">mpolednik(a)redhat.com<br =
class=3D"">><br class=3D""><br
class=3D"">wrote:<br =
class=3D""><blockquote type=3D"cite"
class=3D""><br class=3D"">On =
11/04/18 16:52 +0300, Elad Ben Aharon wrote:<br class=3D""><br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D"">We can =
test this on iSCSI, NFS and GlusterFS. As for ceph and<br =
class=3D"">cinder,<br class=3D""><br
class=3D"">will<br =
class=3D""><blockquote type=3D"cite"
class=3D"">have to check, since =
usually, we don't execute our automation on<br class=3D"">them.<br
=
class=3D""><br class=3D""><br class=3D"">Any
update on this? I believe =
the gluster tests were successful,<br class=3D"">OST<br
class=3D""><br =
class=3D""></blockquote>passes fine and unit tests pass fine, that
makes =
the storage<br class=3D"">backends<br class=3D"">test the
last required =
piece.<br class=3D""><br class=3D""><br
class=3D"">On Wed, Apr 11, 2018 =
at 4:38 PM, Raz Tamir &lt;ratamir(a)redhat.com&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D""><br
class=3D"">+Elad<br class=3D""><blockquote =
type=3D"cite" class=3D""><br class=3D""><br
class=3D"">On Wed, Apr 11, =
2018 at 4:28 PM, Dan Kenigsberg &lt;danken(a)redhat.com<br =
class=3D""><blockquote type=3D"cite"
class=3D"">><br =
class=3D"">wrote:<br class=3D""><br
class=3D"">On Wed, Apr 11, 2018 at =
12:34 PM, Nir Soffer &lt;nsoffer(a)redhat.com&gt;<br
class=3D"">wrote:<br =
class=3D""><br class=3D""><br class=3D"">On
Wed, Apr 11, 2018 at 12:31 =
PM Eyal Edri &lt;eedri(a)redhat.com&gt;<br class=3D""><blockquote
=
type=3D"cite" class=3D"">wrote:<br
class=3D""><br class=3D""><br =
class=3D"">Please make sure to run as much OST suites on this patch =
as<br class=3D""><blockquote type=3D"cite"
class=3D"">possible<br =
class=3D""><br class=3D"">before merging ( using 'ci
please build' )<br =
class=3D""><br class=3D""><blockquote
type=3D"cite" class=3D""><br =
class=3D""><br class=3D"">But note that OST is not a way to
verify the =
patch.<br class=3D""><br class=3D""><br
class=3D""></blockquote>Such =
changes require testing with all storage types we support.<br =
class=3D""><br class=3D"">Nir<br
class=3D""><br class=3D"">On Tue, Apr =
10, 2018 at 4:09 PM, Martin Polednik <<br =
class=3D"">mpolednik(a)redhat.com<br class=3D"">><br
class=3D""><br =
class=3D"">wrote:<br class=3D""><br
class=3D""><br class=3D""><blockquote =
type=3D"cite" class=3D"">Hey,<br class=3D""><br
class=3D""><br =
class=3D"">I've created a patch[0] that is finally able to activate<br
=
class=3D""><blockquote type=3D"cite"
class=3D"">libvirt's<br =
class=3D"">dynamic_ownership for VDSM while not negatively affecting<br =
class=3D"">functionality of our storage code.<br
class=3D""><br =
class=3D"">That of course comes with quite a bit of code removal, =
mostly<br class=3D"">in<br class=3D"">the<br
class=3D"">area of host =
devices, hwrng and anything that touches devices;<br
class=3D"">bunch<br =
class=3D"">of test changes and one XML generation caveat (storage is<br =
class=3D"">handled<br class=3D"">by<br
class=3D"">VDSM, therefore disk =
relabelling needs to be disabled on the<br class=3D"">VDSM<br =
class=3D"">level).<br class=3D""><br
class=3D"">Because of the scope of =
the patch, I welcome<br class=3D"">storage/virt/network<br =
class=3D"">people to review the code and consider the implication =
this<br class=3D"">change<br class=3D"">has<br
class=3D"">on =
current/future features.<br class=3D""><br class=3D"">[0]
=
https://gerrit.ovirt.org/#/c/89830/<br class=3D""><br
class=3D""><br =
class=3D"">In particular: dynamic_ownership was set to 0 =
prehistorically<br class=3D"">(as<br class=3D""><br
class=3D""><br =
class=3D""></blockquote>part<br class=3D""><br
class=3D""></blockquote><br=
class=3D"">of https://bugzilla.redhat.com/show_bug.cgi?id=3D554961 ) =
because<br class=3D""></blockquote>libvirt,<br
class=3D"">running as =
root, was not able to play properly with root-squash<br
class=3D"">nfs<br =
class=3D"">mounts.<br class=3D""><br
class=3D"">Have you attempted this =
use case?<br class=3D""><br class=3D"">I join to Nir's
request to run =
this with storage QE.<br class=3D""><br class=3D""><br
class=3D""><br =
class=3D""><br class=3D"">--<br
class=3D""></blockquote><br class=3D""><br=
class=3D"">Raz Tamir<br class=3D"">Manager, RHV QE<br
class=3D""><br =
class=3D""><br class=3D""><br
class=3D""><br class=3D""><br =
class=3D""></blockquote></blockquote></blockquote></blockquote></blockquot=
e></blockquote></blockquote><br
class=3D""></blockquote></blockquote><br =
class=3D""><br =
class=3D""></blockquote></blockquote>_____________________________________=
__________<br class=3D"">Devel mailing list<br
class=3D""><a =
href=3D"mailto:Devel@ovirt.org"
class=3D"">Devel(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/devel<br
class=3D""><br=
class=3D""><br
class=3D""></div></div></blockquote></div><br =
class=3D""></div></body></html>=
--Apple-Mail=_807B5345-431A-48DA-A469-7819B1EAE0EF--
9:43 a.m.
On 11/04/18 16:28 +0300, Dan Kenigsberg wrote:
On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri <eedri(a)redhat.com> wrote:
>
>> Please make sure to run as much OST suites on this patch as possible
>> before merging ( using 'ci please build' )
>>
>
> But note that OST is not a way to verify the patch.
>
> Such changes require testing with all storage types we support.
>
> Nir
>
> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik <mpolednik(a)redhat.com>
>> wrote:
>>
>>> Hey,
>>>
>>> I've created a patch[0] that is finally able to activate libvirt's
>>> dynamic_ownership for VDSM while not negatively affecting
>>> functionality of our storage code.
>>>
>>> That of course comes with quite a bit of code removal, mostly in the
>>> area of host devices, hwrng and anything that touches devices; bunch
>>> of test changes and one XML generation caveat (storage is handled by
>>> VDSM, therefore disk relabelling needs to be disabled on the VDSM
>>> level).
>>>
>>> Because of the scope of the patch, I welcome storage/virt/network
>>> people to review the code and consider the implication this change has
>>> on current/future features.
>>>
>>> [0] https://gerrit.ovirt.org/#/c/89830/
>>>
>>
In particular: dynamic_ownership was set to 0 prehistorically (as part of
https://bugzilla.redhat.com/show_bug.cgi?id=554961 ) because libvirt,
running as root, was not able to play properly with root-squash nfs mounts.
Have you attempted this use case?
I have not. Added this to my to-do list.
The important part to note about this patch (compared to my previous
attempts in the past) is that it explicitly disables dynamic_ownership
for FILE/BLOCK-backed disks. That means, unless `seclabel` is broken
on libivrt side, the behavior would be unchanged for storage.
I join to Nir's request to run this with storage QE.
2426
days inactive
2450
days old
25 comments
8 participants
participants (8)
-
Dan Kenigsberg -
Elad Ben Aharon -
Eyal Edri -
Martin Polednik -
Michal Skrivanek -
Nir Soffer -
Raz Tamir -
Yaniv Kaul