[ovirt-devel] [ OST Failure Report ] [ oVirtMaster ] [ 07-11-2017 ] [007_sd_reattach.deactivate_storage_domain ]

Sun Dec 10 10:38:25 UTC 2017

On Fri, Dec 8, 2017 at 11:20 PM, Yaniv Kaul <ykaul at redhat.com> wrote:

>
>
> On Fri, Dec 8, 2017 at 10:39 PM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>>
>>
>> On Fri, Dec 8, 2017 at 9:31 PM, Dafna Ron <dron at redhat.com> wrote:
>>
>>> I opened a bug: https://bugzilla.redhat.com/show_bug.cgi?id=1523813
>>>
>>
>> I'm optimistically hoping https://gerrit.ovirt.org/#/c/85195/ will fix
>> it.
>> Not sure.
>>
>
> Keeps failing with:
> Operation Failed: [Cannot deactivate Storage. The relevant Storage
> Domain's status is Maintenance.]
>
> Which is strange:
> 1. I do check if the SD is in Maint. mode before trying to deactive and
> the test is supposed to be skipped if it is.
>

I've added a comment in the patch,
I suspect it is related to the fact that we fetch the domain from
storage_domains and not attached_storage_domains

> 2. Why can't I deactive a SD when in Maint. mode?
>

I assume that the engine fails since that means that the storage domain is
not connected to the Host and the execute phase of maintenance performs
operations like update ovf store which depend on the storage domain to be
connected.

>
> Probably missing something here.
> Y.
>
> Y.
>>
>>
>>>
>>> Allon, can you please assign someone to help fix this test?
>>> Please let me know if you need help from me.
>>>
>>> Thanks!
>>> Dafna
>>>
>>>
Thanks Dafna, besides the fix which should be done in the OST, here is an
old old bug which might also become useful:
     https://bugzilla.redhat.com/879248
<https://bugzilla.redhat.com/show_bug.cgi?id=879248>

>
>>>
>>> On 12/07/2017 11:59 AM, Yaniv Kaul wrote:
>>>
>>>
>>>
>>> On Thu, Dec 7, 2017 at 1:30 PM, Eyal Shenitzky <eshenitz at redhat.com>
>>> wrote:
>>>
>>>> I think that the maybe the QE can share their methods on how to avoid
>>>> those issues.
>>>> From what I remember, before deactivating storage domain they make sure
>>>> that there are no running tasks related to
>>>> the storage domain.
>>>>
>>>
>>> Looks like an easy fix is to wrap it with try, except sdk4.Error and let
>>> it sit within the testlib.assert_true_within_short() loop.
>>> Y.
>>>
>>>
>>>>
>>>> On Thu, Dec 7, 2017 at 1:22 PM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 7, 2017 at 1:12 PM, Dafna Ron <dron at redhat.com> wrote:
>>>>>
>>>>>> Maor, I either need to get new glasses or a magnifier glass to read
>>>>>> what you wrote :-P
>>>>>> when you say running tasks - these are actually running tasks that
>>>>>> may be running because of other tests in ost - correct? wouldn't killing or
>>>>>> blocking those can cause other tests to fail?
>>>>>>
>>>>>
>>>>> It might well be the OVF update. How can we, from the API, wait for
>>>>> those tasks to complete? Or should we catch exception and retry?
>>>>> Y.
>>>>>
>>>>>
>>>>>>
>>>>>> On 12/07/2017 11:06 AM, Maor Lipchuk wrote:
>>>>>>
>>>>>> CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS is a known issue, the problem is
>>>>>> that we might have tasks which will start running internally using
>>>>>> scheduling (like OVF_UPDATE) and we can't really know how much time every
>>>>>> task will take until it will end.
>>>>>>
>>>>>> Even if we check that there are no running tasks it will not
>>>>>> guarantee that no task will start until you deactivate the storage
>>>>>> domain.
>>>>>>
>>>>>> I think that the best solution for it is an engine support to cancel
>>>>>> running tasks or block tasks from running.
>>>>>>
>>>>>> On Thu, Dec 7, 2017 at 12:14 PM, Dafna Ron <dron at redhat.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We had a failure on master basic suite for test
>>>>>>> 007_sd_reattach.deactivate_storage_domain.
>>>>>>>
>>>>>>> The failure was that we failed to deactivate domain due to running
>>>>>>> tasks.
>>>>>>>
>>>>>>> It does not seem to be related to the patch it was testing and I
>>>>>>> think that the test itself needs to be modified to check there are no
>>>>>>> running tasks.
>>>>>>>
>>>>>>> Is there perhaps a way to query if there are running tasks before
>>>>>>> running the command? can you please take a look at the test on OST?
>>>>>>>
>>>>>>> *Link and headline of suspected patches: Not related to error*
>>>>>>>
>>>>>>> *Link to Job:
>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/
>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/>*
>>>>>>>
>>>>>>>
>>>>>>> * Link to all logs:
>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/artifact/
>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/artifact/>
>>>>>>> (Relevant) error snippet from the log: <error> 2017-12-06 20:13:23,166-05
>>>>>>> WARN
>>>>>>> [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand]
>>>>>>> (default task-7) [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] Validation of
>>>>>>> action 'DeactivateStorageDomainWithOvfUpdate' fa iled for user
>>>>>>> admin at internal-authz. Reasons:
>>>>>>> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ERROR_CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS
>>>>>>> 2017-12-06 20:13:23,167-05 INFO
>>>>>>> [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand]
>>>>>>> (default task-7) [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] Lock freed to
>>>>>>> object 'EngineLock:{exclusiveLocks='[ea2fd992-8a
>>>>>>> a4-44fe-aa43-e96754a975ba=STORAGE]',
>>>>>>> sharedLocks='[5e0a0183-0e25-4f43-b5b0-0cfb5510248e=POOL]'}' 2017-12-06
>>>>>>> 20:13:23,172-05 DEBUG
>>>>>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>>>>>>> (default task-7) [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] method: runAction,
>>>>>>> params: [DeactivateStorageDomainWithOvfUpdate, DeactivateSto
>>>>>>> rageDomainWithOvfUpdateParameters:{commandId='630c28e1-41ab-43db-9755-a2bb870dbcb3',
>>>>>>> user='null', commandType='Unknown'}], timeElapsed: 65ms 2017-12-06
>>>>>>> 20:13:23,176-05 ERROR
>>>>>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
>>>>>>> task-7) [] Operation Failed: [Cannot deactivate Storage while there are
>>>>>>> running tasks on this Storage. -Please wait until tasks will finish and try
>>>>>>> again.] *<error>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Eyal Shenitzky
>>>>
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20171210/b1ea60c1/attachment-0001.html>