[ovirt-devel] Re: ovirt-system-tests_compat-4.2-suite-master failing for 3 weeks?

On Tue, May 7, 2019 at 7:16 AM Galit Rosenthal <grosenth(a)redhat.com&gt; wrote: > By default, the SD type is set to V5. > and in the test, there was a condition when version < 4.1 was set to V3. > We added a fix on 4.1<=versions <=4.2 it will set to V4 > Sounds good. Can you link to the patches fixing this? On Tue, May 7, 2019 at 2:35 AM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > >> On Mon, May 6, 2019 at 6:18 PM Galit Rosenthal <grosenth(a)redhat.com&gt; >> wrote: >> >>> >>> The issue is solved now for both compat 4.2 and for compat 4.1 >>> the suites passed successfully >>> >> >> Great! >> >> Can we have more details? what was the root cause? how it was fixed? >> >> On Thu, May 2, 2019 at 11:15 AM Shani Leviim <sleviim(a)redhat.com&gt; wrote: >>> >>>> It seems that the storage domain's format wasn't properly set by the >>>> cluster's level (although the cluster level is 4.1 or 4.2, the storage >>>> format was set as v5). >>>> Galit and I are working on it. >>>> >>>> >>>> *Regards,* >>>> >>>> *Shani Leviim* >>>> >>>> >>>> On Wed, May 1, 2019 at 5:34 PM Tal Nisan <tnisan(a)redhat.com&gt; wrote: >>>> >>>>> Shani, please have a look >>>>> >>>>> On Wed, May 1, 2019 at 2:34 PM Galit Rosenthal <grosenth(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> There is a different issue now (jenkins job [1]): >>>>>> >>>>>> Fault reason is "Operation Failed". Fault detail is "[Cannot attach Storage. Storage Domain format V5 is illegal.]". HTTP response code is 409. >>>>>> >>>>>> >>>>>> <http://goog_2029942553> >>>>>> Can someone take a look? >>>>>> >>>>>> *Error from engine.log:* >>>>>> >>>>>> 2019-05-01 02:00:54,015-04 DEBUG [org.ovirt.engine.core.bll.Backend] (default task-3) [] Executing command AttachStorageDomainToPool for user admin@internal-authz. >>>>>> 2019-05-01 02:00:54,032-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Checking whether user 'caa94e16-6bd5-11e9-b8ca-5452c0a8ca02' or one of the groups he is member of, have the following permissions: ID: bb8f0934-0325-474d-9f62-f2983972a16b Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: b2c7ddfb-b9ba-4218-b3f1-4b952031de8f Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >>>>>> 2019-05-01 02:00:54,032-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Found permission 'caa959ec-6bd5-11e9-b8cb-5452c0a8ca02' for user when running 'AttachStorageDomainToPool', on 'Storage' with id 'bb8f0934-0325-474d-9f62-f2983972a16b' >>>>>> 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Found permission 'caa959ec-6bd5-11e9-b8cb-5452c0a8ca02' for user when running 'AttachStorageDomainToPool', on 'Data Center' with id 'b2c7ddfb-b9ba-4218-b3f1-4b952031de8f' >>>>>> 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Before acquiring lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' >>>>>> 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Success acquiring lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' >>>>>> 2019-05-01 02:00:54,033-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' >>>>>> 2019-05-01 02:00:54,039-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] method: getAllForStoragePoolAndStatus, params: [b2c7ddfb-b9ba-4218-b3f1-4b952031de8f, Up], timeElapsed: 6ms >>>>>> 2019-05-01 02:00:54,040-04 WARN [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Validation of action 'AttachStorageDomainToPool' failed for user admin@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ATTACH,ACTION_TYPE_FAILED_STORAGE_DOMAIN_FORMAT_ILLEGAL,$storageFormat V5 >>>>>> 2019-05-01 02:00:54,041-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Before releasing a lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' >>>>>> 2019-05-01 02:00:54,041-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] The exclusive lock for key 'bb8f0934-0325-474d-9f62-f2983972a16bSTORAGE' is released and lock is removed from map >>>>>> 2019-05-01 02:00:54,041-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Lock freed to object 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' >>>>>> 2019-05-01 02:00:54,045-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] method: runAction, params: [AttachStorageDomainToPool, AttachStorageDomainToPoolParameters:{commandId='07b837fe-6d86-44f6-81b6-9ddcc4f0cc49', user='null', commandType='Unknown'}], timeElapsed: 30ms >>>>>> 2019-05-01 02:00:54,049-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-3) [] Operation Failed: [Cannot attach Storage. Storage Domain format V5 is illegal.] >>>>>> 2019-05-01 02:00:54,053-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-3) [] Entered SsoRestApiAuthFilter >>>>>> >>>>>> >>>>>> [1] >>>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/561/ >>>>>> <http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/5... >>>>>> >>>>>> Thanks, >>>>>> Galit >>>>>> >>>>>> >>>>>> On Mon, Apr 29, 2019 at 5:26 PM Fred Rolland <frolland(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Galit is investigating the problem. >>>>>>> The issue is that before adding the storage domain, we check that >>>>>>> the hosts are on UP state. >>>>>>> However, it is not always the case and the root cause is unclear. >>>>>>> >>>>>>> On Mon, Apr 29, 2019 at 3:16 PM Doron Fediuck <dfediuck(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/... >>>>>>>> >>>>>>>> The current failure is on adding a storage domain. Did anyone >>>>>>>> review it? >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QRMG6BISWXU... >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: >>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: >>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YIGTLVYSA5V... >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> GALIT ROSENTHAL >>>>>> >>>>>> SOFTWARE ENGINEER >>>>>> >>>>>> Red Hat >>>>>> >>>>>> <https://www.redhat.com/> >>>>>> >>>>>> galit(a)redhat.com T: 972-9-7692230 >>>>>> <https://red.ht/sig> >>>>>> _______________________________________________ >>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/T7Y2WW4OBBS... >>>>>> >>>>> >>> >>> -- >>> >>> GALIT ROSENTHAL >>> >>> SOFTWARE ENGINEER >>> >>> Red Hat >>> >>> <https://www.redhat.com/> >>> >>> galit(a)redhat.com T: 972-9-7692230 >>> <https://red.ht/sig> >>> _______________________________________________ >>> Devel mailing list -- devel(a)ovirt.org >>> To unsubscribe send an email to devel-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CNEA3DP6TJN... >>> >> > > -- > > GALIT ROSENTHAL > > SOFTWARE ENGINEER > > Red Hat > > <https://www.redhat.com/> > > galit(a)redhat.com T: 972-9-7692230 > <https://red.ht/sig> >