[ovirt-devel] Re: ovirt-system-tests_compat-4.2-suite-master failing for 3 weeks?

Hi, There is a different issue now (jenkins job [1]): Fault reason is "Operation Failed". Fault detail is "[Cannot attach Storage. Storage Domain format V5 is illegal.]". HTTP response code is 409. <http://goog_2029942553> Can someone take a look? *Error from engine.log:* 2019-05-01 02:00:54,015-04 DEBUG [org.ovirt.engine.core.bll.Backend] (default task-3) [] Executing command AttachStorageDomainToPool for user admin@internal-authz. 2019-05-01 02:00:54,032-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Checking whether user 'caa94e16-6bd5-11e9-b8ca-5452c0a8ca02' or one of the groups he is member of, have the following permissions: ID: bb8f0934-0325-474d-9f62-f2983972a16b Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: b2c7ddfb-b9ba-4218-b3f1-4b952031de8f Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2019-05-01 02:00:54,032-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Found permission 'caa959ec-6bd5-11e9-b8cb-5452c0a8ca02' for user when running 'AttachStorageDomainToPool', on 'Storage' with id 'bb8f0934-0325-474d-9f62-f2983972a16b' 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Found permission 'caa959ec-6bd5-11e9-b8cb-5452c0a8ca02' for user when running 'AttachStorageDomainToPool', on 'Data Center' with id 'b2c7ddfb-b9ba-4218-b3f1-4b952031de8f' 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Before acquiring lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' 2019-05-01 02:00:54,033-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Success acquiring lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' 2019-05-01 02:00:54,033-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' 2019-05-01 02:00:54,039-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] method: getAllForStoragePoolAndStatus, params: [b2c7ddfb-b9ba-4218-b3f1-4b952031de8f, Up], timeElapsed: 6ms 2019-05-01 02:00:54,040-04 WARN [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Validation of action 'AttachStorageDomainToPool' failed for user admin@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ATTACH,ACTION_TYPE_FAILED_STORAGE_DOMAIN_FORMAT_ILLEGAL,$storageFormat V5 2019-05-01 02:00:54,041-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Before releasing a lock 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' 2019-05-01 02:00:54,041-04 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] The exclusive lock for key 'bb8f0934-0325-474d-9f62-f2983972a16bSTORAGE' is released and lock is removed from map 2019-05-01 02:00:54,041-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] Lock freed to object 'EngineLock:{exclusiveLocks='[bb8f0934-0325-474d-9f62-f2983972a16b=STORAGE]', sharedLocks=''}' 2019-05-01 02:00:54,045-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-3) [13e92697-ec26-478b-a29e-c652b6006e63] method: runAction, params: [AttachStorageDomainToPool, AttachStorageDomainToPoolParameters:{commandId='07b837fe-6d86-44f6-81b6-9ddcc4f0cc49', user='null', commandType='Unknown'}], timeElapsed: 30ms 2019-05-01 02:00:54,049-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-3) [] Operation Failed: [Cannot attach Storage. Storage Domain format V5 is illegal.] 2019-05-01 02:00:54,053-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-3) [] Entered SsoRestApiAuthFilter [1] http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/561/ <http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/5... Thanks, Galit On Mon, Apr 29, 2019 at 5:26 PM Fred Rolland <frolland(a)redhat.com&gt; wrote: > Galit is investigating the problem. > The issue is that before adding the storage domain, we check that the > hosts are on UP state. > However, it is not always the case and the root cause is unclear. > > On Mon, Apr 29, 2019 at 3:16 PM Doron Fediuck <dfediuck(a)redhat.com&gt; > wrote: > >> >> https://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/... >> >> The current failure is on adding a storage domain. Did anyone review it? >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QRMG6BISWXU... >> > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YIGTLVYSA5V... > -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)redhat.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/T7Y2WW4OBBS...