[ovirt-devel] Re: [ OST Failure Report ] [ oVirt Master (ovirt-engine-nodejs-modules) ] [ 27-02-2019 ] [ 002_bootstrap.add_vm2_lease ]

On Tue, Mar 5, 2019, 13:27 Eyal Shenitzky <eshenitz(a)redhat.com wrote: > > > On Tue, Mar 5, 2019 at 12:58 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> Tal, >> >> I see the bug is in post but the patch was not merged yet: >> https://gerrit.ovirt.org/#/c/98191/ >> >> can you tell me when will we merge the patch? as I rather not add >> SkipTest if this will be merged soon, >> >> thanks, >> Dafna >> >> >> On Mon, Mar 4, 2019 at 10:42 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> As I had another failure of this today I will be disabling this test >>> until issue is resolved (https://bugzilla.redhat.com/1684267) >>> >>> Thanks, >>> Dafna >>> >>> >>> On Thu, Feb 28, 2019 at 8:48 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>> >>>> On Thu, Feb 28, 2019 at 11:52 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> Hi, >>>>> >>>>> We have a failure on the project in basic suite, master branch. The >>>>> recent failure was in patch: >>>>> https://gerrit.ovirt.org/#/c/98087/1 - Add pre-seed for ovirt-web-ui >>>>> >>>>> CQ is pointing at the below as the root cause (which was merged a >>>>> while back): >>>>> https://gerrit.ovirt.org/#/c/97491/ - Add pre-seed for ovirt-web-ui >>>>> >>>>> Can you please check the issue as it seems both patches are changing >>>>> the same thing and the project seem to be broken since >>>>> https://gerrit.ovirt.org/#/c/97491/3 >>>>> >>>>> Latest failure: >>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/ >>>>> >>>>> Logs: >>>>> >>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/arti... >>>>> >>>>> errors from logs: >>>>> Engine: >>>>> >>>>> 2019-02-27 13:37:28,479-05 ERROR >>>>> [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25] >>>>> Transaction rolled-back for command >>>>> 'org.ovirt.engine.core.bll.UpdateVmCommand'. >>>>> 2019-02-27 13:37:28,483-05 ERROR >>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> (default task-1) [74283e25] EVENT_ID: USER_FAILED_UPDATE_VM(58), Failed to >>>>> update VM vm2 (User: admin@inter >>>>> nal-authz). >>>>> 2019-02-27 13:37:28,485-05 INFO >>>>> [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25] >>>>> Lock freed to object 'EngineLock:{exclusiveLocks='[vm2=VM_NAME]', >>>>> sharedLocks='[3500eb82-e5e2-4e24-b41c-ea >>>>> 02d9f6adee=VM]'}' >>>>> 2019-02-27 13:37:28,485-05 DEBUG >>>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>>>> (default task-1) [74283e25] method: runAction, params: [UpdateVm, >>>>> VmManagementParametersBase:{commandId='340 >>>>> 59769-05b9-429e-8356-f6b9b9953f55', user='admin', >>>>> commandType='UpdateVm', vmId='3500eb82-e5e2-4e24-b41c-ea02d9f6adee'}], >>>>> timeElapsed: 6618ms >>>>> 2019-02-27 13:37:28,486-05 ERROR >>>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >>>>> task-1) [] Operation Failed: [] >>>>> 2019-02-27 13:37:28,579-05 DEBUG >>>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] method: get, params: >>>>> [e29c0ba1-464c-4eb4-a8f2-c6933d9 >>>>> 9969a], timeElapsed: 3ms >>>>> >>>>> >>>>> vdsm: >>>>> >>>>> 2019-02-27 13:37:21,987-0500 INFO (jsonrpc/1) [vdsm.api] FINISH >>>>> lease_info error=No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee >>>>> from=::ffff:192.168.201.4,43920, >>>>> flow_id=117dec74-ad59-4b12-8148-b2c130337c10, >>>>> task_id=9c297d41-0aa7-4c74-b268-b710e666bc6c (api:52) >>>>> 2019-02-27 13:37:21,988-0500 ERROR (jsonrpc/1) >>>>> [storage.TaskManager.Task] (Task='9c297d41-0aa7-4c74-b268-b710e666bc6c') >>>>> Unexpected error (task:875) >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >>>>> 882, in _run >>>>> return fn(*args, **kargs) >>>>> File "<string>", line 2, in lease_info >>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line >>>>> 50, in method >>>>> ret = func(*args, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line >>>>> 3702, in lease_info >>>>> info = dom.lease_info(lease.lease_id) >>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line >>>>> 674, in lease_info >>>>> return vol.lookup(lease_id) >>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", >>>>> line 553, in lookup >>>>> raise NoSuchLease(lease_id) >>>>> NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee >>>>> >>>> >>>> This is not an error of vdsm. Someone asked for information about a >>>> non-existing lease, >>>> and vdsm fail the request with the expected error: >>>> >>>> NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee >>>> >>>> But since the error is not a public error, vdsm logger handle it as >>>> unexpected error and >>>> create lot of noise in vdsm log. >>>> >>>> On engine side, this creates even worse noise, as we can see here: >>>> >>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/arti... >>>> (look for 2019-02-27 13:37:21,996-05) >>>> >>>> All these noise could be avoided if vdsm was using a proper public >>>> error. >>>> I filed https://bugzilla.redhat.com/1684267 >>>> >>>> But, this error is not related to the actual failing test, this is >>>> just noise hiding the real problem. >>>> >>>> > Hi, > > As Nir already said, This log is just noise, this is not the real problem. > > I think that we already talk about this issue but I can't find the thread. > > This lease failure in the log is expected and this is not an error please > ignore this in the next failure analysis. > > From the log I can see the following error: > > {"jsonrpc": "2.0", "id": "6a42a9cf-e220-4916-ba0e-db2182978a5d", "error": {"message": "Cannot find master domain: u'spUUID=717575a9-7818-460d-ba3a-d5bdd8ef9ed3, msdUUID=34311cc1-c4d2-4cfe-88b5-dd5ad72261d3'", "code": 304}}� > 2019-02-27 13:35:14,175-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "6a42a9cf-e220-4916-ba0e-db2182978a5d", "error": {"message": "Cannot find master domain: u'spUUID=717575a9-7818-460d-ba3a-d5bdd8ef9ed3, msdUUID=34311cc1-c4d2-4cfe-88b5-dd5ad72261d3'", "code": 304}} > 2019-02-27 13:35:14,182-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-23) [73da8d8b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-basic-suite-master-host-1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=717575a9-7818-460d-ba3a-d5bdd8ef9ed3, msdUUID=34311cc1-c4d2-4cfe-88b5-dd5ad72261d3' > 2019-02-27 13:35:14,182-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [73da8d8b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=717575a9-7818-460d-ba3a-d5bdd8ef9ed3, msdUUID=34311cc1-c4d2-4cfe-88b5-dd5ad72261d3']]' > 2019-02-27 13:35:14,182-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [73da8d8b] HostName = lago-basic-suite-master-host-1 > > This failure occured before the lease info was requested (and failed as > expected). > > Therefore the fix for - https://bugzilla.redhat.com/1684267 will not > solve anything that related to the failure. > I agree with Eyal. Please find the owner of the failing test. Every test should have an owner. If there is no owner the test should be deleted. > T hanks for reporting this. >>>> >>>> Nir >>>> >>> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PLQMYYARPAY... >> > > > -- > Regards, > Eyal Shenitzky >