Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]

On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini(a)redhat.com&gt; wrote: > That was more a less the my guess from the brief look I took at the > stacktrace. > > I don't want to +2 a patch in the network area, but I'd suggest merging > this patch (which seems right regardless of the recent failure), and see if > it solves the OST issue. > > OST will run with the network patch reverted, so I need to bring it into the Q again (revert the reverted :)) On my side, with the MacPool changes it works, imported vms from domains, started vm, basic editing. To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm not mistaken here it should be available _before_ the Backend in order to keep compensation running? in case we have a complete action that needs to during compensation. Maybe this is a reason for MacPool to NOT depend on Backend, so compensation would complete? Needs a second look. >

On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan(a)redhat.com&gt; wrote: > >> >> >> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan(a)redhat.com&gt; wrote: >> >>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/ so the >>> revert can be reverted. +Martin Mucha <mmucha(a)redhat.com&gt; want to test >>> it? It is working on my end. >>> >>> sorry its this https://gerrit.ovirt.org/c/81637/2 >> >>> >>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>> >>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor(a)redhat.com&gt; >>>> wrote: >>>> > >>>> > Hello all, >>>> > this same error still causes OST to fail. >>>> > >>>> >>>> We are aware of it, but do not understand it at all. >>>> >>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/ >>>> would be merged once approved by CI. >>>> >>>> > >>>> > Jobs: >>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466 >>>> > >>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471 >>>> > >>>> > Logs: >>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue- >>>> tester/2466/artifact/exported-artifacts/basic-suit-master- >>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log >>>> > >>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue- >>>> tester/2471/artifact/exported-artifacts/basic-suit-master- >>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log >>>> > >>>> > >>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>> >> >>>> >> >>>> >> >>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg <danken(a)redhat.com&gt; >>>> wrote: >>>> >>> >>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg <danken(a)redhat.com&gt; >>>> wrote: >>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin >>>> >>> > <ederevea(a)redhat.com&gt; wrote: >>>> >>> >> Test failed: [ import_template_from_glance ] >>>> >>> >> >>>> >>> >> Link to suspected patches: >>>> >>> >> https://gerrit.ovirt.org/#/c/80450/ >>>> >>> > >>>> >>> > Martin, this is "core: when initializing MacPool also register >>>> in it >>>> >>> > nics in snapshots" >>>> >>> > and the error seems somewhat related to it >>>> >>> >>>> >>> Gil, can you tell us which is the last Engine patch that passed >>>> OST? I >>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm >>>> guessing >>>> >>> that "ci please build" does not work on merged patches. >>>> >> >>>> >> >>>> >> Not sure I follow, the latest working engine should be deployed to >>>> tested repo, >>>> >> So if you'll run the manual job w/o any custom repos it should work >>>> ( if it doesn't, then it means something slipped in the tested repo and we >>>> should investigate ). >>>> >> >>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs >>>> on the job, so you can still try it: >>>> >> >>>> >> Patch Set 26: >>>> >> Build Successful >>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build- >>>> artifacts-on-demand-fc25-x86_64/377/ : SUCCESS >>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build- >>>> artifacts-on-demand-el7-x86_64/428/ : SUCCESS >>>> >> >>>> >> >>>> >>> >>>> >>> >>>> >>> > >>>> >>> >> >>>> >>> >> Link to Job: >>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue- >>>> tester/2397/ >>>> >>> >> >>>> >>> >> Link to logs: >>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue- >>>> tester/2397/artifact/exported-artifacts/basic-suit-master- >>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log >>>> >>> >> >>>> >>> >> Error snippet from log: >>>> >>> >> >>>> >>> >> <error> >>>> >>> >> >>>> >>> >> 2017-09-07 08:21:48,657-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Lock Acquired >>>> >>> >> to object >>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0- >>>> 9744309ca8c1=TEMPLATE, >>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', >>>> sharedLocks='[]'}' >>>> >>> >> 2017-09-07 08:21:48,675-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Running >>>> >>> >> command: AddVmTemplateCommand internal: true. Entities affected >>>> : ID: >>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction >>>> group >>>> >>> >> CREATE_TEMPLATE with role type USER >>>> >>> >> 2017-09-07 08:21:48,695-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.storage.disk. >>>> CreateAllTemplateDisksCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Running >>>> >>> >> command: CreateAllTemplateDisksCommand internal: true. >>>> >>> >> 2017-09-07 08:21:48,722-04 INFO >>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> transaction >>>> >>> >> rolled back >>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Command >>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null >>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Exception: >>>> >>> >> java.lang.NullPointerException >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings. >>>> getVramMultiplier(VgamemVideoSettings.java:63) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings. >>>> getQxlVideoDeviceSettings(VgamemVideoSettings.java:40) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings. >>>> getVideoDeviceSettings(VideoDeviceSettings.java:34) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings. >>>> getVideoDeviceSpecParams(VideoDeviceSettings.java:49) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils. >>>> getVideoDeviceSpecParams(VmDeviceUtils.java:586) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils. >>>> copyVmDevices(VmDeviceUtils.java:1435) >>>> >>> >> [bll.jar:] >>>> >>> >> at >>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils. >>>> copyVmDevices(VmDeviceUtils.java:1559) >>>> >>> >> [bll.jar:] >>>> >>> >> >>>> >>> >> ... >>>> >>> >> >>>> >>> >> 2017-09-07 08:21:48,739-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Command >>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating >>>> NEW_ENTITY_ID of >>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate; >>>> snapshot: >>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1. >>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR >>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >>>> AuditLogDirector] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> EVENT_ID: >>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template >>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template. >>>> >>> >> 2017-09-07 08:21:48,834-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Lock freed to >>>> >>> >> object >>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0- >>>> 9744309ca8c1=TEMPLATE, >>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', >>>> sharedLocks='[]'}' >>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG >>>> >>> >> [org.ovirt.engine.core.dal.dbbroker. >>>> CustomSQLErrorCodeSQLExceptionTranslator] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Translating >>>> >>> >> SQLException with SQL state '23503', error code '0', message >>>> [ERROR: insert >>>> >>> >> or update on table "disk_vm_element" violates foreign key >>>> constraint >>>> >>> >> "fk_disk_vm_element_vm_static" >>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) >>>> is not present >>>> >>> >> in table "vm_static". >>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>> >>> >> disk_id, >>>> >>> >> vm_id, >>>> >>> >> is_boot, >>>> >>> >> pass_discard, >>>> >>> >> disk_interface, >>>> >>> >> is_using_scsi_reservation) >>>> >>> >> VALUES ( >>>> >>> >> v_disk_id, >>>> >>> >> v_vm_id, >>>> >>> >> v_is_boot, >>>> >>> >> v_pass_discard, >>>> >>> >> v_disk_interface, >>>> >>> >> v_is_using_scsi_reservation)" >>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid, >>>> boolean,boolean,character >>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call >>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task >>>> [CallableStatementCallback] >>>> >>> >> 2017-09-07 08:21:48,850-04 INFO >>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> transaction >>>> >>> >> rolled back >>>> >>> >> 2017-09-07 08:21:48,877-04 INFO >>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >>>> AuditLogDirector] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> EVENT_ID: >>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User >>>> >>> >> admin@internal-authz successfully imported image >>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template >>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi. >>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR >>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>> Failed >>>> >>> >> invoking callback end method 'onSucceeded' for command >>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception >>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?, >>>> ?, ?, ?, ?, >>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element" >>>> violates foreign >>>> >>> >> key constraint "fk_disk_vm_element_vm_static" >>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) >>>> is not present >>>> >>> >> in table "vm_static". >>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>> >>> >> disk_id, >>>> >>> >> vm_id, >>>> >>> >> is_boot, >>>> >>> >> pass_discard, >>>> >>> >> disk_interface, >>>> >>> >> is_using_scsi_reservation) >>>> >>> >> VALUES ( >>>> >>> >> v_disk_id, >>>> >>> >> v_vm_id, >>>> >>> >> v_is_boot, >>>> >>> >> v_pass_discard, >>>> >>> >> v_disk_interface, >>>> >>> >> v_is_using_scsi_reservation)" >>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid, >>>> boolean,boolean,character >>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is >>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on >>>> table >>>> >>> >> "disk_vm_element" violates foreign key constraint >>>> >>> >> "fk_disk_vm_element_vm_static" >>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) >>>> is not present >>>> >>> >> in table "vm_static". >>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>> >>> >> disk_id, >>>> >>> >> vm_id, >>>> >>> >> is_boot, >>>> >>> >> pass_discard, >>>> >>> >> disk_interface, >>>> >>> >> is_using_scsi_reservation) >>>> >>> >> VALUES ( >>>> >>> >> v_disk_id, >>>> >>> >> v_vm_id, >>>> >>> >> v_is_boot, >>>> >>> >> v_pass_discard, >>>> >>> >> v_disk_interface, >>>> >>> >> v_is_using_scsi_reservation)" >>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid, >>>> boolean,boolean,character >>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is >>>> marked for end >>>> >>> >> method retries >>>> >>> >> 2017-09-07 08:21:49,887-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCa >>>> llback] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb] >>>> Command >>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' >>>> execution didn't >>>> >>> >> complete, not proceeding to perform the next operation >>>> >>> >> 2017-09-07 08:21:49,887-04 INFO >>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCa >>>> llback] >>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb] >>>> Command >>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' >>>> child commands >>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were >>>> completed, status >>>> >>> >> 'FAILED' >>>> >>> >> >>>> >>> >> >>>> >>> >> </error> >>>> >>> >> >>>> >>> >> -- >>>> >>> >> Regards, >>>> >>> >> Evgheni Dereveanchin >>>> >>> >> >>>> >>> >> _______________________________________________ >>>> >>> >> Devel mailing list >>>> >>> >> Devel(a)ovirt.org >>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> _______________________________________________ >>>> >>> Devel mailing list >>>> >>> Devel(a)ovirt.org >>>> >>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> >>>> >> Eyal edri >>>> >> >>>> >> >>>> >> ASSOCIATE MANAGER >>>> >> >>>> >> RHV DevOps >>>> >> >>>> >> EMEA VIRTUALIZATION R&D >>>> >> >>>> >> >>>> >> Red Hat EMEA >>>> >> >>>> >> TRIED. TESTED. TRUSTED. >>>> >> phone: +972-9-7692018 <+972%209-769-2018> >>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >> >>>> >> _______________________________________________ >>>> >> Devel mailing list >>>> >> Devel(a)ovirt.org >>>> >> http://lists.ovirt.org/mailman/listinfo/devel >>>> > >>>> > >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel