Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]

On Mon, Sep 11, 2017 at 10:47 PM, Roy Golan <rgolan(a)redhat.com&gt; wrote: > > > On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini(a)redhat.com&gt; wrote: >> >> That was more a less the my guess from the brief look I took at the >> stacktrace. >> >> I don't want to +2 a patch in the network area, but I'd suggest merging >> this patch (which seems right regardless of the recent failure), and see if >> it solves the OST issue. >> > OST will run with the network patch reverted, so I need to bring it into the > Q again (revert the reverted :)) Even I can do that. I've rebase your patches on top of current master, and reintroduced Mucha's patch (https://gerrit.ovirt.org/#/c/81639/). If OST passes, I'll ask you to backport your injection fixes - we need to solve the MAC collision bug in ovirt-4.1, too. http://jenkins.ovirt.org/job/ovirt-system-tests_manual/1129/ is green

> > On my side, with the MacPool changes it works, imported vms from domains, > started vm, basic editing. > > To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm > not mistaken here it should be available _before_ the Backend in order to > keep compensation running? in case we have a complete action that needs to > during compensation. > Maybe this is a reason for MacPool to NOT depend on Backend, so compensation > would complete? > Needs a second look. > > >> On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan(a)redhat.com&gt; wrote: >>> >>> >>> >>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan(a)redhat.com&gt; wrote: >>>> >>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/ so the >>>> revert can be reverted. +Martin Mucha want to test it? It is working on my >>>> end. >>>> >>> sorry its this https://gerrit.ovirt.org/c/81637/2 >>>> >>>> >>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>>>> >>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor(a)redhat.com&gt; wrote: >>>>> > >>>>> > Hello all, >>>>> > this same error still causes OST to fail. >>>>> > >>>>> >>>>> We are aware of it, but do not understand it at all. >>>>> >>>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/ >>>>> would be merged once approved by CI. >>>>> >>>>> > >>>>> > Jobs: >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466 >>>>> > >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471 >>>>> > >>>>> > Logs: >>>>> > >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466/artifa... >>>>> > >>>>> > >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471/artifa... >>>>> > >>>>> > >>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg < danken(a)redhat.com&gt; >>>>> >> wrote: >>>>> >>> >>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg < danken(a)redhat.com&gt; >>>>> >>> wrote: >>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin >>>>> >>> > <ederevea(a)redhat.com&gt; wrote: >>>>> >>> >> Test failed: [ import_template_from_glance ] >>>>> >>> >> >>>>> >>> >> Link to suspected patches: >>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/ >>>>> >>> > >>>>> >>> > Martin, this is "core: when initializing MacPool also register in >>>>> >>> > it >>>>> >>> > nics in snapshots" >>>>> >>> > and the error seems somewhat related to it >>>>> >>> >>>>> >>> Gil, can you tell us which is the last Engine patch that passed >>>>> >>> OST? I >>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm guessing >>>>> >>> that "ci please build" does not work on merged patches. >>>>> >> >>>>> >> >>>>> >> Not sure I follow, the latest working engine should be deployed to >>>>> >> tested repo, >>>>> >> So if you'll run the manual job w/o any custom repos it should work >>>>> >> ( if it doesn't, then it means something slipped in the tested repo and we >>>>> >> should investigate ). >>>>> >> >>>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs on >>>>> >> the job, so you can still try it: >>>>> >> >>>>> >> Patch Set 26: >>>>> >> Build Successful >>>>> >> >>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-deman... >>>>> >> : SUCCESS >>>>> >> >>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-deman... >>>>> >> : SUCCESS >>>>> >> >>>>> >> >>>>> >>> >>>>> >>> >>>>> >>> > >>>>> >>> >> >>>>> >>> >> Link to Job: >>>>> >>> >> >>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/ >>>>> >>> >> >>>>> >>> >> Link to logs: >>>>> >>> >> >>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/artifa... >>>>> >>> >> >>>>> >>> >> Error snippet from log: >>>>> >>> >> >>>>> >>> >> <error> >>>>> >>> >> >>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Lock Acquired >>>>> >>> >> to object >>>>> >>> >> >>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE, >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', >>>>> >>> >> sharedLocks='[]'}' >>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Running >>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities affected >>>>> >>> >> : ID: >>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction >>>>> >>> >> group >>>>> >>> >> CREATE_TEMPLATE with role type USER >>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO >>>>> >>> >> >>>>> >>> >> [org.ovirt.engine.core.bll.storage.disk.CreateAllTemplateDisksCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Running >>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true. >>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> transaction >>>>> >>> >> rolled back >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Command >>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Exception: >>>>> >>> >> java.lang.NullPointerException >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getVramMultiplier(VgamemVideoSettings.java:63) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getQxlVideoDeviceSettings(VgamemVideoSettings.java:40) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSettings(VideoDeviceSettings.java:34) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSpecParams(VideoDeviceSettings.java:49) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.getVideoDeviceSpecParams(VmDeviceUtils.java:586) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1435) >>>>> >>> >> [bll.jar:] >>>>> >>> >> at >>>>> >>> >> >>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1559) >>>>> >>> >> [bll.jar:] >>>>> >>> >> >>>>> >>> >> ... >>>>> >>> >> >>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Command >>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating >>>>> >>> >> NEW_ENTITY_ID of >>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate; >>>>> >>> >> snapshot: >>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1. >>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR >>>>> >>> >> >>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> EVENT_ID: >>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template. >>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Lock freed to >>>>> >>> >> object >>>>> >>> >> >>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE, >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', >>>>> >>> >> sharedLocks='[]'}' >>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG >>>>> >>> >> >>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.CustomSQLErrorCodeSQLExceptionTranslator] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Translating >>>>> >>> >> SQLException with SQL state '23503', error code '0', message >>>>> >>> >> [ERROR: insert >>>>> >>> >> or update on table "disk_vm_element" violates foreign key >>>>> >>> >> constraint >>>>> >>> >> "fk_disk_vm_element_vm_static" >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is >>>>> >>> >> not present >>>>> >>> >> in table "vm_static". >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>>> >>> >> disk_id, >>>>> >>> >> vm_id, >>>>> >>> >> is_boot, >>>>> >>> >> pass_discard, >>>>> >>> >> disk_interface, >>>>> >>> >> is_using_scsi_reservation) >>>>> >>> >> VALUES ( >>>>> >>> >> v_disk_id, >>>>> >>> >> v_vm_id, >>>>> >>> >> v_is_boot, >>>>> >>> >> v_pass_discard, >>>>> >>> >> v_disk_interface, >>>>> >>> >> v_is_using_scsi_reservation)" >>>>> >>> >> PL/pgSQL function >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character >>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call >>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task >>>>> >>> >> [CallableStatementCallback] >>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> transaction >>>>> >>> >> rolled back >>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO >>>>> >>> >> >>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> EVENT_ID: >>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User >>>>> >>> >> admin@internal-authz successfully imported image >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi. >>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR >>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb] >>>>> >>> >> Failed >>>>> >>> >> invoking callback end method 'onSucceeded' for command >>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception >>>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?, ?, >>>>> >>> >> ?, ?, ?, >>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element" >>>>> >>> >> violates foreign >>>>> >>> >> key constraint "fk_disk_vm_element_vm_static" >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is >>>>> >>> >> not present >>>>> >>> >> in table "vm_static". >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>>> >>> >> disk_id, >>>>> >>> >> vm_id, >>>>> >>> >> is_boot, >>>>> >>> >> pass_discard, >>>>> >>> >> disk_interface, >>>>> >>> >> is_using_scsi_reservation) >>>>> >>> >> VALUES ( >>>>> >>> >> v_disk_id, >>>>> >>> >> v_vm_id, >>>>> >>> >> v_is_boot, >>>>> >>> >> v_pass_discard, >>>>> >>> >> v_disk_interface, >>>>> >>> >> v_is_using_scsi_reservation)" >>>>> >>> >> PL/pgSQL function >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character >>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is >>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on >>>>> >>> >> table >>>>> >>> >> "disk_vm_element" violates foreign key constraint >>>>> >>> >> "fk_disk_vm_element_vm_static" >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is >>>>> >>> >> not present >>>>> >>> >> in table "vm_static". >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( >>>>> >>> >> disk_id, >>>>> >>> >> vm_id, >>>>> >>> >> is_boot, >>>>> >>> >> pass_discard, >>>>> >>> >> disk_interface, >>>>> >>> >> is_using_scsi_reservation) >>>>> >>> >> VALUES ( >>>>> >>> >> v_disk_id, >>>>> >>> >> v_vm_id, >>>>> >>> >> v_is_boot, >>>>> >>> >> v_pass_discard, >>>>> >>> >> v_disk_interface, >>>>> >>> >> v_is_using_scsi_reservation)" >>>>> >>> >> PL/pgSQL function >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character >>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is >>>>> >>> >> marked for end >>>>> >>> >> method retries >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb] >>>>> >>> >> Command >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' >>>>> >>> >> execution didn't >>>>> >>> >> complete, not proceeding to perform the next operation >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO >>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb] >>>>> >>> >> Command >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' child >>>>> >>> >> commands >>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were >>>>> >>> >> completed, status >>>>> >>> >> 'FAILED' >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> </error> >>>>> >>> >> >>>>> >>> >> -- >>>>> >>> >> Regards, >>>>> >>> >> Evgheni Dereveanchin >>>>> >>> >> >>>>> >>> >> _______________________________________________ >>>>> >>> >> Devel mailing list >>>>> >>> >> Devel(a)ovirt.org >>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>> _______________________________________________ >>>>> >>> Devel mailing list >>>>> >>> Devel(a)ovirt.org >>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> >>>>> >> Eyal edri >>>>> >> >>>>> >> >>>>> >> ASSOCIATE MANAGER >>>>> >> >>>>> >> RHV DevOps >>>>> >> >>>>> >> EMEA VIRTUALIZATION R&D >>>>> >> >>>>> >> >>>>> >> Red Hat EMEA >>>>> >> >>>>> >> TRIED. TESTED. TRUSTED. >>>>> >> phone: +972-9-7692018 <+972%209-769-2018> >>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >> >>>>> >> _______________________________________________ >>>>> >> Devel mailing list >>>>> >> Devel(a)ovirt.org >>>>> >> http://lists.ovirt.org/mailman/listinfo/devel >>>>> > >>>>> > >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >> >> >