Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]

On Mon, 11 Sep 2017 at 23:41 Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > On Mon, Sep 11, 2017 at 10:47 PM, Roy Golan <rgolan(a)redhat.com&gt; wrote: > > > > > > On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini(a)redhat.com&gt; > wrote: > >> > >> That was more a less the my guess from the brief look I took at the > >> stacktrace. > >> > >> I don't want to +2 a patch in the network area, but I'd suggest merging > >> this patch (which seems right regardless of the recent failure), and > see if > >> it solves the OST issue. > >> > > OST will run with the network patch reverted, so I need to bring it > into the > > Q again (revert the reverted :)) > > Even I can do that. I've rebase your patches on top of current master, > and reintroduced Mucha's patch (https://gerrit.ovirt.org/#/c/81639/). > If OST passes, I'll ask you to backport your injection fixes - we need > to solve the MAC collision bug in ovirt-4.1, too. > > http://jenkins.ovirt.org/job/ovirt-system-tests_manual/1129/ is green > > > > > On my side, with the MacPool changes it works, imported vms from > domains, > > started vm, basic editing. > > > > To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if > I'm > > not mistaken here it should be available _before_ the Backend in order > to > > keep compensation running? in case we have a complete action that needs > to > > during compensation. > > Maybe this is a reason for MacPool to NOT depend on Backend, so > compensation > > would complete? > > Needs a second look. > > > > > >> On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan(a)redhat.com&gt; wrote: > >>> > >>> > >>> > >>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan(a)redhat.com&gt; wrote: > >>>> > >>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/ so the > >>>> revert can be reverted. +Martin Mucha want to test it? It is working > on my > >>>> end. > >>>> > >>> sorry its this https://gerrit.ovirt.org/c/81637/2 > >>>> > >>>> > >>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken(a)redhat.com&gt; > wrote: > >>>>> > >>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor(a)redhat.com&gt; > wrote: > >>>>> > > >>>>> > Hello all, > >>>>> > this same error still causes OST to fail. > >>>>> > > >>>>> > >>>>> We are aware of it, but do not understand it at all. > >>>>> > >>>>> Anyway, a revert is proposed and > https://gerrit.ovirt.org/#/c/81618/ > >>>>> would be merged once approved by CI. > >>>>> > >>>>> > > >>>>> > Jobs: > >>>>> > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466 > >>>>> > > >>>>> > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471 > >>>>> > > >>>>> > Logs: > >>>>> > > >>>>> > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466/artifa... > >>>>> > > >>>>> > > >>>>> > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471/artifa... > >>>>> > > >>>>> > > >>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri(a)redhat.com&gt; > wrote: > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg < > danken(a)redhat.com&gt; > >>>>> >> wrote: > >>>>> >>> > >>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg < > danken(a)redhat.com&gt; > >>>>> >>> wrote: > >>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin > >>>>> >>> > <ederevea(a)redhat.com&gt; wrote: > >>>>> >>> >> Test failed: [ import_template_from_glance ] > >>>>> >>> >> > >>>>> >>> >> Link to suspected patches: > >>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/ > >>>>> >>> > > >>>>> >>> > Martin, this is "core: when initializing MacPool also > register in > >>>>> >>> > it > >>>>> >>> > nics in snapshots" > >>>>> >>> > and the error seems somewhat related to it > >>>>> >>> > >>>>> >>> Gil, can you tell us which is the last Engine patch that passed > >>>>> >>> OST? I > >>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm > guessing > >>>>> >>> that "ci please build" does not work on merged patches. > >>>>> >> > >>>>> >> > >>>>> >> Not sure I follow, the latest working engine should be deployed > to > >>>>> >> tested repo, > >>>>> >> So if you'll run the manual job w/o any custom repos it should > work > >>>>> >> ( if it doesn't, then it means something slipped in the tested > repo and we > >>>>> >> should investigate ). > >>>>> >> > >>>>> >> But anyhow, your 'ci please build' also worked, and produced > RPMs on > >>>>> >> the job, so you can still try it: > >>>>> >> > >>>>> >> Patch Set 26: > >>>>> >> Build Successful > >>>>> >> > >>>>> >> > http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-deman... > >>>>> >> : SUCCESS > >>>>> >> > >>>>> >> > http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-deman... > >>>>> >> : SUCCESS > >>>>> >> > >>>>> >> > >>>>> >>> > >>>>> >>> > >>>>> >>> > > >>>>> >>> >> > >>>>> >>> >> Link to Job: > >>>>> >>> >> > >>>>> >>> >> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/ > >>>>> >>> >> > >>>>> >>> >> Link to logs: > >>>>> >>> >> > >>>>> >>> >> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/artifa... > >>>>> >>> >> > >>>>> >>> >> Error snippet from log: > >>>>> >>> >> > >>>>> >>> >> <error> > >>>>> >>> >> > >>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Lock Acquired > >>>>> >>> >> to object > >>>>> >>> >> > >>>>> >>> >> > 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE, > >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', > >>>>> >>> >> sharedLocks='[]'}' > >>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Running > >>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities > affected > >>>>> >>> >> : ID: > >>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction > >>>>> >>> >> group > >>>>> >>> >> CREATE_TEMPLATE with role type USER > >>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO > >>>>> >>> >> > >>>>> >>> >> > [org.ovirt.engine.core.bll.storage.disk.CreateAllTemplateDisksCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Running > >>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true. > >>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> transaction > >>>>> >>> >> rolled back > >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Command > >>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null > >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Exception: > >>>>> >>> >> java.lang.NullPointerException > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getVramMultiplier(VgamemVideoSettings.java:63) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getQxlVideoDeviceSettings(VgamemVideoSettings.java:40) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSettings(VideoDeviceSettings.java:34) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSpecParams(VideoDeviceSettings.java:49) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VmDeviceUtils.getVideoDeviceSpecParams(VmDeviceUtils.java:586) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1435) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> at > >>>>> >>> >> > >>>>> >>> >> > org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1559) > >>>>> >>> >> [bll.jar:] > >>>>> >>> >> > >>>>> >>> >> ... > >>>>> >>> >> > >>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Command > >>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating > >>>>> >>> >> NEW_ENTITY_ID of > >>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate; > >>>>> >>> >> snapshot: > >>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1. > >>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR > >>>>> >>> >> > >>>>> >>> >> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> EVENT_ID: > >>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template > >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template. > >>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Lock freed to > >>>>> >>> >> object > >>>>> >>> >> > >>>>> >>> >> > 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE, > >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]', > >>>>> >>> >> sharedLocks='[]'}' > >>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG > >>>>> >>> >> > >>>>> >>> >> > [org.ovirt.engine.core.dal.dbbroker.CustomSQLErrorCodeSQLExceptionTranslator] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Translating > >>>>> >>> >> SQLException with SQL state '23503', error code '0', message > >>>>> >>> >> [ERROR: insert > >>>>> >>> >> or update on table "disk_vm_element" violates foreign key > >>>>> >>> >> constraint > >>>>> >>> >> "fk_disk_vm_element_vm_static" > >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) > is > >>>>> >>> >> not present > >>>>> >>> >> in table "vm_static". > >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( > >>>>> >>> >> disk_id, > >>>>> >>> >> vm_id, > >>>>> >>> >> is_boot, > >>>>> >>> >> pass_discard, > >>>>> >>> >> disk_interface, > >>>>> >>> >> is_using_scsi_reservation) > >>>>> >>> >> VALUES ( > >>>>> >>> >> v_disk_id, > >>>>> >>> >> v_vm_id, > >>>>> >>> >> v_is_boot, > >>>>> >>> >> v_pass_discard, > >>>>> >>> >> v_disk_interface, > >>>>> >>> >> v_is_using_scsi_reservation)" > >>>>> >>> >> PL/pgSQL function > >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character > >>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call > >>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task > >>>>> >>> >> [CallableStatementCallback] > >>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO > >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> transaction > >>>>> >>> >> rolled back > >>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO > >>>>> >>> >> > >>>>> >>> >> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> EVENT_ID: > >>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User > >>>>> >>> >> admin@internal-authz successfully imported image > >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template > >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi. > >>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR > >>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) > [74b41edb] > >>>>> >>> >> Failed > >>>>> >>> >> invoking callback end method 'onSucceeded' for command > >>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception > >>>>> >>> >> 'CallableStatementCallback; SQL [{call > insertdiskvmelement(?, ?, > >>>>> >>> >> ?, ?, ?, > >>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element" > >>>>> >>> >> violates foreign > >>>>> >>> >> key constraint "fk_disk_vm_element_vm_static" > >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) > is > >>>>> >>> >> not present > >>>>> >>> >> in table "vm_static". > >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( > >>>>> >>> >> disk_id, > >>>>> >>> >> vm_id, > >>>>> >>> >> is_boot, > >>>>> >>> >> pass_discard, > >>>>> >>> >> disk_interface, > >>>>> >>> >> is_using_scsi_reservation) > >>>>> >>> >> VALUES ( > >>>>> >>> >> v_disk_id, > >>>>> >>> >> v_vm_id, > >>>>> >>> >> v_is_boot, > >>>>> >>> >> v_pass_discard, > >>>>> >>> >> v_disk_interface, > >>>>> >>> >> v_is_using_scsi_reservation)" > >>>>> >>> >> PL/pgSQL function > >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character > >>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is > >>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on > >>>>> >>> >> table > >>>>> >>> >> "disk_vm_element" violates foreign key constraint > >>>>> >>> >> "fk_disk_vm_element_vm_static" > >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) > is > >>>>> >>> >> not present > >>>>> >>> >> in table "vm_static". > >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element ( > >>>>> >>> >> disk_id, > >>>>> >>> >> vm_id, > >>>>> >>> >> is_boot, > >>>>> >>> >> pass_discard, > >>>>> >>> >> disk_interface, > >>>>> >>> >> is_using_scsi_reservation) > >>>>> >>> >> VALUES ( > >>>>> >>> >> v_disk_id, > >>>>> >>> >> v_vm_id, > >>>>> >>> >> v_is_boot, > >>>>> >>> >> v_pass_discard, > >>>>> >>> >> v_disk_interface, > >>>>> >>> >> v_is_using_scsi_reservation)" > >>>>> >>> >> PL/pgSQL function > >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character > >>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is > >>>>> >>> >> marked for end > >>>>> >>> >> method retries > >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO > >>>>> >>> >> > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) > [74b41edb] > >>>>> >>> >> Command > >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' > >>>>> >>> >> execution didn't > >>>>> >>> >> complete, not proceeding to perform the next operation > >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO > >>>>> >>> >> > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) > [74b41edb] > >>>>> >>> >> Command > >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' > child > >>>>> >>> >> commands > >>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were > >>>>> >>> >> completed, status > >>>>> >>> >> 'FAILED' > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> </error> > >>>>> >>> >> > >>>>> >>> >> -- > >>>>> >>> >> Regards, > >>>>> >>> >> Evgheni Dereveanchin > >>>>> >>> >> > >>>>> >>> >> _______________________________________________ > >>>>> >>> >> Devel mailing list > >>>>> >>> >> Devel(a)ovirt.org > >>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel > >>>>> >>> _______________________________________________ > >>>>> >>> Devel mailing list > >>>>> >>> Devel(a)ovirt.org > >>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel > >>>>> >>> > >>>>> >>> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> > >>>>> >> Eyal edri > >>>>> >> > >>>>> >> > >>>>> >> ASSOCIATE MANAGER > >>>>> >> > >>>>> >> RHV DevOps > >>>>> >> > >>>>> >> EMEA VIRTUALIZATION R&D > >>>>> >> > >>>>> >> > >>>>> >> Red Hat EMEA > >>>>> >> > >>>>> >> TRIED. TESTED. TRUSTED. > >>>>> >> phone: +972-9-7692018 <+972%209-769-2018> > >>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) > >>>>> >> > >>>>> >> _______________________________________________ > >>>>> >> Devel mailing list > >>>>> >> Devel(a)ovirt.org > >>>>> >> http://lists.ovirt.org/mailman/listinfo/devel > >>>>> > > >>>>> > > >>>>> _______________________________________________ > >>>>> Devel mailing list > >>>>> Devel(a)ovirt.org > >>>>> http://lists.ovirt.org/mailman/listinfo/devel > >>> > >>> > >>> _______________________________________________ > >>> Devel mailing list > >>> Devel(a)ovirt.org > >>> http://lists.ovirt.org/mailman/listinfo/devel > >> > >> > > >