On Mon, 11 Sep 2017 at 23:41 Dan Kenigsberg <danken@redhat.com> wrote:

On Mon, Sep 11, 2017 at 10:47 PM, Roy Golan <rgolan@redhat.com> wrote:
>
>
> On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini@redhat.com> wrote:
>>
>> That was more a less the my guess from the brief look I took at the
>> stacktrace.
>>
>> I don't want to +2 a patch in the network area, but I'd suggest merging
>> this patch (which seems right regardless of the recent failure), and see if
>> it solves the OST issue.
>>
> OST will run with the network patch reverted, so I need to bring it into the
> Q again (revert the reverted :))

Even I can do that. I've rebase your patches on top of current master,
and reintroduced Mucha's patch (https://gerrit.ovirt.org/#/c/81639/).
If OST passes, I'll ask you to backport your injection fixes - we need
to solve the MAC collision bug in ovirt-4.1, too.

http://jenkins.ovirt.org/job/ovirt-system-tests_manual/1129/ is green

>
> On my side, with the MacPool changes it works, imported vms from domains,
> started vm, basic editing.
>
> To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm
> not mistaken here it should be available _before_ the Backend in order to
> keep compensation running? in case we have a complete action that needs to
> during compensation.
> Maybe this is a reason for MacPool to NOT depend on Backend, so compensation
> would complete?
> Needs a second look.
>
>
>> On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan@redhat.com> wrote:
>>>
>>>
>>>
>>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan@redhat.com> wrote:
>>>>
>>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/ so the
>>>> revert can be reverted. +Martin Mucha want to test it? It is working on my
>>>> end.
>>>>
>>> sorry its this https://gerrit.ovirt.org/c/81637/2
>>>>
>>>>
>>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken@redhat.com> wrote:
>>>>>
>>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor@redhat.com> wrote:
>>>>> >
>>>>> > Hello all,
>>>>> > this same error still causes OST to fail.
>>>>> >
>>>>>
>>>>> We are aware of it, but do not understand it at all.
>>>>>
>>>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/
>>>>> would be merged once approved by CI.
>>>>>
>>>>> >
>>>>> > Jobs:
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466
>>>>> >
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471
>>>>> >
>>>>> > Logs:
>>>>> >
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >
>>>>> >
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >
>>>>> >
>>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri@redhat.com> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg <danken@redhat.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg <danken@redhat.com>
>>>>> >>> wrote:
>>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin
>>>>> >>> > <ederevea@redhat.com> wrote:
>>>>> >>> >> Test failed: [ import_template_from_glance ]
>>>>> >>> >>
>>>>> >>> >> Link to suspected patches:
>>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/
>>>>> >>> >
>>>>> >>> > Martin, this is "core: when initializing MacPool also register in
>>>>> >>> > it
>>>>> >>> > nics in snapshots"
>>>>> >>> > and the error seems somewhat related to it
>>>>> >>>
>>>>> >>> Gil, can you tell us which is the last Engine patch that passed
>>>>> >>> OST? I
>>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm guessing
>>>>> >>> that "ci please build" does not work on merged patches.
>>>>> >>
>>>>> >>
>>>>> >> Not sure I follow, the latest working engine should be deployed to
>>>>> >> tested repo,
>>>>> >> So if you'll run the manual job w/o any custom repos it should work
>>>>> >> ( if it doesn't, then it means something slipped in the tested repo and we
>>>>> >> should investigate ).
>>>>> >>
>>>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs on
>>>>> >> the job, so you can still try it:
>>>>> >>
>>>>> >> Patch Set 26:
>>>>> >> Build Successful
>>>>> >>
>>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-fc25-x86_64/377/
>>>>> >> : SUCCESS
>>>>> >>
>>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-el7-x86_64/428/
>>>>> >> : SUCCESS
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>>
>>>>> >>> >
>>>>> >>> >>
>>>>> >>> >> Link to Job:
>>>>> >>> >>
>>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/
>>>>> >>> >>
>>>>> >>> >> Link to logs:
>>>>> >>> >>
>>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >>> >>
>>>>> >>> >> Error snippet from log:
>>>>> >>> >>
>>>>> >>> >> <error>
>>>>> >>> >>
>>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Lock Acquired
>>>>> >>> >> to object
>>>>> >>> >>
>>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>>> >>> >> sharedLocks='[]'}'
>>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Running
>>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities affected
>>>>> >>> >> : ID:
>>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction
>>>>> >>> >> group
>>>>> >>> >> CREATE_TEMPLATE with role type USER
>>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO
>>>>> >>> >>
>>>>> >>> >> [org.ovirt.engine.core.bll.storage.disk.CreateAllTemplateDisksCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Running
>>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true.
>>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> transaction
>>>>> >>> >> rolled back
>>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Command
>>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null
>>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Exception:
>>>>> >>> >> java.lang.NullPointerException
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getVramMultiplier(VgamemVideoSettings.java:63)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getQxlVideoDeviceSettings(VgamemVideoSettings.java:40)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSettings(VideoDeviceSettings.java:34)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSpecParams(VideoDeviceSettings.java:49)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.getVideoDeviceSpecParams(VmDeviceUtils.java:586)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1435)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >>
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1559)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >>
>>>>> >>> >> ...
>>>>> >>> >>
>>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Command
>>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating
>>>>> >>> >> NEW_ENTITY_ID of
>>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate;
>>>>> >>> >> snapshot:
>>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1.
>>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR
>>>>> >>> >>
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> EVENT_ID:
>>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template.
>>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Lock freed to
>>>>> >>> >> object
>>>>> >>> >>
>>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>>> >>> >> sharedLocks='[]'}'
>>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG
>>>>> >>> >>
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.CustomSQLErrorCodeSQLExceptionTranslator]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Translating
>>>>> >>> >> SQLException with SQL state '23503', error code '0', message
>>>>> >>> >> [ERROR: insert
>>>>> >>> >> or update on table "disk_vm_element" violates foreign key
>>>>> >>> >> constraint
>>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>>> >>> >> not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >> disk_id,
>>>>> >>> >> vm_id,
>>>>> >>> >> is_boot,
>>>>> >>> >> pass_discard,
>>>>> >>> >> disk_interface,
>>>>> >>> >> is_using_scsi_reservation)
>>>>> >>> >> VALUES (
>>>>> >>> >> v_disk_id,
>>>>> >>> >> v_vm_id,
>>>>> >>> >> v_is_boot,
>>>>> >>> >> v_pass_discard,
>>>>> >>> >> v_disk_interface,
>>>>> >>> >> v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function
>>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call
>>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task
>>>>> >>> >> [CallableStatementCallback]
>>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> transaction
>>>>> >>> >> rolled back
>>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO
>>>>> >>> >>
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> EVENT_ID:
>>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User
>>>>> >>> >> admin@internal-authz successfully imported image
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi.
>>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> >>> >> Failed
>>>>> >>> >> invoking callback end method 'onSucceeded' for command
>>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception
>>>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?, ?,
>>>>> >>> >> ?, ?, ?,
>>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element"
>>>>> >>> >> violates foreign
>>>>> >>> >> key constraint "fk_disk_vm_element_vm_static"
>>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>>> >>> >> not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >> disk_id,
>>>>> >>> >> vm_id,
>>>>> >>> >> is_boot,
>>>>> >>> >> pass_discard,
>>>>> >>> >> disk_interface,
>>>>> >>> >> is_using_scsi_reservation)
>>>>> >>> >> VALUES (
>>>>> >>> >> v_disk_id,
>>>>> >>> >> v_vm_id,
>>>>> >>> >> v_is_boot,
>>>>> >>> >> v_pass_discard,
>>>>> >>> >> v_disk_interface,
>>>>> >>> >> v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function
>>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is
>>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on
>>>>> >>> >> table
>>>>> >>> >> "disk_vm_element" violates foreign key constraint
>>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>>> >>> >> not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >> disk_id,
>>>>> >>> >> vm_id,
>>>>> >>> >> is_boot,
>>>>> >>> >> pass_discard,
>>>>> >>> >> disk_interface,
>>>>> >>> >> is_using_scsi_reservation)
>>>>> >>> >> VALUES (
>>>>> >>> >> v_disk_id,
>>>>> >>> >> v_vm_id,
>>>>> >>> >> v_is_boot,
>>>>> >>> >> v_pass_discard,
>>>>> >>> >> v_disk_interface,
>>>>> >>> >> v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function
>>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is
>>>>> >>> >> marked for end
>>>>> >>> >> method retries
>>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>>> >>> >> Command
>>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
>>>>> >>> >> execution didn't
>>>>> >>> >> complete, not proceeding to perform the next operation
>>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>>> >>> >> Command
>>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' child
>>>>> >>> >> commands
>>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were
>>>>> >>> >> completed, status
>>>>> >>> >> 'FAILED'
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> </error>
>>>>> >>> >>
>>>>> >>> >> --
>>>>> >>> >> Regards,
>>>>> >>> >> Evgheni Dereveanchin
>>>>> >>> >>
>>>>> >>> >> _______________________________________________
>>>>> >>> >> Devel mailing list
>>>>> >>> >> Devel@ovirt.org
>>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >>> _______________________________________________
>>>>> >>> Devel mailing list
>>>>> >>> Devel@ovirt.org
>>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >>
>>>>> >> Eyal edri
>>>>> >>
>>>>> >>
>>>>> >> ASSOCIATE MANAGER
>>>>> >>
>>>>> >> RHV DevOps
>>>>> >>
>>>>> >> EMEA VIRTUALIZATION R&D
>>>>> >>
>>>>> >>
>>>>> >> Red Hat EMEA
>>>>> >>
>>>>> >> TRIED. TESTED. TRUSTED.
>>>>> >> phone: +972-9-7692018
>>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Devel mailing list
>>>>> >> Devel@ovirt.org
>>>>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >
>>>>> >
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel@ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>