[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]

Roy Golan rgolan at redhat.com
Mon Sep 11 19:47:28 UTC 2017 

On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini at redhat.com> wrote:

> That was more a less the my guess from the brief look I took at the
> stacktrace.
>
> I don't want to +2 a patch in the network area, but I'd suggest merging
> this patch (which seems right regardless of the recent failure), and see if
> it solves the OST issue.
>
> OST will run with the network patch reverted, so I need to bring it into
the Q again (revert the reverted :))

On my side, with the MacPool changes it works, imported vms from domains,
started vm, basic editing.

To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm
not mistaken here it should be available _before_ the Backend in order to
keep compensation running? in case we have a complete action that needs to
during compensation.
Maybe this is a reason for MacPool to NOT depend on Backend, so
compensation would complete?
Needs a second look.


On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan at redhat.com> wrote:
>
>>
>>
>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan at redhat.com> wrote:
>>
>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/  so the
>>> revert can be reverted. +Martin Mucha <mmucha at redhat.com> want to test
>>> it? It is working on my end.
>>>
>>> sorry its this https://gerrit.ovirt.org/c/81637/2
>>
>>>
>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken at redhat.com> wrote:
>>>
>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor at redhat.com> wrote:
>>>> >
>>>> > Hello all,
>>>> > this same error still causes OST to fail.
>>>> >
>>>>
>>>> We are aware of it, but do not understand it at all.
>>>>
>>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/
>>>> would be merged once approved by CI.
>>>>
>>>> >
>>>> > Jobs:
>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466
>>>> >
>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471
>>>> >
>>>> > Logs:
>>>> >
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>> >
>>>> >
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>> >
>>>> >
>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg <danken at redhat.com>
>>>> wrote:
>>>> >>>
>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg <danken at redhat.com>
>>>> wrote:
>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin
>>>> >>> > <ederevea at redhat.com> wrote:
>>>> >>> >> Test failed: [ import_template_from_glance ]
>>>> >>> >>
>>>> >>> >> Link to suspected patches:
>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/
>>>> >>> >
>>>> >>> > Martin, this is "core: when initializing MacPool also register in
>>>> it
>>>> >>> > nics in snapshots"
>>>> >>> > and the error seems somewhat related to it
>>>> >>>
>>>> >>> Gil, can you tell us which is the last Engine patch that passed
>>>> OST? I
>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm guessing
>>>> >>> that "ci please build" does not work on merged patches.
>>>> >>
>>>> >>
>>>> >> Not sure I follow, the latest working engine should be deployed to
>>>> tested repo,
>>>> >> So if you'll run the manual job w/o any custom repos it should work
>>>> ( if it doesn't, then it means something slipped in the tested repo and we
>>>> should investigate ).
>>>> >>
>>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs on
>>>> the job, so you can still try it:
>>>> >>
>>>> >> Patch Set 26:
>>>> >> Build Successful
>>>> >>
>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-fc25-x86_64/377/
>>>> : SUCCESS
>>>> >>
>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-el7-x86_64/428/
>>>> : SUCCESS
>>>> >>
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> >
>>>> >>> >>
>>>> >>> >> Link to Job:
>>>> >>> >>
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/
>>>> >>> >>
>>>> >>> >> Link to logs:
>>>> >>> >>
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>> >>> >>
>>>> >>> >> Error snippet from log:
>>>> >>> >>
>>>> >>> >> <error>
>>>> >>> >>
>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Lock Acquired
>>>> >>> >> to object
>>>> >>> >>
>>>> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>> sharedLocks='[]'}'
>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Running
>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities affected
>>>> :  ID:
>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction
>>>> group
>>>> >>> >> CREATE_TEMPLATE with role type USER
>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO
>>>> >>> >>
>>>> [org.ovirt.engine.core.bll.storage.disk.CreateAllTemplateDisksCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Running
>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true.
>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO
>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> transaction
>>>> >>> >> rolled back
>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Command
>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null
>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Exception:
>>>> >>> >> java.lang.NullPointerException
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getVramMultiplier(VgamemVideoSettings.java:63)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getQxlVideoDeviceSettings(VgamemVideoSettings.java:40)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSettings(VideoDeviceSettings.java:34)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSpecParams(VideoDeviceSettings.java:49)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VmDeviceUtils.getVideoDeviceSpecParams(VmDeviceUtils.java:586)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1435)
>>>> >>> >> [bll.jar:]
>>>> >>> >> at
>>>> >>> >>
>>>> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1559)
>>>> >>> >> [bll.jar:]
>>>> >>> >>
>>>> >>> >> ...
>>>> >>> >>
>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Command
>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating
>>>> NEW_ENTITY_ID of
>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate;
>>>> snapshot:
>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1.
>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR
>>>> >>> >>
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> EVENT_ID:
>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template
>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template.
>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Lock freed to
>>>> >>> >> object
>>>> >>> >>
>>>> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>> sharedLocks='[]'}'
>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG
>>>> >>> >>
>>>> [org.ovirt.engine.core.dal.dbbroker.CustomSQLErrorCodeSQLExceptionTranslator]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Translating
>>>> >>> >> SQLException with SQL state '23503', error code '0', message
>>>> [ERROR: insert
>>>> >>> >> or update on table "disk_vm_element" violates foreign key
>>>> constraint
>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>> not present
>>>> >>> >> in table "vm_static".
>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>> >>> >>         disk_id,
>>>> >>> >>         vm_id,
>>>> >>> >>         is_boot,
>>>> >>> >>         pass_discard,
>>>> >>> >>         disk_interface,
>>>> >>> >>         is_using_scsi_reservation)
>>>> >>> >>     VALUES (
>>>> >>> >>         v_disk_id,
>>>> >>> >>         v_vm_id,
>>>> >>> >>         v_is_boot,
>>>> >>> >>         v_pass_discard,
>>>> >>> >>         v_disk_interface,
>>>> >>> >>         v_is_using_scsi_reservation)"
>>>> >>> >> PL/pgSQL function
>>>> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call
>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task
>>>> [CallableStatementCallback]
>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO
>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> transaction
>>>> >>> >> rolled back
>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO
>>>> >>> >>
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> EVENT_ID:
>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User
>>>> >>> >> admin at internal-authz successfully imported image
>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template
>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi.
>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR
>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>> Failed
>>>> >>> >> invoking callback end method 'onSucceeded' for command
>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception
>>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?, ?,
>>>> ?, ?, ?,
>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element"
>>>> violates foreign
>>>> >>> >> key constraint "fk_disk_vm_element_vm_static"
>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>> not present
>>>> >>> >> in table "vm_static".
>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>> >>> >>         disk_id,
>>>> >>> >>         vm_id,
>>>> >>> >>         is_boot,
>>>> >>> >>         pass_discard,
>>>> >>> >>         disk_interface,
>>>> >>> >>         is_using_scsi_reservation)
>>>> >>> >>     VALUES (
>>>> >>> >>         v_disk_id,
>>>> >>> >>         v_vm_id,
>>>> >>> >>         v_is_boot,
>>>> >>> >>         v_pass_discard,
>>>> >>> >>         v_disk_interface,
>>>> >>> >>         v_is_using_scsi_reservation)"
>>>> >>> >> PL/pgSQL function
>>>> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is
>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on
>>>> table
>>>> >>> >> "disk_vm_element" violates foreign key constraint
>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1) is
>>>> not present
>>>> >>> >> in table "vm_static".
>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>> >>> >>         disk_id,
>>>> >>> >>         vm_id,
>>>> >>> >>         is_boot,
>>>> >>> >>         pass_discard,
>>>> >>> >>         disk_interface,
>>>> >>> >>         is_using_scsi_reservation)
>>>> >>> >>     VALUES (
>>>> >>> >>         v_disk_id,
>>>> >>> >>         v_vm_id,
>>>> >>> >>         v_is_boot,
>>>> >>> >>         v_pass_discard,
>>>> >>> >>         v_disk_interface,
>>>> >>> >>         v_is_using_scsi_reservation)"
>>>> >>> >> PL/pgSQL function
>>>> insertdiskvmelement(uuid,uuid,boolean,boolean,character
>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is
>>>> marked for end
>>>> >>> >> method retries
>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>> Command
>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
>>>> execution didn't
>>>> >>> >> complete, not proceeding to perform the next operation
>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>> Command
>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7' child
>>>> commands
>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were
>>>> completed, status
>>>> >>> >> 'FAILED'
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> </error>
>>>> >>> >>
>>>> >>> >> --
>>>> >>> >> Regards,
>>>> >>> >> Evgheni Dereveanchin
>>>> >>> >>
>>>> >>> >> _______________________________________________
>>>> >>> >> Devel mailing list
>>>> >>> >> Devel at ovirt.org
>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>> >>> _______________________________________________
>>>> >>> Devel mailing list
>>>> >>> Devel at ovirt.org
>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >>
>>>> >> Eyal edri
>>>> >>
>>>> >>
>>>> >> ASSOCIATE MANAGER
>>>> >>
>>>> >> RHV DevOps
>>>> >>
>>>> >> EMEA VIRTUALIZATION R&D
>>>> >>
>>>> >>
>>>> >> Red Hat EMEA
>>>> >>
>>>> >> TRIED. TESTED. TRUSTED.
>>>> >> phone: +972-9-7692018 <+972%209-769-2018>
>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>> >>
>>>> >> _______________________________________________
>>>> >> Devel mailing list
>>>> >> Devel at ovirt.org
>>>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>> >
>>>> >
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170911/103800a0/attachment-0001.html>

More information about the Devel mailing list