[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]

Martin Perina mperina at redhat.com
Mon Sep 11 19:56:23 UTC 2017 

On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan at redhat.com> wrote:

>
>
> On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini at redhat.com> wrote:
>
>> That was more a less the my guess from the brief look I took at the
>> stacktrace.
>>
>> I don't want to +2 a patch in the network area, but I'd suggest merging
>> this patch (which seems right regardless of the recent failure), and see if
>> it solves the OST issue.
>>
>> OST will run with the network patch reverted, so I need to bring it into
> the Q again (revert the reverted :))
>
> On my side, with the MacPool changes it works, imported vms from domains,
> started vm, basic editing.
>
> To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm
> not mistaken here it should be available _before_ the Backend in order to
> keep compensation running? in case we have a complete action that needs to
> during compensation.
> Maybe this is a reason for MacPool to NOT depend on Backend, so
> compensation would complete?
> Needs a second look.
>
>>
​Ravi/Moti, could you please comment?
​


> On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan at redhat.com> wrote:
>>
>>>
>>>
>>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan at redhat.com> wrote:
>>>
>>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/  so the
>>>> revert can be reverted. +Martin Mucha <mmucha at redhat.com> want to test
>>>> it? It is working on my end.
>>>>
>>>> sorry its this https://gerrit.ovirt.org/c/81637/2
>>>
>>>>
>>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken at redhat.com> wrote:
>>>>
>>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor at redhat.com>
>>>>> wrote:
>>>>> >
>>>>> > Hello all,
>>>>> > this same error still causes OST to fail.
>>>>> >
>>>>>
>>>>> We are aware of it, but do not understand it at all.
>>>>>
>>>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/
>>>>> would be merged once approved by CI.
>>>>>
>>>>> >
>>>>> > Jobs:
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466
>>>>> >
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471
>>>>> >
>>>>> > Logs:
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-
>>>>> tester/2466/artifact/exported-artifacts/basic-suit-master-
>>>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/
>>>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >
>>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-
>>>>> tester/2471/artifact/exported-artifacts/basic-suit-master-
>>>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/
>>>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >
>>>>> >
>>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg <danken at redhat.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg <danken at redhat.com>
>>>>> wrote:
>>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin
>>>>> >>> > <ederevea at redhat.com> wrote:
>>>>> >>> >> Test failed: [ import_template_from_glance ]
>>>>> >>> >>
>>>>> >>> >> Link to suspected patches:
>>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/
>>>>> >>> >
>>>>> >>> > Martin, this is "core: when initializing MacPool also register
>>>>> in it
>>>>> >>> > nics in snapshots"
>>>>> >>> > and the error seems somewhat related to it
>>>>> >>>
>>>>> >>> Gil, can you tell us which is the last Engine patch that passed
>>>>> OST? I
>>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm
>>>>> guessing
>>>>> >>> that "ci please build" does not work on merged patches.
>>>>> >>
>>>>> >>
>>>>> >> Not sure I follow, the latest working engine should be deployed to
>>>>> tested repo,
>>>>> >> So if you'll run the manual job w/o any custom repos it should work
>>>>> ( if it doesn't, then it means something slipped in the tested repo and we
>>>>> should investigate ).
>>>>> >>
>>>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs
>>>>> on the job, so you can still try it:
>>>>> >>
>>>>> >> Patch Set 26:
>>>>> >> Build Successful
>>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-
>>>>> artifacts-on-demand-fc25-x86_64/377/ : SUCCESS
>>>>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-
>>>>> artifacts-on-demand-el7-x86_64/428/ : SUCCESS
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>>
>>>>> >>> >
>>>>> >>> >>
>>>>> >>> >> Link to Job:
>>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
>>>>> tester/2397/
>>>>> >>> >>
>>>>> >>> >> Link to logs:
>>>>> >>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
>>>>> tester/2397/artifact/exported-artifacts/basic-suit-master-
>>>>> el7/test_logs/basic-suite-master/post-002_bootstrap.py/
>>>>> lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
>>>>> >>> >>
>>>>> >>> >> Error snippet from log:
>>>>> >>> >>
>>>>> >>> >> <error>
>>>>> >>> >>
>>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Lock Acquired
>>>>> >>> >> to object
>>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-
>>>>> 9744309ca8c1=TEMPLATE,
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>>> sharedLocks='[]'}'
>>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Running
>>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities affected
>>>>> :  ID:
>>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction
>>>>> group
>>>>> >>> >> CREATE_TEMPLATE with role type USER
>>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.storage.disk.
>>>>> CreateAllTemplateDisksCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Running
>>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true.
>>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> transaction
>>>>> >>> >> rolled back
>>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Command
>>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null
>>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Exception:
>>>>> >>> >> java.lang.NullPointerException
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.
>>>>> getVramMultiplier(VgamemVideoSettings.java:63)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.
>>>>> getQxlVideoDeviceSettings(VgamemVideoSettings.java:40)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.
>>>>> getVideoDeviceSettings(VideoDeviceSettings.java:34)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.
>>>>> getVideoDeviceSpecParams(VideoDeviceSettings.java:49)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.
>>>>> getVideoDeviceSpecParams(VmDeviceUtils.java:586)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.
>>>>> copyVmDevices(VmDeviceUtils.java:1435)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >> at
>>>>> >>> >> org.ovirt.engine.core.bll.utils.VmDeviceUtils.
>>>>> copyVmDevices(VmDeviceUtils.java:1559)
>>>>> >>> >> [bll.jar:]
>>>>> >>> >>
>>>>> >>> >> ...
>>>>> >>> >>
>>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Command
>>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating
>>>>> NEW_ENTITY_ID of
>>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate;
>>>>> snapshot:
>>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1.
>>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.
>>>>> AuditLogDirector]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> EVENT_ID:
>>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template.
>>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Lock freed to
>>>>> >>> >> object
>>>>> >>> >> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-
>>>>> 9744309ca8c1=TEMPLATE,
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
>>>>> sharedLocks='[]'}'
>>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.
>>>>> CustomSQLErrorCodeSQLExceptionTranslator]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Translating
>>>>> >>> >> SQLException with SQL state '23503', error code '0', message
>>>>> [ERROR: insert
>>>>> >>> >> or update on table "disk_vm_element" violates foreign key
>>>>> constraint
>>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
>>>>> is not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >>         disk_id,
>>>>> >>> >>         vm_id,
>>>>> >>> >>         is_boot,
>>>>> >>> >>         pass_discard,
>>>>> >>> >>         disk_interface,
>>>>> >>> >>         is_using_scsi_reservation)
>>>>> >>> >>     VALUES (
>>>>> >>> >>         v_disk_id,
>>>>> >>> >>         v_vm_id,
>>>>> >>> >>         v_is_boot,
>>>>> >>> >>         v_pass_discard,
>>>>> >>> >>         v_disk_interface,
>>>>> >>> >>         v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid,
>>>>> boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call
>>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task
>>>>> [CallableStatementCallback]
>>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> transaction
>>>>> >>> >> rolled back
>>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.
>>>>> AuditLogDirector]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> EVENT_ID:
>>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User
>>>>> >>> >> admin at internal-authz successfully imported image
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template
>>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi.
>>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR
>>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
>>>>> Failed
>>>>> >>> >> invoking callback end method 'onSucceeded' for command
>>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception
>>>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?,
>>>>> ?, ?, ?, ?,
>>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element"
>>>>> violates foreign
>>>>> >>> >> key constraint "fk_disk_vm_element_vm_static"
>>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
>>>>> is not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >>         disk_id,
>>>>> >>> >>         vm_id,
>>>>> >>> >>         is_boot,
>>>>> >>> >>         pass_discard,
>>>>> >>> >>         disk_interface,
>>>>> >>> >>         is_using_scsi_reservation)
>>>>> >>> >>     VALUES (
>>>>> >>> >>         v_disk_id,
>>>>> >>> >>         v_vm_id,
>>>>> >>> >>         v_is_boot,
>>>>> >>> >>         v_pass_discard,
>>>>> >>> >>         v_disk_interface,
>>>>> >>> >>         v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid,
>>>>> boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is
>>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on
>>>>> table
>>>>> >>> >> "disk_vm_element" violates foreign key constraint
>>>>> >>> >> "fk_disk_vm_element_vm_static"
>>>>> >>> >>   Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
>>>>> is not present
>>>>> >>> >> in table "vm_static".
>>>>> >>> >>   Where: SQL statement "INSERT INTO disk_vm_element (
>>>>> >>> >>         disk_id,
>>>>> >>> >>         vm_id,
>>>>> >>> >>         is_boot,
>>>>> >>> >>         pass_discard,
>>>>> >>> >>         disk_interface,
>>>>> >>> >>         is_using_scsi_reservation)
>>>>> >>> >>     VALUES (
>>>>> >>> >>         v_disk_id,
>>>>> >>> >>         v_vm_id,
>>>>> >>> >>         v_is_boot,
>>>>> >>> >>         v_pass_discard,
>>>>> >>> >>         v_disk_interface,
>>>>> >>> >>         v_is_using_scsi_reservation)"
>>>>> >>> >> PL/pgSQL function insertdiskvmelement(uuid,uuid,
>>>>> boolean,boolean,character
>>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is
>>>>> marked for end
>>>>> >>> >> method retries
>>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCa
>>>>> llback]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>>> Command
>>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
>>>>> execution didn't
>>>>> >>> >> complete, not proceeding to perform the next operation
>>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
>>>>> >>> >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCa
>>>>> llback]
>>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
>>>>> Command
>>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
>>>>> child commands
>>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were
>>>>> completed, status
>>>>> >>> >> 'FAILED'
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> </error>
>>>>> >>> >>
>>>>> >>> >> --
>>>>> >>> >> Regards,
>>>>> >>> >> Evgheni Dereveanchin
>>>>> >>> >>
>>>>> >>> >> _______________________________________________
>>>>> >>> >> Devel mailing list
>>>>> >>> >> Devel at ovirt.org
>>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >>> _______________________________________________
>>>>> >>> Devel mailing list
>>>>> >>> Devel at ovirt.org
>>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >>
>>>>> >> Eyal edri
>>>>> >>
>>>>> >>
>>>>> >> ASSOCIATE MANAGER
>>>>> >>
>>>>> >> RHV DevOps
>>>>> >>
>>>>> >> EMEA VIRTUALIZATION R&D
>>>>> >>
>>>>> >>
>>>>> >> Red Hat EMEA
>>>>> >>
>>>>> >> TRIED. TESTED. TRUSTED.
>>>>> >> phone: +972-9-7692018 <+972%209-769-2018>
>>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Devel mailing list
>>>>> >> Devel at ovirt.org
>>>>> >> http://lists.ovirt.org/mailman/listinfo/devel
>>>>> >
>>>>> >
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170911/458b3b10/attachment-0001.html>

More information about the Devel mailing list