[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 2017-09-07 ] [import_template_from_glance]
Roy Golan
rgolan at redhat.com
Mon Sep 11 21:15:06 UTC 2017
On Mon, 11 Sep 2017 at 23:41 Dan Kenigsberg <danken at redhat.com> wrote:
> On Mon, Sep 11, 2017 at 10:47 PM, Roy Golan <rgolan at redhat.com> wrote:
> >
> >
> > On Mon, 11 Sep 2017 at 22:34 Allon Mureinik <amureini at redhat.com> wrote:
> >>
> >> That was more a less the my guess from the brief look I took at the
> >> stacktrace.
> >>
> >> I don't want to +2 a patch in the network area, but I'd suggest merging
> >> this patch (which seems right regardless of the recent failure), and
> see if
> >> it solves the OST issue.
> >>
> > OST will run with the network patch reverted, so I need to bring it into
> the
> > Q again (revert the reverted :))
>
> Even I can do that. I've rebase your patches on top of current master,
> and reintroduced Mucha's patch (https://gerrit.ovirt.org/#/c/81639/).
> If OST passes, I'll ask you to backport your injection fixes - we need
> to solve the MAC collision bug in ovirt-4.1, too.
>
> http://jenkins.ovirt.org/job/ovirt-system-tests_manual/1129/ is green
>
> >
> > On my side, with the MacPool changes it works, imported vms from domains,
> > started vm, basic editing.
> >
> > To add to that, CoCoAsyncTaskHelper is another @Singleton EJB, but if I'm
> > not mistaken here it should be available _before_ the Backend in order to
> > keep compensation running? in case we have a complete action that needs
> to
> > during compensation.
> > Maybe this is a reason for MacPool to NOT depend on Backend, so
> compensation
> > would complete?
> > Needs a second look.
> >
> >
> >> On Mon, Sep 11, 2017 at 9:47 PM, Roy Golan <rgolan at redhat.com> wrote:
> >>>
> >>>
> >>>
> >>> On Mon, 11 Sep 2017 at 21:30 Roy Golan <rgolan at redhat.com> wrote:
> >>>>
> >>>> I think I solved it with https://gerrit.ovirt.org/#/c/81636/ so the
> >>>> revert can be reverted. +Martin Mucha want to test it? It is working
> on my
> >>>> end.
> >>>>
> >>> sorry its this https://gerrit.ovirt.org/c/81637/2
> >>>>
> >>>>
> >>>> On Mon, 11 Sep 2017 at 18:41 Dan Kenigsberg <danken at redhat.com>
> wrote:
> >>>>>
> >>>>> On Mon, Sep 11, 2017 at 4:46 PM, Dusan Fodor <dfodor at redhat.com>
> wrote:
> >>>>> >
> >>>>> > Hello all,
> >>>>> > this same error still causes OST to fail.
> >>>>> >
> >>>>>
> >>>>> We are aware of it, but do not understand it at all.
> >>>>>
> >>>>> Anyway, a revert is proposed and https://gerrit.ovirt.org/#/c/81618/
> >>>>> would be merged once approved by CI.
> >>>>>
> >>>>> >
> >>>>> > Jobs:
> >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466
> >>>>> >
> >>>>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471
> >>>>> >
> >>>>> > Logs:
> >>>>> >
> >>>>> >
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2466/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
> >>>>> >
> >>>>> >
> >>>>> >
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2471/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
> >>>>> >
> >>>>> >
> >>>>> > On Mon, Sep 11, 2017 at 1:22 PM, Eyal Edri <eedri at redhat.com>
> wrote:
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> On Mon, Sep 11, 2017 at 1:21 PM, Dan Kenigsberg <
> danken at redhat.com>
> >>>>> >> wrote:
> >>>>> >>>
> >>>>> >>> On Mon, Sep 11, 2017 at 8:59 AM, Dan Kenigsberg <
> danken at redhat.com>
> >>>>> >>> wrote:
> >>>>> >>> > On Thu, Sep 7, 2017 at 6:12 PM, Evgheni Dereveanchin
> >>>>> >>> > <ederevea at redhat.com> wrote:
> >>>>> >>> >> Test failed: [ import_template_from_glance ]
> >>>>> >>> >>
> >>>>> >>> >> Link to suspected patches:
> >>>>> >>> >> https://gerrit.ovirt.org/#/c/80450/
> >>>>> >>> >
> >>>>> >>> > Martin, this is "core: when initializing MacPool also register
> in
> >>>>> >>> > it
> >>>>> >>> > nics in snapshots"
> >>>>> >>> > and the error seems somewhat related to it
> >>>>> >>>
> >>>>> >>> Gil, can you tell us which is the last Engine patch that passed
> >>>>> >>> OST? I
> >>>>> >>> tried to build https://gerrit.ovirt.org/#/c/76309/ but I'm
> guessing
> >>>>> >>> that "ci please build" does not work on merged patches.
> >>>>> >>
> >>>>> >>
> >>>>> >> Not sure I follow, the latest working engine should be deployed to
> >>>>> >> tested repo,
> >>>>> >> So if you'll run the manual job w/o any custom repos it should
> work
> >>>>> >> ( if it doesn't, then it means something slipped in the tested
> repo and we
> >>>>> >> should investigate ).
> >>>>> >>
> >>>>> >> But anyhow, your 'ci please build' also worked, and produced RPMs
> on
> >>>>> >> the job, so you can still try it:
> >>>>> >>
> >>>>> >> Patch Set 26:
> >>>>> >> Build Successful
> >>>>> >>
> >>>>> >>
> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-fc25-x86_64/377/
> >>>>> >> : SUCCESS
> >>>>> >>
> >>>>> >>
> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-on-demand-el7-x86_64/428/
> >>>>> >> : SUCCESS
> >>>>> >>
> >>>>> >>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> >
> >>>>> >>> >>
> >>>>> >>> >> Link to Job:
> >>>>> >>> >>
> >>>>> >>> >>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/
> >>>>> >>> >>
> >>>>> >>> >> Link to logs:
> >>>>> >>> >>
> >>>>> >>> >>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2397/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
> >>>>> >>> >>
> >>>>> >>> >> Error snippet from log:
> >>>>> >>> >>
> >>>>> >>> >> <error>
> >>>>> >>> >>
> >>>>> >>> >> 2017-09-07 08:21:48,657-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Lock Acquired
> >>>>> >>> >> to object
> >>>>> >>> >>
> >>>>> >>> >>
> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
> >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
> >>>>> >>> >> sharedLocks='[]'}'
> >>>>> >>> >> 2017-09-07 08:21:48,675-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Running
> >>>>> >>> >> command: AddVmTemplateCommand internal: true. Entities
> affected
> >>>>> >>> >> : ID:
> >>>>> >>> >> d279c4e9-09e7-4dd9-9eff-10b31ee2adfc Type: StoragePoolAction
> >>>>> >>> >> group
> >>>>> >>> >> CREATE_TEMPLATE with role type USER
> >>>>> >>> >> 2017-09-07 08:21:48,695-04 INFO
> >>>>> >>> >>
> >>>>> >>> >>
> [org.ovirt.engine.core.bll.storage.disk.CreateAllTemplateDisksCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Running
> >>>>> >>> >> command: CreateAllTemplateDisksCommand internal: true.
> >>>>> >>> >> 2017-09-07 08:21:48,722-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> transaction
> >>>>> >>> >> rolled back
> >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Command
> >>>>> >>> >> 'org.ovirt.engine.core.bll.AddVmTemplateCommand' failed: null
> >>>>> >>> >> 2017-09-07 08:21:48,722-04 ERROR
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Exception:
> >>>>> >>> >> java.lang.NullPointerException
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getVramMultiplier(VgamemVideoSettings.java:63)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VgamemVideoSettings.getQxlVideoDeviceSettings(VgamemVideoSettings.java:40)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSettings(VideoDeviceSettings.java:34)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VideoDeviceSettings.getVideoDeviceSpecParams(VideoDeviceSettings.java:49)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VmDeviceUtils.getVideoDeviceSpecParams(VmDeviceUtils.java:586)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1435)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >> at
> >>>>> >>> >>
> >>>>> >>> >>
> org.ovirt.engine.core.bll.utils.VmDeviceUtils.copyVmDevices(VmDeviceUtils.java:1559)
> >>>>> >>> >> [bll.jar:]
> >>>>> >>> >>
> >>>>> >>> >> ...
> >>>>> >>> >>
> >>>>> >>> >> 2017-09-07 08:21:48,739-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Command
> >>>>> >>> >> [id=3d633182-982a-46f2-b4f1-27c5d04397a7]: Compensating
> >>>>> >>> >> NEW_ENTITY_ID of
> >>>>> >>> >> org.ovirt.engine.core.common.businessentities.VmTemplate;
> >>>>> >>> >> snapshot:
> >>>>> >>> >> 2c2b56b5-cac6-469d-b0e0-9744309ca8c1.
> >>>>> >>> >> 2017-09-07 08:21:48,813-04 ERROR
> >>>>> >>> >>
> >>>>> >>> >>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> EVENT_ID:
> >>>>> >>> >> USER_ADD_VM_TEMPLATE_FAILURE(36), Failed creating Template
> >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template.
> >>>>> >>> >> 2017-09-07 08:21:48,834-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.bll.AddVmTemplateCommand]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Lock freed to
> >>>>> >>> >> object
> >>>>> >>> >>
> >>>>> >>> >>
> 'EngineLock:{exclusiveLocks='[2c2b56b5-cac6-469d-b0e0-9744309ca8c1=TEMPLATE,
> >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template=TEMPLATE_NAME]',
> >>>>> >>> >> sharedLocks='[]'}'
> >>>>> >>> >> 2017-09-07 08:21:48,848-04 DEBUG
> >>>>> >>> >>
> >>>>> >>> >>
> [org.ovirt.engine.core.dal.dbbroker.CustomSQLErrorCodeSQLExceptionTranslator]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Translating
> >>>>> >>> >> SQLException with SQL state '23503', error code '0', message
> >>>>> >>> >> [ERROR: insert
> >>>>> >>> >> or update on table "disk_vm_element" violates foreign key
> >>>>> >>> >> constraint
> >>>>> >>> >> "fk_disk_vm_element_vm_static"
> >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
> is
> >>>>> >>> >> not present
> >>>>> >>> >> in table "vm_static".
> >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
> >>>>> >>> >> disk_id,
> >>>>> >>> >> vm_id,
> >>>>> >>> >> is_boot,
> >>>>> >>> >> pass_discard,
> >>>>> >>> >> disk_interface,
> >>>>> >>> >> is_using_scsi_reservation)
> >>>>> >>> >> VALUES (
> >>>>> >>> >> v_disk_id,
> >>>>> >>> >> v_vm_id,
> >>>>> >>> >> v_is_boot,
> >>>>> >>> >> v_pass_discard,
> >>>>> >>> >> v_disk_interface,
> >>>>> >>> >> v_is_using_scsi_reservation)"
> >>>>> >>> >> PL/pgSQL function
> >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
> >>>>> >>> >> varying,boolean) line 3 at SQL statement]; SQL was [{call
> >>>>> >>> >> insertdiskvmelement(?, ?, ?, ?, ?, ?)}] for task
> >>>>> >>> >> [CallableStatementCallback]
> >>>>> >>> >> 2017-09-07 08:21:48,850-04 INFO
> >>>>> >>> >> [org.ovirt.engine.core.utils.transaction.TransactionSupport]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> transaction
> >>>>> >>> >> rolled back
> >>>>> >>> >> 2017-09-07 08:21:48,877-04 INFO
> >>>>> >>> >>
> >>>>> >>> >>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> EVENT_ID:
> >>>>> >>> >> USER_IMPORT_IMAGE_AS_TEMPLATE_FINISHED_SUCCESS(3,018), User
> >>>>> >>> >> admin at internal-authz successfully imported image
> >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template as template
> >>>>> >>> >> CirrOS_0.3.5_for_x86_64_glance_template to domain iscsi.
> >>>>> >>> >> 2017-09-07 08:21:48,878-04 ERROR
> >>>>> >>> >> [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-21) [74b41edb]
> >>>>> >>> >> Failed
> >>>>> >>> >> invoking callback end method 'onSucceeded' for command
> >>>>> >>> >> '9c1ba73a-f6bb-4291-b1f2-d5806e38ba18' with exception
> >>>>> >>> >> 'CallableStatementCallback; SQL [{call insertdiskvmelement(?,
> ?,
> >>>>> >>> >> ?, ?, ?,
> >>>>> >>> >> ?)}]; ERROR: insert or update on table "disk_vm_element"
> >>>>> >>> >> violates foreign
> >>>>> >>> >> key constraint "fk_disk_vm_element_vm_static"
> >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
> is
> >>>>> >>> >> not present
> >>>>> >>> >> in table "vm_static".
> >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
> >>>>> >>> >> disk_id,
> >>>>> >>> >> vm_id,
> >>>>> >>> >> is_boot,
> >>>>> >>> >> pass_discard,
> >>>>> >>> >> disk_interface,
> >>>>> >>> >> is_using_scsi_reservation)
> >>>>> >>> >> VALUES (
> >>>>> >>> >> v_disk_id,
> >>>>> >>> >> v_vm_id,
> >>>>> >>> >> v_is_boot,
> >>>>> >>> >> v_pass_discard,
> >>>>> >>> >> v_disk_interface,
> >>>>> >>> >> v_is_using_scsi_reservation)"
> >>>>> >>> >> PL/pgSQL function
> >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
> >>>>> >>> >> varying,boolean) line 3 at SQL statement; nested exception is
> >>>>> >>> >> org.postgresql.util.PSQLException: ERROR: insert or update on
> >>>>> >>> >> table
> >>>>> >>> >> "disk_vm_element" violates foreign key constraint
> >>>>> >>> >> "fk_disk_vm_element_vm_static"
> >>>>> >>> >> Detail: Key (vm_id)=(2c2b56b5-cac6-469d-b0e0-9744309ca8c1)
> is
> >>>>> >>> >> not present
> >>>>> >>> >> in table "vm_static".
> >>>>> >>> >> Where: SQL statement "INSERT INTO disk_vm_element (
> >>>>> >>> >> disk_id,
> >>>>> >>> >> vm_id,
> >>>>> >>> >> is_boot,
> >>>>> >>> >> pass_discard,
> >>>>> >>> >> disk_interface,
> >>>>> >>> >> is_using_scsi_reservation)
> >>>>> >>> >> VALUES (
> >>>>> >>> >> v_disk_id,
> >>>>> >>> >> v_vm_id,
> >>>>> >>> >> v_is_boot,
> >>>>> >>> >> v_pass_discard,
> >>>>> >>> >> v_disk_interface,
> >>>>> >>> >> v_is_using_scsi_reservation)"
> >>>>> >>> >> PL/pgSQL function
> >>>>> >>> >> insertdiskvmelement(uuid,uuid,boolean,boolean,character
> >>>>> >>> >> varying,boolean) line 3 at SQL statement', the callback is
> >>>>> >>> >> marked for end
> >>>>> >>> >> method retries
> >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
> >>>>> >>> >>
> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
> >>>>> >>> >> Command
> >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
> >>>>> >>> >> execution didn't
> >>>>> >>> >> complete, not proceeding to perform the next operation
> >>>>> >>> >> 2017-09-07 08:21:49,887-04 INFO
> >>>>> >>> >>
> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
> >>>>> >>> >> (EE-ManagedThreadFactory-engineScheduled-Thread-49) [74b41edb]
> >>>>> >>> >> Command
> >>>>> >>> >> 'AddVmTemplate' id: '3d633182-982a-46f2-b4f1-27c5d04397a7'
> child
> >>>>> >>> >> commands
> >>>>> >>> >> '[d168e950-e9c5-4c9b-9502-bff613d7f14d]' executions were
> >>>>> >>> >> completed, status
> >>>>> >>> >> 'FAILED'
> >>>>> >>> >>
> >>>>> >>> >>
> >>>>> >>> >> </error>
> >>>>> >>> >>
> >>>>> >>> >> --
> >>>>> >>> >> Regards,
> >>>>> >>> >> Evgheni Dereveanchin
> >>>>> >>> >>
> >>>>> >>> >> _______________________________________________
> >>>>> >>> >> Devel mailing list
> >>>>> >>> >> Devel at ovirt.org
> >>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/devel
> >>>>> >>> _______________________________________________
> >>>>> >>> Devel mailing list
> >>>>> >>> Devel at ovirt.org
> >>>>> >>> http://lists.ovirt.org/mailman/listinfo/devel
> >>>>> >>>
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >>
> >>>>> >> Eyal edri
> >>>>> >>
> >>>>> >>
> >>>>> >> ASSOCIATE MANAGER
> >>>>> >>
> >>>>> >> RHV DevOps
> >>>>> >>
> >>>>> >> EMEA VIRTUALIZATION R&D
> >>>>> >>
> >>>>> >>
> >>>>> >> Red Hat EMEA
> >>>>> >>
> >>>>> >> TRIED. TESTED. TRUSTED.
> >>>>> >> phone: +972-9-7692018 <+972%209-769-2018>
> >>>>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>>>> >>
> >>>>> >> _______________________________________________
> >>>>> >> Devel mailing list
> >>>>> >> Devel at ovirt.org
> >>>>> >> http://lists.ovirt.org/mailman/listinfo/devel
> >>>>> >
> >>>>> >
> >>>>> _______________________________________________
> >>>>> Devel mailing list
> >>>>> Devel at ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
> >>>
> >>>
> >>> _______________________________________________
> >>> Devel mailing list
> >>> Devel at ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/devel
> >>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170911/4b1e00b7/attachment-0001.html>
More information about the Devel
mailing list