- Devel - oVirt List Archives

Testing oVirt with Gluster 6
by Simone Tiraboschi 13 May '19

13 May '19

Hi, according to https://bugzilla.redhat.com/show_bug.cgi?id=1673058 the network throughput usage increases by a x5 upgrading from Gluster 3.12 to Gluster 5.y. This seems definitively problematic on our hyper-converged environments: I read more than one report where, over 1 Gbps network, sanlock could not be refreshed quickly enough and so killed the engine VM if not rebooted the host due to the watchdog. This is probably due to the fact that, according to that bug, the network got completely saturated. Faster networks seems still fine. https://bugzilla.redhat.com/show_bug.cgi?id=1673058 is probably going to be solved for Gluster 5.6. On the other side we have https://bugzilla.redhat.com/1693998 <https://bugzilla.redhat.com/show_bug.cgi?id=1693998> about rebasing over Gluster 6 which seems not affected by that issue. Do you have concrete plans about testing oVirt with Gluster 6? thanks, Simone

5 7

Build artifacts failed - not enough free space on file system
by Nir Soffer 12 May '19

12 May '19

I had this failure in yum install: [2019-05-12T14:54:55.706Z] Error Summary [2019-05-12T14:54:55.706Z] ------------- [2019-05-12T14:54:55.706Z] Disk Requirements: [2019-05-12T14:54:55.706Z] At least 70MB more space needed on the / filesystem. Build: https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… Trying build again, hopefully will get a slave with more space... Nir

2 4

Re: [CQ]: 99533,1 (ovirt-vmconsole) failed "ovirt-4.2" system tests
by Eyal Edri 12 May '19

12 May '19

Adding Galit. Not sure why it requires new image, can you elaborate? On Fri, May 10, 2019, 17:10 Dafna Ron <dron(a)redhat.com> wrote: > we may need new lago images as I think the selinux conf is overwritten > when we upgrade newer packages. > > > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/4323/testReport/… > > On Fri, May 10, 2019 at 2:25 PM oVirt Jenkins <jenkins(a)ovirt.org> wrote: > >> Change 99533,1 (ovirt-vmconsole) is probably the reason behind recent >> system >> test failures in the "ovirt-4.2" change queue and needs to be fixed. >> >> This change had been removed from the testing queue. Artifacts build from >> this >> change will not be released until it is fixed. >> >> For further details about the change see: >> https://gerrit.ovirt.org/#/c/99533/1 >> >> For failed test results see: >> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/4323/ >> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/NI7TPHGBKRGWI… >> > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/VDWLW6XCGNAGQ… >

2 3

[Ovirt] [CQ weekly status] [10-05-2019]
by Dafna Ron 10 May '19

10 May '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: RED (#1) mom is failing on package dependency: mom-0.5.12-0.0.master.el7.noarch requires python2-libvirt. The package is a fc package and should not be a dependency for centos. This was introduced with this patch: https://gerrit.ovirt.org/#/c/99791/ Sandro has been notified and he will have a look once he finishes handling more urgent issues. *CQ-4.3*: RED (#1) Last Failure was yesterday for mom. its the same issue as in 4.2 *CQ-Master:* RED (#1) we have a failure in vdsm project. mail was reported to the list and being investigated further by Marcin and Milan. The issue seems to be in the upgrade from release suite in run_vm test (RPC call VM.create failed (error 16)) patch submitted to fix https://gerrit.ovirt.org/#/c/99899/ The random failures seemed to have been solved by Galit and help from developers. I will continue to monitor and report in case we have further failures Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

2 1

[ OST Failure Report ] [ oVirt Master (vdsm) ] [ 10-05-2019 ] [ 004_basic_sanity.vm_run ]
by Dafna Ron 10 May '19

10 May '19

Hi, We are failing upgrade-from-release-suite.el7.x86_64 / 004_basic_sanity.vm_run The issue is an unexpected exception in vdsm. root cause based on CQ is this patch: https://gerrit.ovirt.org/#/c/99854/ - yum: Allow downloading only 'vdsm' package Logs can be found here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14079/artifac… Marcin, can you please take a look? error: vdsm: 2019-05-10 05:06:38,329-0400 ERROR (jsonrpc/0) [api] FINISH create error=Unexpected exception (api:131) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 245, in create raise exception.UnexpectedError() UnexpectedError: Unexpected exception 2019-05-10 05:06:38,330-0400 INFO (jsonrpc/0) [api.virt] FINISH create return={'status': {'message': 'Unexpected exception', 'code': 16}} from=::ffff:192.168.201.2,39486, flow_id=83e4f0c4-a39a-45aa-891f-4022765e1a87, vmId=397013a0-b7d4-4d38-86c3-e944cebd75a7 (api:54) 2019-05-10 05:06:38,331-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.create failed (error 16) in 0.00 seconds (__init__:312) 2019-05-10 05:06:38,629-0400 INFO (jsonrpc/7) [vdsm.api] FINISH getStorageDomainInfo return={'info': {'uuid': '24498263-8985-46a9-9161-f65ee776cb7f', 'vgMetadataDevice': '360014051f333158820a4cc6ab3be5b55', 'vguuid': 'rXTLNc-JwEz-0dtL-Z2EC-ASqn-gzkg-jDJa1b', 'metadataDevice': '360014051f333158820a4cc6ab3be5b55', 'state': 'OK', 'version': '4', 'role': 'Master', 'type': 'ISCSI', 'class': 'Data', 'pool': ['3e330122-c587-4078-b9a4-13dbb697c5cc'], 'name': 'iscsi'}} from=::ffff:192.168.201.2,39486, flow_id=3f76d32a, task_id=084e7ee5-3c43-4905-833f-9a522043a862 (api:54) engine: 2019-05-10 05:06:38,328-04 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] VDS::create Failed creating vm 'v m0' in vds = '566eda0f-3ef0-4791-a618-1e649af4b0be' error = 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to C reateBrokerVDS, error = Unexpected exception, code = 16' 2019-05-10 05:06:38,328-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] FINISH, CreateVDSCommand, return: Down, log id: 3cb7c62d 2019-05-10 05:06:38,328-04 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Failed to run VM 'vm0': EngineException: or g.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateBrokerVDS, error = Unexpected exception, code = 16 (Failed with error unexpected and code 16) 2019-05-10 05:06:38,328-04 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Lock freed to object 'EngineLock:{exclu siveLocks='[397013a0-b7d4-4d38-86c3-e944cebd75a7=VM]', sharedLocks=''}' 2019-05-10 05:06:38,329-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Trying to rerun VM 'vm0' 2019-05-10 05:06:38,373-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] EVENT_ID: USE R_INITIATED_RUN_VM_FAILED(151), Failed to run VM vm0 on Host lago-upgrade-from-release-suite-master-host-0. 2019-05-10 05:06:38,381-04 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Lock Acquired to object 'EngineLock:{ex clusiveLocks='[397013a0-b7d4-4d38-86c3-e944cebd75a7=VM]', sharedLocks=''}' 2019-05-10 05:06:38,393-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] START, IsVmDuringIn itiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='397013a0-b7d4-4d38-86c3-e944cebd75a7'}), log id: 96be328 2019-05-10 05:06:38,393-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] FINISH, IsVmDuringI nitiatingVDSCommand, return: false, log id: 96be328 2019-05-10 05:06:38,403-04 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS 2019-05-10 05:06:38,404-04 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] Lock freed to object 'EngineLock:{exclu siveLocks='[397013a0-b7d4-4d38-86c3-e944cebd75a7=VM]', sharedLocks=''}' 2019-05-10 05:06:38,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [83e4f0c4-a39a-45aa-891f-4022765e1a87] EVENT_ID: USE R_FAILED_RUN_VM(54), Failed to run VM vm0 (User: admin@internal-authz). 2019-05-10 05:06:38,462-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [There are no hosts to use. Check that the cluster contains at least one host in Up state.] 2019-05-10 05:06:38,466-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-35) [3f40b3e8] Running command: ProcessDownVmCommand internal: true. Thanks, Dafna

3 4

Subject: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 05-05-2019 ] [ upgrade_hosts ]
by Dafna Ron 10 May '19

10 May '19

Hi, We are failing test upgrade_hosts on upgrade-from-release-suite-master. From the logs I can see that we are calling migrate vm when we have only one host and the vm seem to have been shut down before the maintenance call is issued. Can you please look into this? suspected patch reported as root cause by CQ is: https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms command and use it for host maintenance logs are found here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artifac… I can see the issue is vm migration when putting host in maintenance: 2019-05-07 06:02:04,170-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [05592db2-f859-487b-b779-4b32eec5bab 3] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS 2019-05-07 06:02:04,215-04 WARN [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation of action 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND 2019-05-07 06:02:04,221-04 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to migrate one or more VMs. 2019-05-07 06:02:04,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. 2019-05-07 06:02:04,239-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock Acquired to object 'Eng ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', sharedLocks=''}' 2019-05-07 06:02:04,242-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running command: ActivateVds Command internal: true. Entities affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-05-07 06:02:04,243-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before acquiring lock in ord er to prevent monitoring for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,243-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock acquired, from now a mo nitoring of host will be skipped for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,252-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, SetVdsStatu sVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', status='Unassigned', n onOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c8aa211 2019-05-07 06:02:04,256-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, SetVdsStat usVDSCommand, return: , log id: 2c8aa211 2019-05-07 06:02:04,261-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate host finished. Lock released. Monitoring can run now for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,265-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN T_ID: VDS_ACTIVATE(16), Activation of host lago-upgrade-from-release-suite-master-host-0 initiated by admin@internal-authz. 2019-05-07 06:02:04,266-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed to object 'Engine Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', sharedLocks=''}' 2019-05-07 06:02:04,484-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [05592db2-f859-487b-b779-4b32 eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' failed to move to maintenance mode. Upgrade process is terminated. I can see there was only one vm running: drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu [dron@dron post-004_basic_sanity.py]$ ls -l lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ total 6 -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log and I can see that there was an attempt to terminate it with an error that it does not exist: stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', gracefully='false', reason='', ig noreNoVm='false'}), log id: 24278e9b 2019-05-07 06:01:41,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma nd(HostName = lago-upgrade-from-release-suite-master-host-0, DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a169 54db4abf', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 78bba2f8 2019-05-07 06:01:42,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm and, return: , log id: 78bba2f8 2019-05-07 06:01:42,090-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r eturn: , log id: 24278e9b 2019-05-07 06:01:42,094-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was reported as Down on VDS '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) 2019-05-07 06:01:42,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = lago-upgrade- from-release-suite-master-host-0, DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 ', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1dbd31eb 2019-05-07 06:01:42,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-4) [] Failed to destroy VM 'dfbd75e2-a9cb-4fca-8788-a16 954db4abf' because VM does not exist, ignoring

3 10

manipulating VM configuration via Python SDK disables serial console
by Denis Karpov 10 May '19

10 May '19

Hello, I have hit what seems to be a bug in oVIrt Python SDK, when deploying OpenStack 14 (Rocky) in configuration virtualized controllers, oVirt VMs; Ironic oVirt driver is used in this case, which uses oVirt Python SDK to talk to manipulate VMs. The driver seem to work in such a way that at certain stages it retrieves a VM configuration and updates it (e.g. changes the boot device). However during such update some other changes in configuration of the VM happen: e.g. if the VM had VirtIO serial console enabled, after the update that serial console becomes disabled. Perhaps when a VMs config is updated some XML fields of its configuration are not written back as they should be? We have noticed that since the OpenStack deployment fails on controllers if it cannot output to the serial console. I would appreciate any help/workaround for this issue. Thanks, Denis

2 2

SATA interface?
by Hetz Ben Hamo 09 May '19

09 May '19

Hi, It looks like the Q35 chipset has SATA support for CDROM, but not for virtual hard disk. In many OS's/Appliances - there is no way to use virtio disks, and I can either use the extremly slow IDE with 440FX or nothing else. on Virt-Manager though, you can choose a SATA hard disk. So, is it possible to add the SATA support back for hard drives please?

3 3

manipulating VM configuration via Python SDK disables serial console
by Denis Karpov 08 May '19

08 May '19

Hello, I have hit what seems to be a bug in oVIrt Python SDK, when deploying OpenStack 14 (Rocky) in configuration virtualized controllers, oVirt VMs; Ironic oVirt driver is used in this case, which uses oVirt Python SDK to talk to manipulate VMs. The driver seem to work in such a way that at certain stages it retrieves a VM configuration and updates it (e.g. changes the boot device). However during such update some other changes in configuration of the VM happen: e.g. if the VM had VirtIO serial console enabled, after the update that serial console becomes disabled. Perhaps when a VMs config is updated some XML fields of its configuration are not written back as they should be? We have noticed that since the OpenStack deployment fails on controllers if it cannot output to the serial console. I would appreciate any help/workaround for this issue. Thanks, Denis

1 0

Fedora 29 support?
by Nir Soffer 08 May '19

08 May '19

Fedora 28 will be EOL in few month, and at this point people are migration to Fedora 29. I tried yesterday to build engine development setup on Fedora 29 VM, and found that it does not work: - some packages are missing on engine side - after hacking the ovirt.repo files, pointing to "28" instead of "$releasever", I got the packages... - but then adding a host fail with "Python is missing" on the host, which is working Fedora 28 host used with engine before. What are our plans for supporting Fedora 29 on engine and host? I hope that the work we did for Fedora 28 will make it easier to port to Fedora 29. Nir

3 5