we would like to ask about interest in community about oVirt moving to CentOS Stream.
There were some requests before but it’s hard to see how many people would really like to see that.
With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS.
So..would you like us to support CentOS Stream?
We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS?
There are some concerns about Stream being a bit less stable, do you share those concerns?
Thank you for your comments,
we are defining a device migration compatibility interface that helps upper
layer stack like openstack/ovirt/libvirt to check if two devices are
live migration compatible.
The "devices" here could be MDEVs, physical devices, or hybrid of the two.
e.g. we could use it to check whether
- a src MDEV can migrate to a target MDEV,
- a src VF in SRIOV can migrate to a target VF in SRIOV,
- a src MDEV can migration to a target VF in SRIOV.
(e.g. SIOV/SRIOV backward compatibility case)
The upper layer stack could use this interface as the last step to check
if one device is able to migrate to another device before triggering a real
live migration procedure.
we are not sure if this interface is of value or help to you. please don't
hesitate to drop your valuable comments.
(1) interface definition
The interface is defined in below way:
/ read \
| migration_version | | migration_version |-->check migration
--------------------- --------------------- compatibility
device A device B
a device attribute named migration_version is defined under each device's
sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version).
userspace tools read the migration_version as a string from the source device,
and write it to the migration_version sysfs attribute in the target device.
The userspace should treat ANY of below conditions as two devices not compatible:
- any one of the two devices does not have a migration_version attribute
- error when reading from migration_version attribute of one device
- error when writing migration_version string of one device to
migration_version attribute of the other device
The string read from migration_version attribute is defined by device vendor
driver and is completely opaque to the userspace.
for a Intel vGPU, string format can be defined like
"parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count".
for an NVMe VF connecting to a remote storage. it could be
"PCI ID" + "driver version" + "configured remote storage URL"
for a QAT VF, it may be
"PCI ID" + "driver version" + "supported encryption set".
(to avoid namespace confliction from each vendor, we may prefix a driver name to
each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1)
The reason we hope the migration_version string is opaque to the userspace
is that it is hard to generalize standard comparing fields and comparing
methods for different devices from different vendors.
Though userspace now could still do a simple string compare to check if
two devices are compatible, and result should also be right, it's still
too limited as it excludes the possible candidate whose migration_version
string fails to be equal.
e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible
with another MDEV with mdev_type_3, aggregator count 1, even their
migration_version strings are not equal.
(assumed mdev_type_3 is of 3 times equal resources of mdev_type_1).
besides that, driver version + configured resources are all elements demanding
to take into account.
So, we hope leaving the freedom to vendor driver and let it make the final decision
in a simple reading from source side and writing for test in the target side way.
we then think the device compatibility issues for live migration with assigned
devices can be divided into two steps:
a. management tools filter out possible migration target devices.
Tags could be created according to info from product specification.
we think openstack/ovirt may have vendor proprietary components to create
those customized tags for each product from each vendor.
for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to
search target vGPU are like:
a tag for compatible parent PCI IDs,
a tag for a range of gvt driver versions,
a tag for a range of mdev type + aggregator count
for NVMe VF, the tags to search target VF may be like:
a tag for compatible PCI IDs,
a tag for a range of driver versions,
a tag for URL of configured remote storage.
b. with the output from step a, openstack/ovirt/libvirt could use our proposed
device migration compatibility interface to make sure the two devices are
indeed live migration compatible before launching the real live migration
process to start stream copying, src device stopping and target device
It is supposed that this step would not bring any performance penalty as
-in kernel it's just a simple string decoding and comparing
-in openstack/ovirt, it could be done by extending current function
check_can_live_migrate_destination, along side claiming target resources.
We are currently trying to make oVirt work on ARM64 platform, since I'm
quite new to oVirt community, I'm wondering what is the current status
about ARM64 support in the oVirt upstream, as I saw the oVirt Wikipedia
page mentioned there is an ongoing efforts to support ARM platform. We have
a small team here and we are willing to also help to make this work.
Lookup for el7 repo fails:
[2020-06-22T07:46:48.800Z] ERROR: Command failed:
[2020-06-22T07:46:48.800Z] # /usr/bin/yum --installroot
--releasever 7 install yum yum-utils --setopt=tsflags=nocontexts
[2020-06-22T07:46:48.800Z] Failed to set locale, defaulting to C
[Errno 14] HTTPS Error 404 - Not Found
[2020-06-22T07:46:48.800Z] Trying other mirror.
[2020-06-22T07:46:48.800Z] To address this issue please refer to the below
So in an attempt to fix change CD functionality we discovered a few other potential issues and what Nir suggested was to implement two [somewhat] new functions in VDSM: hotplug and hotunplug for CDs similar to how it works for normal disks now. Existing changeCD function will be left as is for backwards compatibility.
As I found out, engine already calculates iface and index before invoking VDSM functions, so we will just pass these along with PDIV to the VDSM.
Suggested flow is, let me quote:
>So the complete change CD flow should be:
>1. get the previous drivespec from vm metadata
>2. prepare new drivespec
>3. add new drivespec to vm metadata
>4. attach a new device to vm
>5. teardown the previous drivespec
>6. remove previous drivespec from vm metadata
>When the vm is stopped, it must do:
>1. get drive spec from vm metadata
>2. teardown drivespec
>During attach, there are interesting races:
>- what happens if vdsm crashes after step 2? who will teardown the volume?
> maybe you need to add the new drivespec to the metadata first,
>before preparing it.
>- what happens if attach failed? who will remove the new drive from
Now, what makes hotplugDisk/hotunplugDisk different? From what I understand, the flow is same there, so what difference is there as far as VDSM is concerned? If none, this means if I more or less copy that code, changing minor details and data accordingly for CDs, this should work, shouldn't it?
In OVirt, we have a property propagate_error at the disk level that
decides in case of an error, how this error be propagated to the VM.
This value is maintained in the database table with the default value
set as Off. The default setting(Off) results in a policy that ends up
pausing the VM rather than propagating the errors to VM. There is no
provision in the UI currently to configure this property for disk
(images or luns). So there is no easy way to set this value. Further,
even if the value is manually set to "On" in db, it gets overwriiten by
UI everytime some other property is updated as described here -
Setting the value to "Off" is not ideal for multipath devices where a
single path failure causes vm to pause. It puts serious restrictions for
the DR situation and unlike VMWare * Hyper-V, oVirt is not able to
support the DR functionality -
While we wait for RFE, the proposal here is to revise the out of the box
behavior for LUNs. For LUNs, we should propagate the errors to VM rather
than directly stopping those. This will allow us to handle short-term
multipath outages and improve availability. This is a simple change in
behavior but will have good positive impact. I would like to seek
feedback about this to make sure that everyone is ok with the proposal.
I could not merge:
Although it was approved and verified, and got +1 from Continuous Integration.
There is a new "RHEL-Continuous-Integration", which does not run with
even when I trigger the tests manually with "ci test".
I tried to add +1 for "RHEL-Continuous-Integration" but this is not
possible, I see:
RHEL-Continuous-Integration You don't have permission to edit this label.
So finally I downloaded the patch and pushed it manually.
I think this patch will fix the problem:
But I need CI experts to review this. Since we have 50 projects that need this,
this probably should be fixed elsewhere, and inherited by all projects.
I know that Germano was not able to trigger tests because he was missing in the
jenkins whitelist, but this was fixed last week.
Germano, maybe just to check that everything works for you, you can post some
i have attached my ovirt zip. the issue im facing is that the kickstart is
not generated for el8 and im not able to figure out how to use .j2 file
that is being generated. i've manage to frankenstein a kickstart file that
is in the zip with all the settings but its also giving error while
compressing (dracut module 'livenet' cannot be found or installed) while if
i search installed modules of dracut i can see livenet is installed. see
see my zip it have logs of last compilation also.
sir/mam it would be a great help for my college final year project project
on hci and my research paper.
please sir/mam please help me
communication (including any
attachment/s) is intended only for the use of the
addressee(s) and contains
information that is PRIVILEGED AND CONFIDENTIAL.
dissemination, distribution, or copying of this
Any views or
opinions presented in
this email are solely those of the author and do not
those of the University.
any attachments, may not be free of viruses,
interceptions or interference,
and may not be compatible with your systems. You
should carry out your own
virus checks before opening any attachment to this
e-mail. The sender of
this e-mail and Galgotias University shall not be liable
for any damage
that you may sustain as a result of viruses, incompleteness of
message, a delay in receipt of this message or computer problems
FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, saving 900MB of data transferred all the time (and copied back and forth during test) as well as lower memory requirements (down to 128MB from 384MB, times 3 VMs).
For a good measure a memory hot unplug test was added too.
Let me know if you see anything unusual, other than faster run times.
When I'm trying to add a storage domain via the webadmin I'm getting the
error message that's on the subject.
in the vdsm.log I see this error:
2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
523, in _pathChecked
delay = result.delay()
File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391,
raise exception.MiscFileReadException(self.path, self.rc, self.err)
vdsm.storage.exception.MiscFileReadException: Internal file read failure:
1, bytearray(b"/usr/bin/dd: failed to open
No such file or directory\n"))
when looking at the directory I do see that metadata was created under this
directory, with the following content:
[delfassy@delfassy mnt]$ cd
[delfassy@delfassy dom_md]$ less metadata
The permissions on the directory are:
drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
Are there any other configurations I'm missing? Any idea how this issue can