Re: Image Transfer mechanism queries/API support
by Ranjit DSouza
Hello oVirt Development team
Not sure you got a chance to see this email, resending it.
Thanks
Ranjit
From: Ranjit DSouza
Sent: Monday, July 9, 2018 1:16 PM
To: 'Nir Soffer' <nsoffer(a)redhat.com>
Cc: devel <devel(a)ovirt.org>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com>; Navin Tah <Navin.Tah(a)veritas.com>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com>; Nisan, Tal <tnisan(a)redhat.com>; Daniel Erez <derez(a)redhat.com>
Subject: RE: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support
Hi Nir
Thanks for getting back!
We have few follow up questions:
1. On the new ‘Allocated extents API’:
Can you share the release timeline for 4.2.z? From the link below it seems like it will be available on 7/30/2018.
https://www.ovirt.org/develop/release-management/releases/4.2.z/release-m...
However, we thought we would double check on this.
2. If this will be available in 4.2.z, does It mean, we can assume it will be backported to 4.3 also?
3. When we downloaded the snapshot disk using Image Transfer API, the resulting format of the disk is “raw”. However, for upload, we must upload a qcow2 disk (to enable further snapshots).
It means, we need to convert it first using qemu-img convert. Or is there a way we directly ask via API for a qcow2 instead?
Portion of the response of “GET /storagedomains/{storagedomain:id}/disksnapshots”
"format" : "raw",
"shareable" : "false",
"sparse" : "true",
"status" : "ok",
"snapshot" : {
"id" : "4756036e-92aa-4ebb-ae4b-052a30cd5109"
},
"actual_size" : "1345228800",
"content_type" : "data",
"propagate_errors" : "false",
"provisioned_size" : "21474836480",
"storage_type" : "image",
"total_size" : "0",
"wipe_after_delete" : "false",
4. When downloading a snapshot in chunks, Is there any recommended chunk size? For our study we used 512 KB. Checked the documentation too.
5. Regarding your ask on ‘Can you file RFE for this, explaining the use case and the current performance’.
Since you do not recommend direct access to NFS storage, we will consider it only if we see significant performance degradation using http download.
Also, if time permits, we may check if there is a significant performance benefit using Unix socket, over http download (via Rest API).
But as it stands now, getting the allocated extents support (soon), will alleviate most of our performance concerns.
Thanks
Ranjit
From: Nir Soffer [mailto:nsoffer@redhat.com]
Sent: Tuesday, July 3, 2018 4:29 PM
To: Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>>
Cc: devel <devel(a)ovirt.org<mailto:devel@ovirt.org>>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com<mailto:DL-VTAS-ENG-NBU-EverestFalcons@veritas.com>>; Navin Tah <Navin.Tah(a)veritas.com<mailto:Navin.Tah@veritas.com>>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com<mailto:Sudhakar.Paulzagade@veritas.com>>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com<mailto:ylavi@redhat.com>>; Nisan, Tal <tnisan(a)redhat.com<mailto:tnisan@redhat.com>>; Daniel Erez <derez(a)redhat.com<mailto:derez@redhat.com>>
Subject: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support
On Tue, Jul 3, 2018 at 11:32 AM Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>> wrote:
...
We had a conversation with Pavan Chavva for supporting RHV. He had suggested to contact you with queries related to oVirt APIs we plan to use.
We have following queries:
1. While downloading a snapshot disk, can we identify allocated extents and download only those using oVirt API? We are able to download the disk using the Image Transfer API mechanism.
However, this method downloads the entire disk including the non-allocated extents, which is a performance overhead. If this functionality does not exist at this point will it be available in near future?
Hi Ranjit,
There is no way to do this in current 4.2, but we plan to introduce in in 4.2.z.
The API will be something like:
GET /images/xxx-yyy/map
...
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0},
{ "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536},
{ "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576},
{ "start": 1114112, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 1114112},
...
{ "start": 5465571328, "length": 22675456, "depth": 0, "zero": false, "data": true, "offset": 5465571328},
{ "start": 5488246784, "length": 954138624, "depth": 0, "zero": true, "data": false, "offset": 5488246784},
{ "start": 6442385408, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 6442385408}]
This is basically what you get using qemu-img map.
You can test play this with this using:
virt-builder Fedora-27 -o /var/tmp/fedora-27.img
qemu-img map -f raw --output json /var/tmp/fedora-27.img
This is the first data segment:
{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0}
This is a hole between the first data segment and the second:
{ "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536}
This is the second data segment:
{ "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576}
Based on this output, you will be able to get the allocated parts of the image using:
Request:
GET /image/xxx-yyy HTTP/1.1
Range: bytes=0-65535
Response:
HTTP/1.1 206 Partial Content
Content-Range: bytes 0-65535/6442450944
<data of first segment>
Request:
GET /image/xxx-yyy HTTP/1.1
Range: bytes=1048576-1114111
Response:
HTTP/1.1 206 Partial Content
Content-Range: bytes 1048576-1114111/6442450944
<data of second segment>
And so on.
If you create a sparse file on your backup media, and download and write
the data segments at the correct offset, you will get the a sparse version of
the image as on the server side.
This will work for raw or qcow2 images on NFS >= 4.2, or for qcow2 images on block
storage.
For older NFS versions, or raw images on block storage, we can solve the issue by
reading the entire image and detecting zeroes - which is quite expensive, so I'm not
sure we will implement this, maybe it will be done later.
We have experimental patch using special sparse format, that can support this use
case, downloading entire image in one pass. This commit message explain the
format:
https://gerrit.ovirt.org/#/c/85413/12//COMMIT_MSG
For more info on using random I/O APIs, see:
http://ovirt.github.io/ovirt-imageio/random-io
(available since 4.2.3)
For example code uploading sparse images see:
https://github.com/oVirt/ovirt-imageio/blob/master/examples/upload
For best performance, you should run your application on a oVirt host, using unix
socket to communicate with imageio. See:
http://ovirt.github.io/ovirt-imageio/unix-socket
(will be available in 4.2.5)
All this will work only for the non-active layer in a qcow2 chain. We are working now
on incremental backup which will allow the same for the active layer with a running
vm. This is expected in 4.3, and we may have a tech preview at some point in 4.2.z.
Incremantal backup will use the similar API, allowing detection of dirty parts of an image,
so you can download only the data that was changed since the last backup.
Please watch and comment on the feature page:
https://ovirt.org/develop/release-management/features/storage/incremental...
We are also considering exposing images using NBD. This will allow downloading
and uploading images using qemu-img from any host. This work depends on TLS-PSK
support in qemu-img and qemu-nbd. You can follow this work here:
https://lists.nongnu.org/archive/html/qemu-devel/2018-06/threads.html#08491
2. Is there an alternate method to transfer a snapshot to and from RHV storage? Are there other methods such as NFS share where we can download snapshot image to and from RHV storage?
We don't support direct access to storage by 3rd party. You should use imageio API.
Can you file RFE for this, explaning the use case and the current performance
issues you experience?
Nir
5 years, 8 months
new storage domain question
by Hetz Ben Hamo
Hi,
I just found something very interesting.
While installing latest ovirt, I've created the shared_storage on my nas1
share folder (NFS). So far, so good.
However, when trying to create additional 2 new storage domains, I can only
create 1 (which means the shared_storage is 1 and a new one is 2, I cannot
create a 3rd storage domain, 4th etc..). When I try, it shows an error (I
don't remember which error right now).
Why is that? and why this strange (IMHO) shared_storage thing? in a
corporate, if I want to set up a virtualized environment, the storage
administrator asks me which one I want (NFS, or an iSCSI LUN), do I need
one or more, and then I'm adding them as a datastore (in VMWare, for
example) and on top of that I'm installing the management software (for
example, vCenter). with oVirt method, it will confuse VMware admins.
Thanks
5 years, 8 months
planned Jenkins restart
by Evgheni Dereveanchin
Hi everyone,
I'll be performing a planned Jenkins restart within the next hour.
No new jobs will be scheduled during this maintenance period.
I will inform you once it is over.
Regards,
Evgheni Dereveanchin
5 years, 8 months
Is using CompensationContext in child commands broken?
by Andrej Krejcir
Hi,
I'm changing UpdateClusterCommand to be non-transactive,
and it would use CompensationContext for reverting DB operations.
Reading through CommandBase code, I noticed the following flow seems to be
broken:
1. A command without transaction is executed.
2. It executes a child command, with the compensation context of the
current command. So that the result of the child command can be reverted if
the parent fails.
3. Once the child command successfully finishes, it removes all entity
snapshots from the DB (it calls
CommandContext::cleanupCompensationDataAfterSuccessfulCommand), even if the
snapshots were made by the parent command. See code at [1] and [2].
4. The parent command then fails for some other reason, but its
compensation context is empty, so the DB operations are not reverted
correctly.
Am I missing something or is this flow broken?
Thanks,
Andrej
[1] -
https://github.com/oVirt/ovirt-engine/blob/31fbb4cb48c7208e6b37c5c0e27560...
[2] -
https://github.com/oVirt/ovirt-engine/blob/31fbb4cb48c7208e6b37c5c0e27560...
5 years, 8 months
Fwd: Fedora 30 System-Wide Change proposal: Mass Python 2 Package Removal
by Sandro Bonazzola
FYI
---------- Forwarded message ----------
From: Ben Cotton <bcotton(a)redhat.com>
Date: 2018-08-20 22:41 GMT+02:00
Subject: Fedora 30 System-Wide Change proposal: Mass Python 2 Package
Removal
To: devel(a)lists.fedoraproject.org, devel-announce(a)lists.fedoraproject.org
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
== Summary ==
(Sub-)packages only providing python2 importable modules without
additional functionality will be removed from Fedora unless some other
package(s) depends on them.
Python 2 will be deprecated in Fedora. Packagers can mark any other
Python 2 packages as deprecated as well.
== Owner ==
* Miro Hrončok (Churchyard)
* Petr Viktorin (Pviktori)
* Charalampos Stratakis (Cstratak)
* Zbigniew Jędrzejewski-Szmek (Zbyszek)
* Igor Gantenko (Ignatenkobrain)
* Neal Gompa (Ngompa)
== Detailed Description ==
Python 2 reaches End of Life on 2020-01-01. Its current maintainers
would like to orphan it before that date, and so far no one else has
stepped up to maintain Python 2 (with the full ecosystem) past 2020.
Since thousands of packages still depend on python2, we need a more
careful approach than normal orphaning.
Some of those packages are abandoned and/or the Python 2 version is
unnecessary. Others are useful and just need more time to port to
Python 3.
Hence we set up criteria for python2 packages that can remain in the
distribution and we remove everything else. This should allow us to
keep python2 for limited use, not break everything, but should also
send a strong message that it is no longer a first class citizen, and
filter packages we need to focus Python 3 porting efforts on.
(Sub-)packages that only provide a python2 importable module, and are
not required for other packages, will be removed.
Examples of situations where a (sub-)package does not provide only the
importable module:
* A package also provides an application, mostly likely in /usr/bin,
/usr/libexec...
** (Note that according to current guidelines, if the Python 3 version
of a package provides the same functionality, the Python 3 package
should provide the application.)
** (In certain situations the provided application is only useful to
boostrap or manage projects using the module. Such applications don't
count.)
* A package provides a plugin for another application, most likely
trough a setuptools entrypoint interface or some custom location on
disk.
Our process will be:
* File bugs for all packages that look like they only provide a Python
2 importable module. (This includes those needed by other packages we
plan to remove.)
* Leave at least a week for packagers to respond to the bugs.
* If the packager approves or there is no response for a week, we will
remove the package.
There are currently ~3000 source packages that generate python2 dependent
RPMs.
Automation is being created to be able to provide us a rough list of
packages to remove.
=== Packages to remove ===
The list is at https://fedoraproject.org/wiki/Changes/Mass_Python_2_
Package_Removal/List
not to disturb the reading experience of this page.
Packagers can block some packages from removal by responding on
Bugzilla and providing reasons. If no general consensus is reached,
FESCo will decide (as Python SIG does not have a formal process for
such decisions).
We'll also actively try to remove unused or optional dependencies to
reduce the number of packages that need to be kept because other
packages depend on them.
As dependencies evolve over time, we will regularly repeat this proces
on rawhide from now on even on future releases.
Packagers are strongly encouraged to help port their applications to
Python 3. Removing old Python 2 only applications from Fedora is also
encouraged, especially if the upstream is dead. Python SIG is
available to help with porting.
Packaging guidelines will be updated to reflect that packaging for
Python 3 only is the default. Instructions for Python 2 and
dual-support will be moved to the Appendix.
Python 2 will be marked deprecated (see
https://fedoraproject.org/wiki/Packaging:Deprecating_Packages).
Any package that only provides a Python 2 importable module may be
marked as deprecated as well if the maintainer(s) want to.
== Benefit to Fedora ==
A giant pile of unneeded software running on a legacy interperter will
be removed from Fedora before the interpreter stops being supported
upstream.
While we would very much like to remove Python 2 entirely, this way we
are not breaking Fedora, and we can accomodate some exceptions.
== Scope ==
* Proposal owners:
** Provide a list of packages to remove
** Collect feedback from affected packagers
** Retire the python2 only packages
** Remove python2 subpackages from dual-python packages
* Other developers:
** Ask for packages to be removed from the list if needed (with reasons).
** Remove python2 subpackages from dual-python packages (not needed,
but helpful)
** Optionaly mark remainign python2 packages deprecated
* Policies and guidelines:
** Python packaging guidelines will be changed to only describe Python 3
** Python 2 and dual-support packaging will be moved to the appendix
** Only packages providing additional features different than
importable modules will be allowed to be added in Fedora (with
exception for dependenecies of other packages)
** Python 2 will be marked deprecated
https://fedoraproject.org/wiki/Packaging:Deprecating_Packages
* Trademark approval: not needed for this Change
== Upgrade/compatibility impact ==
Packages removed form Fedora repos will remain on the installed OS
until explicitly removed by the user or obsoleted by the packagers.
We'll only obsolete packages that would break upgrade (from
fedora-obsolete-packages).
== How To Test ==
Any program written in Python and any program that has Python plugins
could potentially be influenced by this change. Testing is different
for software that is still using Python 2 and for software that has
been switched to Python 3.
For any ''python2'' programs, make sure those programs are still
functional after the package removal. Since various packages that are
no longer built will not be removed from an existing system, just
upgrading and checking packages is not enough. Either a new
installation should be used, or after a branching point, any packages
which haven't been rebuilt for F30, so any packages with .fc29 or
lower release suffix should be removed from the system before testing.
For the ''python3'' programs, install all upgrades and check if the
software works.
Upgrade failures because of missing dependencies should be treated as
bugs. Any removed python2 subpackages which break upgrades need to be
added to Obsoletes in fedora-obsolete-packages.
== User Experience ==
There will be less Python 2 RPMs in Fedora repos. Users are encouraged
to switch to Python 3 and/or use Python 2 virtual environments and pip
for development.
== Dependencies ==
Ideally, all programs that use python2 would be switched to use
python3. Although we don't expect everything to be switched over, as
much as possible should be, so that the set of remaining python2
subpackages is as small as possible.
== Contingency Plan ==
* Contingency mechanism:
** If for some reason not everything is removed, nothing happens, it
can be removed later. If for some reason something breaks, some
packages can be unretired or restored.
** If someone steps up to maintain Python 2 (including the full
ecosystem of packages now in Fedora), they can decide to discontinue
removing packages, revert the Change, or come up with another plan.
(Note that in this case, current maintainers will most likely orphan
many fundamental python2 packages.)
== Documentation ==
This page is the main documentation.
Also see: https://pythonclock.org/
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.
fedoraproject.org/message/QRXWWZ2QG7JSZMAXNI6M6QMWV2XPCINM/
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://red.ht/sig>
<https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...>
5 years, 8 months
Question about ovirt python sdk4，gluster volume rebalance return none
by chokko＠126.com
when in version 3 sdk ,i use:
req_cluster = api.clusters.get(id=c_uid)
req_volume = req_cluster.glustervolumes.get(id=volume_id)
req_volume.rebalance().job.get_id()
to get a rebalance job id to trace if the job is finished.
Now I use ovirt python sdk4 for develop:
cluster = clustersservice.cluster_service(clusterid)
volume = cluster.gluster_volumes_service().volume_service(volumeid)
response=volume.rebalance()
the response is none,how can i get the rebalance job id?
5 years, 8 months
