August 2018 - Devel - oVirt List Archives

Re: Image Transfer mechanism queries/API support
by Ranjit DSouza 28 Aug '18

28 Aug '18

Hello oVirt Development team Not sure you got a chance to see this email, resending it. Thanks Ranjit From: Ranjit DSouza Sent: Monday, July 9, 2018 1:16 PM To: 'Nir Soffer' <nsoffer(a)redhat.com> Cc: devel <devel(a)ovirt.org>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com>; Navin Tah <Navin.Tah(a)veritas.com>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com>; Nisan, Tal <tnisan(a)redhat.com>; Daniel Erez <derez(a)redhat.com> Subject: RE: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support Hi Nir Thanks for getting back! We have few follow up questions: 1. On the new ‘Allocated extents API’: Can you share the release timeline for 4.2.z? From the link below it seems like it will be available on 7/30/2018. https://www.ovirt.org/develop/release-management/releases/4.2.z/release-man… However, we thought we would double check on this. 2. If this will be available in 4.2.z, does It mean, we can assume it will be backported to 4.3 also? 3. When we downloaded the snapshot disk using Image Transfer API, the resulting format of the disk is “raw”. However, for upload, we must upload a qcow2 disk (to enable further snapshots). It means, we need to convert it first using qemu-img convert. Or is there a way we directly ask via API for a qcow2 instead? Portion of the response of “GET /storagedomains/{storagedomain:id}/disksnapshots” "format" : "raw", "shareable" : "false", "sparse" : "true", "status" : "ok", "snapshot" : { "id" : "4756036e-92aa-4ebb-ae4b-052a30cd5109" }, "actual_size" : "1345228800", "content_type" : "data", "propagate_errors" : "false", "provisioned_size" : "21474836480", "storage_type" : "image", "total_size" : "0", "wipe_after_delete" : "false", 4. When downloading a snapshot in chunks, Is there any recommended chunk size? For our study we used 512 KB. Checked the documentation too. 5. Regarding your ask on ‘Can you file RFE for this, explaining the use case and the current performance’. Since you do not recommend direct access to NFS storage, we will consider it only if we see significant performance degradation using http download. Also, if time permits, we may check if there is a significant performance benefit using Unix socket, over http download (via Rest API). But as it stands now, getting the allocated extents support (soon), will alleviate most of our performance concerns. Thanks Ranjit From: Nir Soffer [mailto:nsoffer@redhat.com] Sent: Tuesday, July 3, 2018 4:29 PM To: Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>> Cc: devel <devel(a)ovirt.org<mailto:devel@ovirt.org>>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com<mailto:DL-VTAS-ENG-NBU-EverestFalcons@veritas.com>>; Navin Tah <Navin.Tah(a)veritas.com<mailto:Navin.Tah@veritas.com>>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com<mailto:Sudhakar.Paulzagade@veritas.com>>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com<mailto:ylavi@redhat.com>>; Nisan, Tal <tnisan(a)redhat.com<mailto:tnisan@redhat.com>>; Daniel Erez <derez(a)redhat.com<mailto:derez@redhat.com>> Subject: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support On Tue, Jul 3, 2018 at 11:32 AM Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>> wrote: ... We had a conversation with Pavan Chavva for supporting RHV. He had suggested to contact you with queries related to oVirt APIs we plan to use. We have following queries: 1. While downloading a snapshot disk, can we identify allocated extents and download only those using oVirt API? We are able to download the disk using the Image Transfer API mechanism. However, this method downloads the entire disk including the non-allocated extents, which is a performance overhead. If this functionality does not exist at this point will it be available in near future? Hi Ranjit, There is no way to do this in current 4.2, but we plan to introduce in in 4.2.z. The API will be something like: GET /images/xxx-yyy/map ... [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0}, { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536}, { "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576}, { "start": 1114112, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 1114112}, ... { "start": 5465571328, "length": 22675456, "depth": 0, "zero": false, "data": true, "offset": 5465571328}, { "start": 5488246784, "length": 954138624, "depth": 0, "zero": true, "data": false, "offset": 5488246784}, { "start": 6442385408, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 6442385408}] This is basically what you get using qemu-img map. You can test play this with this using: virt-builder Fedora-27 -o /var/tmp/fedora-27.img qemu-img map -f raw --output json /var/tmp/fedora-27.img This is the first data segment: { "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0} This is a hole between the first data segment and the second: { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536} This is the second data segment: { "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576} Based on this output, you will be able to get the allocated parts of the image using: Request: GET /image/xxx-yyy HTTP/1.1 Range: bytes=0-65535 Response: HTTP/1.1 206 Partial Content Content-Range: bytes 0-65535/6442450944 <data of first segment> Request: GET /image/xxx-yyy HTTP/1.1 Range: bytes=1048576-1114111 Response: HTTP/1.1 206 Partial Content Content-Range: bytes 1048576-1114111/6442450944 <data of second segment> And so on. If you create a sparse file on your backup media, and download and write the data segments at the correct offset, you will get the a sparse version of the image as on the server side. This will work for raw or qcow2 images on NFS >= 4.2, or for qcow2 images on block storage. For older NFS versions, or raw images on block storage, we can solve the issue by reading the entire image and detecting zeroes - which is quite expensive, so I'm not sure we will implement this, maybe it will be done later. We have experimental patch using special sparse format, that can support this use case, downloading entire image in one pass. This commit message explain the format: https://gerrit.ovirt.org/#/c/85413/12//COMMIT_MSG For more info on using random I/O APIs, see: http://ovirt.github.io/ovirt-imageio/random-io (available since 4.2.3) For example code uploading sparse images see: https://github.com/oVirt/ovirt-imageio/blob/master/examples/upload For best performance, you should run your application on a oVirt host, using unix socket to communicate with imageio. See: http://ovirt.github.io/ovirt-imageio/unix-socket (will be available in 4.2.5) All this will work only for the non-active layer in a qcow2 chain. We are working now on incremental backup which will allow the same for the active layer with a running vm. This is expected in 4.3, and we may have a tech preview at some point in 4.2.z. Incremantal backup will use the similar API, allowing detection of dirty parts of an image, so you can download only the data that was changed since the last backup. Please watch and comment on the feature page: https://ovirt.org/develop/release-management/features/storage/incremental-b… We are also considering exposing images using NBD. This will allow downloading and uploading images using qemu-img from any host. This work depends on TLS-PSK support in qemu-img and qemu-nbd. You can follow this work here: https://lists.nongnu.org/archive/html/qemu-devel/2018-06/threads.html#08491 2. Is there an alternate method to transfer a snapshot to and from RHV storage? Are there other methods such as NFS share where we can download snapshot image to and from RHV storage? We don't support direct access to storage by 3rd party. You should use imageio API. Can you file RFE for this, explaning the use case and the current performance issues you experience? Nir

4 4

vdsm has been tagged (v4.20.39)
by Milan Zamazal 28 Aug '18

28 Aug '18

Vdsm 4.20.39 for oVirt 4.2.6 rc3

1 0

new storage domain question
by Hetz Ben Hamo 27 Aug '18

27 Aug '18

Hi, I just found something very interesting. While installing latest ovirt, I've created the shared_storage on my nas1 share folder (NFS). So far, so good. However, when trying to create additional 2 new storage domains, I can only create 1 (which means the shared_storage is 1 and a new one is 2, I cannot create a 3rd storage domain, 4th etc..). When I try, it shows an error (I don't remember which error right now). Why is that? and why this strange (IMHO) shared_storage thing? in a corporate, if I want to set up a virtualized environment, the storage administrator asks me which one I want (NFS, or an iSCSI LUN), do I need one or more, and then I'm adding them as a datastore (in VMWare, for example) and on top of that I'm installing the management software (for example, vCenter). with oVirt method, it will confuse VMware admins. Thanks

2 5

ovirt-engine has been tagged (ovirt-engine-4.2.6.4)
by Tal Nisan 26 Aug '18

26 Aug '18

1 0

vdsm has been tagged (v4.20.38)
by Milan Zamazal 24 Aug '18

24 Aug '18

Vdsm 4.20.38 for oVirt 4.2.6 rc3

1 0

Fwd: [CentOS-announce] Release for CentOS Linux 7 (1804) on POWER9 (ppc64le)
by Sandro Bonazzola 22 Aug '18

22 Aug '18

FYI ---------- Forwarded message ---------- From: James O'Connor <jpoc(a)linux.vnet.ibm.com> Date: 2018-08-15 20:26 GMT+02:00 Subject: [CentOS-announce] Release for CentOS Linux 7 (1804) on POWER9 (ppc64le) To: centos-announce(a)centos.org I am pleased to announce the general availability of CentOS Linux 7 (1804) for POWER9 processors (ppc64le - powerpc 64-bit little endian). This release is derived from Red Hat Enterprise Linux 7.5 ALT. Note this release is 99% equivalent to the existing CentOS 7 Linux 7 (1804) for POWER8 processors (ppc64le - powerpc 64-bit little endian). The key difference being that kernel-4.14.0 is used for POWER9 enablement and kernel-3.10.0 is used for POWER8 enablement. You will find the CentOS 7 (1804) Power9 installation images here: http://mirror.centos.org/altarch/7.5.1804/isos/power9/ sha256sum power9: 753c5b55c9ff681c925ec7d7c5295890284f40ec981f4da35b1f36f1ecbad95a CentOS-7-Power9-Everything-1804.iso 58d9af172656db71947b5db2d0d1f577217bbc1a86eada7ca3ce7bfcf7d7d9b4 CentOS-7-Power9-NetInstall-1804.iso CentOS 7 (1804) Power9 repositories: http://mirror.centos.org/altarch/7.5.1804/os/power9/ http://mirror.centos.org/altarch/7.5.1804/updates/power9/ http://mirror.centos.org/altarch/7.5.1804/extras/power9/ A special thank you to Johnny Hughes and Pablo Greco for their assistance making this release possible. -James O’Connor _______________________________________________ CentOS-announce mailing list CentOS-announce(a)centos.org https://lists.centos.org/mailman/listinfo/centos-announce -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…>

1 0

planned Jenkins restart
by Evgheni Dereveanchin 22 Aug '18

22 Aug '18

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. Regards, Evgheni Dereveanchin

1 1

Is using CompensationContext in child commands broken?
by Andrej Krejcir 21 Aug '18

21 Aug '18

Hi, I'm changing UpdateClusterCommand to be non-transactive, and it would use CompensationContext for reverting DB operations. Reading through CommandBase code, I noticed the following flow seems to be broken: 1. A command without transaction is executed. 2. It executes a child command, with the compensation context of the current command. So that the result of the child command can be reverted if the parent fails. 3. Once the child command successfully finishes, it removes all entity snapshots from the DB (it calls CommandContext::cleanupCompensationDataAfterSuccessfulCommand), even if the snapshots were made by the parent command. See code at [1] and [2]. 4. The parent command then fails for some other reason, but its compensation context is empty, so the DB operations are not reverted correctly. Am I missing something or is this flow broken? Thanks, Andrej [1] - https://github.com/oVirt/ovirt-engine/blob/31fbb4cb48c7208e6b37c5c0e2756059… [2] - https://github.com/oVirt/ovirt-engine/blob/31fbb4cb48c7208e6b37c5c0e2756059…

3 2

Fwd: Fedora 30 System-Wide Change proposal: Mass Python 2 Package Removal
by Sandro Bonazzola 21 Aug '18

21 Aug '18

FYI ---------- Forwarded message ---------- From: Ben Cotton <bcotton(a)redhat.com> Date: 2018-08-20 22:41 GMT+02:00 Subject: Fedora 30 System-Wide Change proposal: Mass Python 2 Package Removal To: devel(a)lists.fedoraproject.org, devel-announce(a)lists.fedoraproject.org https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal == Summary == (Sub-)packages only providing python2 importable modules without additional functionality will be removed from Fedora unless some other package(s) depends on them. Python 2 will be deprecated in Fedora. Packagers can mark any other Python 2 packages as deprecated as well. == Owner == * Miro Hrončok (Churchyard) * Petr Viktorin (Pviktori) * Charalampos Stratakis (Cstratak) * Zbigniew Jędrzejewski-Szmek (Zbyszek) * Igor Gantenko (Ignatenkobrain) * Neal Gompa (Ngompa) == Detailed Description == Python 2 reaches End of Life on 2020-01-01. Its current maintainers would like to orphan it before that date, and so far no one else has stepped up to maintain Python 2 (with the full ecosystem) past 2020. Since thousands of packages still depend on python2, we need a more careful approach than normal orphaning. Some of those packages are abandoned and/or the Python 2 version is unnecessary. Others are useful and just need more time to port to Python 3. Hence we set up criteria for python2 packages that can remain in the distribution and we remove everything else. This should allow us to keep python2 for limited use, not break everything, but should also send a strong message that it is no longer a first class citizen, and filter packages we need to focus Python 3 porting efforts on. (Sub-)packages that only provide a python2 importable module, and are not required for other packages, will be removed. Examples of situations where a (sub-)package does not provide only the importable module: * A package also provides an application, mostly likely in /usr/bin, /usr/libexec... ** (Note that according to current guidelines, if the Python 3 version of a package provides the same functionality, the Python 3 package should provide the application.) ** (In certain situations the provided application is only useful to boostrap or manage projects using the module. Such applications don't count.) * A package provides a plugin for another application, most likely trough a setuptools entrypoint interface or some custom location on disk. Our process will be: * File bugs for all packages that look like they only provide a Python 2 importable module. (This includes those needed by other packages we plan to remove.) * Leave at least a week for packagers to respond to the bugs. * If the packager approves or there is no response for a week, we will remove the package. There are currently ~3000 source packages that generate python2 dependent RPMs. Automation is being created to be able to provide us a rough list of packages to remove. === Packages to remove === The list is at https://fedoraproject.org/wiki/Changes/Mass_Python_2_ Package_Removal/List not to disturb the reading experience of this page. Packagers can block some packages from removal by responding on Bugzilla and providing reasons. If no general consensus is reached, FESCo will decide (as Python SIG does not have a formal process for such decisions). We'll also actively try to remove unused or optional dependencies to reduce the number of packages that need to be kept because other packages depend on them. As dependencies evolve over time, we will regularly repeat this proces on rawhide from now on even on future releases. Packagers are strongly encouraged to help port their applications to Python 3. Removing old Python 2 only applications from Fedora is also encouraged, especially if the upstream is dead. Python SIG is available to help with porting. Packaging guidelines will be updated to reflect that packaging for Python 3 only is the default. Instructions for Python 2 and dual-support will be moved to the Appendix. Python 2 will be marked deprecated (see https://fedoraproject.org/wiki/Packaging:Deprecating_Packages) Any package that only provides a Python 2 importable module may be marked as deprecated as well if the maintainer(s) want to. == Benefit to Fedora == A giant pile of unneeded software running on a legacy interperter will be removed from Fedora before the interpreter stops being supported upstream. While we would very much like to remove Python 2 entirely, this way we are not breaking Fedora, and we can accomodate some exceptions. == Scope == * Proposal owners: ** Provide a list of packages to remove ** Collect feedback from affected packagers ** Retire the python2 only packages ** Remove python2 subpackages from dual-python packages * Other developers: ** Ask for packages to be removed from the list if needed (with reasons). ** Remove python2 subpackages from dual-python packages (not needed, but helpful) ** Optionaly mark remainign python2 packages deprecated * Policies and guidelines: ** Python packaging guidelines will be changed to only describe Python 3 ** Python 2 and dual-support packaging will be moved to the appendix ** Only packages providing additional features different than importable modules will be allowed to be added in Fedora (with exception for dependenecies of other packages) ** Python 2 will be marked deprecated https://fedoraproject.org/wiki/Packaging:Deprecating_Packages * Trademark approval: not needed for this Change == Upgrade/compatibility impact == Packages removed form Fedora repos will remain on the installed OS until explicitly removed by the user or obsoleted by the packagers. We'll only obsolete packages that would break upgrade (from fedora-obsolete-packages). == How To Test == Any program written in Python and any program that has Python plugins could potentially be influenced by this change. Testing is different for software that is still using Python 2 and for software that has been switched to Python 3. For any ''python2'' programs, make sure those programs are still functional after the package removal. Since various packages that are no longer built will not be removed from an existing system, just upgrading and checking packages is not enough. Either a new installation should be used, or after a branching point, any packages which haven't been rebuilt for F30, so any packages with .fc29 or lower release suffix should be removed from the system before testing. For the ''python3'' programs, install all upgrades and check if the software works. Upgrade failures because of missing dependencies should be treated as bugs. Any removed python2 subpackages which break upgrades need to be added to Obsoletes in fedora-obsolete-packages. == User Experience == There will be less Python 2 RPMs in Fedora repos. Users are encouraged to switch to Python 3 and/or use Python 2 virtual environments and pip for development. == Dependencies == Ideally, all programs that use python2 would be switched to use python3. Although we don't expect everything to be switched over, as much as possible should be, so that the set of remaining python2 subpackages is as small as possible. == Contingency Plan == * Contingency mechanism: ** If for some reason not everything is removed, nothing happens, it can be removed later. If for some reason something breaks, some packages can be unretired or restored. ** If someone steps up to maintain Python 2 (including the full ecosystem of packages now in Fedora), they can decide to discontinue removing packages, revert the Change, or come up with another plan. (Note that in this case, current maintainers will most likely orphan many fundamental python2 packages.) == Documentation == This page is the main documentation. Also see: https://pythonclock.org/ -- Ben Cotton Fedora Program Manager TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org/message/QRXWWZ2QG7JSZMAXNI6M6QMWV2XPCINM/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…>

1 0

Question about ovirt python sdk4，gluster volume rebalance return none
by chokko＠126.com 21 Aug '18

21 Aug '18

when in version 3 sdk ,i use: req_cluster = api.clusters.get(id=c_uid) req_volume = req_cluster.glustervolumes.get(id=volume_id) req_volume.rebalance().job.get_id() to get a rebalance job id to trace if the job is finished. Now I use ovirt python sdk4 for develop: cluster = clustersservice.cluster_service(clusterid) volume = cluster.gluster_volumes_service().volume_service(volumeid) response=volume.rebalance() the response is none,how can i get the rebalance job id?

5 8