[ovirt-users] oVirt Node ng upgrade failed

Kilian Ries mail at kilian-ries.de
Tue Nov 28 16:52:12 UTC 2017


@Ryan Barry


ok, i think i have to remove ALL new lvs ... but for example tmp is mounted:


###


/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.5-0.20171108.0.el7.squashfs.img (deleted) on /tmp/tmp.ipxGZrbQEi type squashfs (ro,relatime)

/tmp/tmp.ipxGZrbQEi/LiveOS/rootfs.img on /tmp/tmp.ipxGZrbQEi type ext4 (ro,relatime,data=ordered)

/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.5-0.20171108.0.el7.squashfs.img (deleted) on /tmp/tmp.1KX8MH6TKC type squashfs (ro,relatime)

/tmp/tmp.1KX8MH6TKC/LiveOS/rootfs.img on /tmp/tmp.1KX8MH6TKC type ext4 (ro,relatime,data=ordered)


###


Is it safe to unmount & delete all of the new lvs?


- home

- tmp

- var_log

- var_log_audit

________________________________
Von: Kilian Ries
Gesendet: Dienstag, 28. November 2017 17:31:32
An: Yuval Turgeman
Cc: Ryan Barry; Lev Veyde; users
Betreff: AW: [ovirt-users] oVirt Node ng upgrade failed


@Yuval Turgeman


###


[17:27:10][root at vm5:~]$semanage permissive -a setfiles_t

SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.

SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory

/sbin/load_policy:  Can't load policy:  No such file or directory

libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).

SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.

SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory

/sbin/load_policy:  Can't load policy:  No such file or directory

libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).

OSError: No such file or directory


###


@Ryan Barry


Manual yum upgrade finished without any error but imgbased.log still shows me the following:


###


2017-11-28 17:25:28,372 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Creating /home as {'attach': True, 'size': '1G'}

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling binary: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>}

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'close_fds': True, 'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>}

2017-11-28 17:25:28,533 [DEBUG] (MainThread) Returned: onn/home

  onn/tmp

  onn/var_log

  onn/var_log_audit

2017-11-28 17:25:28,533 [DEBUG] (MainThread) Calling binary: (['umount', '-l', '/etc'],) {}

2017-11-28 17:25:28,534 [DEBUG] (MainThread) Calling: (['umount', '-l', '/etc'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,539 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {}

2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.tuHU8'],) {}

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,640 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:28,641 [ERROR] (MainThread) Failed to migrate etc

Traceback (most recent call last):

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 109, in on_new_layer

    check_nist_layout(imgbase, new_lv)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 179, in check_nist_layout

    v.create(t, paths[t]["size"], paths[t]["attach"])

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/volume.py", line 48, in create

    "Path is already a volume: %s" % where

AssertionError: Path is already a volume: /home

2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {}

2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.bEW2k'],) {}

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {}

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,625 [DEBUG] (MainThread) Returned:

2017-11-28 17:25:29,625 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.UB5Yg'],) {}

2017-11-28 17:25:29,626 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,631 [DEBUG] (MainThread) Returned:

Traceback (most recent call last):

  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main

    "__main__", fname, loader, pkg_name)

  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module>

    CliApplication()

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication

    app.hooks.emit("post-arg-parse", args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit

    cb(self.context, *args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 56, in post_argparse

    base_lv, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 118, in extract

    "%s" % size, nvr)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 99, in add_base_with_tree

    new_layer_lv = self.imgbase.add_layer(new_base)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 191, in add_layer

    self.hooks.emit("new-layer-added", prev_lv, new_lv)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit

    cb(self.context, *args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 123, in on_new_layer

    raise ConfigMigrationError()

imgbased.plugins.osupdater.ConfigMigrationError


###

________________________________
Von: Yuval Turgeman <yuvalt at redhat.com>
Gesendet: Sonntag, 26. November 2017 17:23:55
An: Kilian Ries
Cc: Ryan Barry; Lev Veyde; users
Betreff: Re: [ovirt-users] oVirt Node ng upgrade failed

Hi,

Can you try to run on your 4.1.1 `semanage permissive -a setfiles_t` and share your output ?

Thanks,
Yuval

On Fri, Nov 24, 2017 at 11:01 AM, Kilian Ries <mail at kilian-ries.de<mailto:mail at kilian-ries.de>> wrote:

This is the imgbased.log:


https://www.dropbox.com/s/v9dmgz14cpzfcsn/imgbased.log.tar.gz?dl=0


Ok, i'll try your steps and come back later ...


________________________________
Von: Ryan Barry <rbarry at redhat.com<mailto:rbarry at redhat.com>>
Gesendet: Donnerstag, 23. November 2017 23:33:34
An: Kilian Ries; Lev Veyde; users
Betreff: Re: [ovirt-users] oVirt Node ng upgrade failed

Can you grab imgbased.log?

To retry, "rpm -e ovirt-node-ng-image-update" and remove the new LVs. "yum install ovirt-node-ng-image-update" from the CLI instead of engine so we can get full logs would be useful

On Thu, Nov 23, 2017 at 16:01 Lev Veyde <lveyde at redhat.com<mailto:lveyde at redhat.com>> wrote:

---------- Forwarded message ----------
From: Kilian Ries <mail at kilian-ries.de<mailto:mail at kilian-ries.de>>
Date: Thu, Nov 23, 2017 at 5:16 PM
Subject: [ovirt-users] oVirt Node ng upgrade failed
To: "Users at ovirt.org<mailto:Users at ovirt.org>" <Users at ovirt.org<mailto:Users at ovirt.org>>



Hi,


just tried to upgrade from


ovirt-node-ng-4.1.1.1-0.20170504.0+1


to


ovirt-node-ng-4.1.7-0.20171108.0+1


but it failed:


###


2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80<http://yumpackager.info:80> Yum Verify: 1/4: ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos - u

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80<http://yumpackager.info:80> Yum Verify: 2/4: ovirt-node-ng-image-update-placeholder.noarch 0:4.1.1.1-1.el7.centos - od

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80<http://yumpackager.info:80> Yum Verify: 3/4: ovirt-node-ng-image.noarch 0:4.1.1.1-1.el7.centos - od

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80<http://yumpackager.info:80> Yum Verify: 4/4: ovirt-node-ng-image-update.noarch 0:4.1.1.1-1.el7.centos - ud

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Transaction processed

2017-11-23 10:19:21 DEBUG otopi.context context._executeMethod:142 method exception

Traceback (most recent call last):

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/context.py", line 132, in _executeMethod

    method['method']()

  File "/tmp/ovirt-3JI9q14aGS/otopi-plugins/otopi/packagers/yumpackager.py", line 261, in _packages

    self._miniyum.processTransaction()

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 1049, in processTransaction

    _('One or more elements within Yum transaction failed')

RuntimeError: One or more elements within Yum transaction failed

2017-11-23 10:19:21 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': One or more elements within Yum transaction failed

2017-11-23 10:19:21 DEBUG otopi.transaction transaction.abort:119 aborting 'Yum Transaction'

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80<http://yumpackager.info:80> Yum Performing yum transaction rollback

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db (0%)

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db 374 k(100%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db (0%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db 53 k(100%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db (0%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 55 k(4%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 201 k(17%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 648 k(56%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(99%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(100%)

2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db (0%)

2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 45 k(14%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 207 k(66%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 311 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db (0%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db 18 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db (0%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db 7.6 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db (0%)

2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db 7.5 M(76%)

2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db 9.9 M(100%)

2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db (0%)

2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db 2.9 M(100%)

2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db (0%)

2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db 6.5 k(100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db 851 (100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db 312 k(100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db 84 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db 4.5 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db 1.4 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db 3.9 k(100%)

2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db (0%)

2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db 4.3 k(100%)

2017-11-23 10:19:33 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum Transaction close failed: Traceback (most recent call last):

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 761, in endTransaction

    if self._yb.history_undo(transactionCurrent):

  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 6086, in history_undo

    if self.install(pkgtup=pkg.pkgtup):

  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4910, in install

    raise Errors.InstallError, _('No package(s) available to install')

InstallError: Kein(e) Paket(e) zum Installieren verf?gbar.


###



Some more information on my system:


###


$ mount

...

/dev/mapper/onn-ovirt--node--ng--4.1.1.1--0.20170504.0+1 on / type ext4 (rw,relatime,discard,stripe=128,data=ordered)



$ imgbase layout

ovirt-node-ng-4.1.1.1-0.20170406.0

ovirt-node-ng-4.1.1.1-0.20170504.0

 +- ovirt-node-ng-4.1.1.1-0.20170504.0+1

ovirt-node-ng-4.1.7-0.20171108.0

 +- ovirt-node-ng-4.1.7-0.20171108.0+1





$ rpm -q ovirt-node-ng-image

Das Paket ovirt-node-ng-image ist nicht installiert



$ nodectl check

Status: OK

Bootloader ... OK

  Layer boot entries ... OK

  Valid boot entries ... OK

Mount points ... OK

  Separate /var ... OK

  Discard is used ... OK

Basic storage ... OK

  Initialized VG ... OK

  Initialized Thin Pool ... OK

  Initialized LVs ... OK

Thin storage ... OK

  Checking available space in thinpool ... OK

  Checking thinpool auto-extend ... OK

vdsmd ... OK


###


I can restart my Node and VMs are running, but oVirt Engine tells me no update is available. It seems 4.1.7 is installed, but Node still boots the old 4.1.1 image.


Can i force run the upgrade again or is there another way to fix this?


Thanks

Greets

Kilian





_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users




--


Lev Veyde

Software Engineer, RHCE | RHCVA | MCITP

Red Hat Israel

<https://www.redhat.com>

lev at redhat.com<mailto:lev at redhat.com> | lveyde at redhat.com<mailto:lveyde at redhat.com>

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig>
TRIED. TESTED. TRUSTED.<https://redhat.com/trusted>
--

RYAN BARRY

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHEV HYPERVISOR

Red Hat NA<https://www.redhat.com/>

rbarry at redhat.com<mailto:rbarry at redhat.com>    M: +1-651-815-9306     IM: rbarry

[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.ht/sig>

_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171128/5f5d883c/attachment.html>


More information about the Users mailing list