Kilian, did you disable selinux by any chance ? (selinux=0 on boot) ?

On Thu, Nov 30, 2017 at 9:57 AM, Yuval Turgeman <yuvalt@redhat.com> wrote:
Looks like selinux is broken on your machine for some reason, can you share /etc/selinux ?

Thanks,
Yuval.

On Tue, Nov 28, 2017 at 6:31 PM, Kilian Ries <mail@kilian-ries.de> wrote:

@Yuval Turgeman


###


[17:27:10][root@vm5:~]$semanage permissive -a setfiles_t

SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.

SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory

/sbin/load_policy:  Can't load policy:  No such file or directory

libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).

SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.

SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory

/sbin/load_policy:  Can't load policy:  No such file or directory

libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).

OSError: No such file or directory


###


@Ryan Barry


Manual yum upgrade finished without any error but imgbased.log still shows me the following:


###


2017-11-28 17:25:28,372 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Creating /home as {'attach': True, 'size': '1G'}

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling binary: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>}

2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'close_fds': True, 'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>}

2017-11-28 17:25:28,533 [DEBUG] (MainThread) Returned: onn/home         

  onn/tmp          

  onn/var_log      

  onn/var_log_audit

2017-11-28 17:25:28,533 [DEBUG] (MainThread) Calling binary: (['umount', '-l', '/etc'],) {}

2017-11-28 17:25:28,534 [DEBUG] (MainThread) Calling: (['umount', '-l', '/etc'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,539 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {}

2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.tuHU8'],) {}

2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:28,640 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:28,641 [ERROR] (MainThread) Failed to migrate etc

Traceback (most recent call last):

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 109, in on_new_layer

    check_nist_layout(imgbase, new_lv)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 179, in check_nist_layout

    v.create(t, paths[t]["size"], paths[t]["attach"])

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/volume.py", line 48, in create

    "Path is already a volume: %s" % where

AssertionError: Path is already a volume: /home

2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {}

2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.bEW2k'],) {}

2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {}

2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,625 [DEBUG] (MainThread) Returned: 

2017-11-28 17:25:29,625 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.UB5Yg'],) {}

2017-11-28 17:25:29,626 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2}

2017-11-28 17:25:29,631 [DEBUG] (MainThread) Returned: 

Traceback (most recent call last):

  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main

    "__main__", fname, loader, pkg_name)

  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module>

    CliApplication()

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication

    app.hooks.emit("post-arg-parse", args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit

    cb(self.context, *args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 56, in post_argparse

    base_lv, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 118, in extract

    "%s" % size, nvr)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 99, in add_base_with_tree

    new_layer_lv = self.imgbase.add_layer(new_base)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 191, in add_layer

    self.hooks.emit("new-layer-added", prev_lv, new_lv)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit

    cb(self.context, *args)

  File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 123, in on_new_layer

    raise ConfigMigrationError()

imgbased.plugins.osupdater.ConfigMigrationError


###

Von: Yuval Turgeman <yuvalt@redhat.com>
Gesendet: Sonntag, 26. November 2017 17:23:55
An: Kilian Ries
Cc: Ryan Barry; Lev Veyde; users

Betreff: Re: [ovirt-users] oVirt Node ng upgrade failed
 
Hi,

Can you try to run on your 4.1.1 `semanage permissive -a setfiles_t` and share your output ?

Thanks,
Yuval

On Fri, Nov 24, 2017 at 11:01 AM, Kilian Ries <mail@kilian-ries.de> wrote:

This is the imgbased.log:


https://www.dropbox.com/s/v9dmgz14cpzfcsn/imgbased.log.tar.gz?dl=0

Ok, i'll try your steps and come back later ...


Von: Ryan Barry <rbarry@redhat.com>
Gesendet: Donnerstag, 23. November 2017 23:33:34
An: Kilian Ries; Lev Veyde; users
Betreff: Re: [ovirt-users] oVirt Node ng upgrade failed
 
Can you grab imgbased.log?

To retry, "rpm -e ovirt-node-ng-image-update" and remove the new LVs. "yum install ovirt-node-ng-image-update" from the CLI instead of engine so we can get full logs would be useful

On Thu, Nov 23, 2017 at 16:01 Lev Veyde <lveyde@redhat.com> wrote:

---------- Forwarded message ----------
From: Kilian Ries <mail@kilian-ries.de>
Date: Thu, Nov 23, 2017 at 5:16 PM
Subject: [ovirt-users] oVirt Node ng upgrade failed
To: "Users@ovirt.org" <Users@ovirt.org>


Hi,


just tried to upgrade from 


ovirt-node-ng-4.1.1.1-0.20170504.0+1


to


ovirt-node-ng-4.1.7-0.20171108.0+1


but it failed:


###


2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Verify: 1/4: ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos - u

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Verify: 2/4: ovirt-node-ng-image-update-placeholder.noarch 0:4.1.1.1-1.el7.centos - od

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Verify: 3/4: ovirt-node-ng-image.noarch 0:4.1.1.1-1.el7.centos - od

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Verify: 4/4: ovirt-node-ng-image-update.noarch 0:4.1.1.1-1.el7.centos - ud

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Transaction processed

2017-11-23 10:19:21 DEBUG otopi.context context._executeMethod:142 method exception

Traceback (most recent call last):

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/context.py", line 132, in _executeMethod

    method['method']()

  File "/tmp/ovirt-3JI9q14aGS/otopi-plugins/otopi/packagers/yumpackager.py", line 261, in _packages

    self._miniyum.processTransaction()

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 1049, in processTransaction

    _('One or more elements within Yum transaction failed')

RuntimeError: One or more elements within Yum transaction failed

2017-11-23 10:19:21 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': One or more elements within Yum transaction failed

2017-11-23 10:19:21 DEBUG otopi.transaction transaction.abort:119 aborting 'Yum Transaction'

2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Performing yum transaction rollback

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db (0%)

2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db 374 k(100%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db (0%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db 53 k(100%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db (0%)

2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 55 k(4%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 201 k(17%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 648 k(56%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(99%)

2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(100%)

2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db (0%)

2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 45 k(14%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 207 k(66%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 311 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db (0%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db 18 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db (0%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db 7.6 k(100%)

2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db (0%)

2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db 7.5 M(76%)

2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db 9.9 M(100%)

2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db (0%)

2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db 2.9 M(100%)

2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db (0%)

2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db 6.5 k(100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db 851 (100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db 312 k(100%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db (0%)

2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db 84 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db 4.5 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db 1.4 k(100%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db (0%)

2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db 3.9 k(100%)

2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db (0%)

2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db 4.3 k(100%)

2017-11-23 10:19:33 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum Transaction close failed: Traceback (most recent call last):

  File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 761, in endTransaction

    if self._yb.history_undo(transactionCurrent):

  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 6086, in history_undo

    if self.install(pkgtup=pkg.pkgtup):

  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4910, in install

    raise Errors.InstallError, _('No package(s) available to install')

InstallError: Kein(e) Paket(e) zum Installieren verfügbar.


###



Some more information on my system:


###


$ mount

...

/dev/mapper/onn-ovirt--node--ng--4.1.1.1--0.20170504.0+1 on / type ext4 (rw,relatime,discard,stripe=128,data=ordered)



$ imgbase layout

ovirt-node-ng-4.1.1.1-0.20170406.0

ovirt-node-ng-4.1.1.1-0.20170504.0

 +- ovirt-node-ng-4.1.1.1-0.20170504.0+1

ovirt-node-ng-4.1.7-0.20171108.0

 +- ovirt-node-ng-4.1.7-0.20171108.0+1

 

  

$ rpm -q ovirt-node-ng-image

Das Paket ovirt-node-ng-image ist nicht installiert



$ nodectl check

Status: OK

Bootloader ... OK

  Layer boot entries ... OK

  Valid boot entries ... OK

Mount points ... OK

  Separate /var ... OK

  Discard is used ... OK

Basic storage ... OK

  Initialized VG ... OK

  Initialized Thin Pool ... OK

  Initialized LVs ... OK

Thin storage ... OK

  Checking available space in thinpool ... OK

  Checking thinpool auto-extend ... OK

vdsmd ... OK


###


I can restart my Node and VMs are running, but oVirt Engine tells me no update is available. It seems 4.1.7 is installed, but Node still boots the old 4.1.1 image. 


Can i force run the upgrade again or is there another way to fix this?


Thanks

Greets

Kilian






_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users




--

Lev Veyde

Software Engineer, RHCE | RHCVA | MCITP

Red Hat Israel

lev@redhat.com | lveyde@redhat.com

--

RYAN BARRY

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHEV HYPERVISOR

Red Hat NA

rbarry@redhat.com    M: +1-651-815-9306     IM: rbarry

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users