Re: [ovirt-users] oVirt Node ng upgrade failed

@Yuval Turgeman ### [17:27:10][root@vm5:~]$semanage permissive -a setfiles_t SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). OSError: No such file or directory ### @Ryan Barry Manual yum upgrade finished without any error but imgbased.log still shows me the following: ### 2017-11-28 17:25:28,372 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:28,434 [DEBUG] (MainThread) Creating /home as {'attach': True, 'size': '1G'} 2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling binary: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>} 2017-11-28 17:25:28,434 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '@imgbased:volume', '-o', 'lv_full_name'],) {'close_fds': True, 'stderr': <open file '/dev/null', mode 'w' at 0x7fa2d1ad8ed0>} 2017-11-28 17:25:28,533 [DEBUG] (MainThread) Returned: onn/home onn/tmp onn/var_log onn/var_log_audit 2017-11-28 17:25:28,533 [DEBUG] (MainThread) Calling binary: (['umount', '-l', '/etc'],) {} 2017-11-28 17:25:28,534 [DEBUG] (MainThread) Calling: (['umount', '-l', '/etc'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:28,539 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {} 2017-11-28 17:25:28,540 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:28,635 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.tuHU8'],) {} 2017-11-28 17:25:28,635 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.tuHU8'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:28,640 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:28,641 [ERROR] (MainThread) Failed to migrate etc Traceback (most recent call last): File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/ imgbased/plugins/osupdater.py", line 109, in on_new_layer check_nist_layout(imgbase, new_lv) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/ imgbased/plugins/osupdater.py", line 179, in check_nist_layout v.create(t, paths[t]["size"], paths[t]["attach"]) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/volume.py", line 48, in create "Path is already a volume: %s" % where AssertionError: Path is already a volume: /home 2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {} 2017-11-28 17:25:28,642 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:29,061 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.bEW2k'],) {} 2017-11-28 17:25:29,061 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.bEW2k'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:29,067 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling binary: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {} 2017-11-28 17:25:29,067 [DEBUG] (MainThread) Calling: (['umount', '-l', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:29,625 [DEBUG] (MainThread) Returned: 2017-11-28 17:25:29,625 [DEBUG] (MainThread) Calling binary: (['rmdir', u'/tmp/mnt.UB5Yg'],) {} 2017-11-28 17:25:29,626 [DEBUG] (MainThread) Calling: (['rmdir', u'/tmp/mnt.UB5Yg'],) {'close_fds': True, 'stderr': -2} 2017-11-28 17:25:29,631 [DEBUG] (MainThread) Returned: Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module> CliApplication() File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication app.hooks.emit("post-arg-parse", args) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 56, in post_argparse base_lv, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 118, in extract "%s" % size, nvr) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 99, in add_base_with_tree new_layer_lv = self.imgbase.add_layer(new_base) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 191, in add_layer self.hooks.emit("new-layer-added", prev_lv, new_lv) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.ipxGZrbQEi/usr/lib/python2.7/site-packages/ imgbased/plugins/osupdater.py", line 123, in on_new_layer raise ConfigMigrationError() imgbased.plugins.osupdater.ConfigMigrationError ### ------------------------------ *Von:* Yuval Turgeman <yuvalt(a)redhat.com&gt; *Gesendet:* Sonntag, 26. November 2017 17:23:55 *An:* Kilian Ries *Cc:* Ryan Barry; Lev Veyde; users *Betreff:* Re: [ovirt-users] oVirt Node ng upgrade failed Hi, Can you try to run on your 4.1.1 `semanage permissive -a setfiles_t` and share your output ? Thanks, Yuval On Fri, Nov 24, 2017 at 11:01 AM, Kilian Ries <mail(a)kilian-ries.de&gt; wrote: > This is the imgbased.log: > > > https://www.dropbox.com/s/v9dmgz14cpzfcsn/imgbased.log.tar.gz?dl=0 > > Ok, i'll try your steps and come back later ... > > > ------------------------------ > *Von:* Ryan Barry <rbarry(a)redhat.com&gt; > *Gesendet:* Donnerstag, 23. November 2017 23:33:34 > *An:* Kilian Ries; Lev Veyde; users > *Betreff:* Re: [ovirt-users] oVirt Node ng upgrade failed > > Can you grab imgbased.log? > > To retry, "rpm -e ovirt-node-ng-image-update" and remove the new LVs. > "yum install ovirt-node-ng-image-update" from the CLI instead of engine so > we can get full logs would be useful > > On Thu, Nov 23, 2017 at 16:01 Lev Veyde <lveyde(a)redhat.com&gt; wrote: > >> >> ---------- Forwarded message ---------- >> From: Kilian Ries <mail(a)kilian-ries.de&gt; >> Date: Thu, Nov 23, 2017 at 5:16 PM >> Subject: [ovirt-users] oVirt Node ng upgrade failed >> To: &quot;Users(a)ovirt.org&quot; <Users(a)ovirt.org&gt; >> >> >> Hi, >> >> >> just tried to upgrade from >> >> >> ovirt-node-ng-4.1.1.1-0.20170504.0+1 >> >> >> to >> >> >> ovirt-node-ng-4.1.7-0.20171108.0+1 >> >> >> but it failed: >> >> >> ### >> >> >> 2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager >> yumpackager.info:80 Yum Verify: 1/4: ovirt-node-ng-image-update.noarch >> 0:4.1.7-1.el7.centos - u >> >> 2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager >> yumpackager.info:80 Yum Verify: 2/4: ovirt-node-ng-image-update-placeholder.noarch >> 0:4.1.1.1-1.el7.centos - od >> >> 2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager >> yumpackager.info:80 Yum Verify: 3/4: ovirt-node-ng-image.noarch >> 0:4.1.1.1-1.el7.centos - od >> >> 2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager >> yumpackager.info:80 Yum Verify: 4/4: ovirt-node-ng-image-update.noarch >> 0:4.1.1.1-1.el7.centos - ud >> >> 2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Transaction processed >> >> 2017-11-23 10:19:21 DEBUG otopi.context context._executeMethod:142 >> method exception >> >> Traceback (most recent call last): >> >> File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/context.py", line 132, in >> _executeMethod >> >> method['method']() >> >> File "/tmp/ovirt-3JI9q14aGS/otopi-plugins/otopi/packagers/yumpackager.py", >> line 261, in _packages >> >> self._miniyum.processTransaction() >> >> File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 1049, >> in processTransaction >> >> _('One or more elements within Yum transaction failed') >> >> RuntimeError: One or more elements within Yum transaction failed >> >> 2017-11-23 10:19:21 ERROR otopi.context context._executeMethod:151 >> Failed to execute stage 'Package installation': One or more elements within >> Yum transaction failed >> >> 2017-11-23 10:19:21 DEBUG otopi.transaction transaction.abort:119 >> aborting 'Yum Transaction' >> >> 2017-11-23 10:19:21 INFO otopi.plugins.otopi.packagers.yumpackager >> yumpackager.info:80 Yum Performing yum transaction rollback >> >> 2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:21 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/filelists_db >> 374 k(100%) >> >> 2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db >> (0%) >> >> 2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: centos-opstools-release/7/x86_64/other_db >> 53 k(100%) >> >> 2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db (0%) >> >> 2017-11-23 10:19:22 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 55 k(4%) >> >> 2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 201 k(17%) >> >> 2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 648 k(56%) >> >> 2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(99%) >> >> 2017-11-23 10:19:23 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/filelists_db 1.1 M(100%) >> >> 2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db (0%) >> >> 2017-11-23 10:19:25 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 45 k(14%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 207 k(66%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1/7/other_db 311 k(100%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/filelists_db >> 18 k(100%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db >> (0%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-centos-gluster38/x86_64/other_db >> 7.6 k(100%) >> >> 2017-11-23 10:19:26 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db >> 7.5 M(76%) >> >> 2017-11-23 10:19:27 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/filelists_db >> 9.9 M(100%) >> >> 2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db (0%) >> >> 2017-11-23 10:19:29 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-epel/x86_64/other_db 2.9 >> M(100%) >> >> 2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:30 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/filelists_db >> 6.5 k(100%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db >> (0%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-4.1-patternfly1-noarch-epel/x86_64/other_db >> 851 (100%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/filelists_db >> 312 k(100%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db >> (0%) >> >> 2017-11-23 10:19:31 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: ovirt-centos-ovirt41/7/x86_64/other_db >> 84 k(100%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db >> (0%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/filelists_db >> 4.5 k(100%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db >> (0%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: rnachimu-gdeploy/x86_64/other_db >> 1.4 k(100%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db (0%) >> >> 2017-11-23 10:19:32 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: virtio-win-stable/filelists_db 3.9 >> k(100%) >> >> 2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db (0%) >> >> 2017-11-23 10:19:33 DEBUG otopi.plugins.otopi.packagers.yumpackager >> yumpackager.verbose:76 Yum Downloading: virtio-win-stable/other_db 4.3 >> k(100%) >> >> 2017-11-23 10:19:33 ERROR otopi.plugins.otopi.packagers.yumpackager >> yumpackager.error:85 Yum Transaction close failed: Traceback (most recent >> call last): >> >> File "/tmp/ovirt-3JI9q14aGS/pythonlib/otopi/miniyum.py", line 761, in >> endTransaction >> >> if self._yb.history_undo(transactionCurrent): >> >> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 6086, >> in history_undo >> >> if self.install(pkgtup=pkg.pkgtup): >> >> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4910, >> in install >> >> raise Errors.InstallError, _('No package(s) available to install') >> >> InstallError: Kein(e) Paket(e) zum Installieren verfügbar. >> >> >> ### >> >> >> >> Some more information on my system: >> >> >> ### >> >> >> $ mount >> >> ... >> >> /dev/mapper/onn-ovirt--node--ng--4.1.1.1--0.20170504.0+1 on / type ext4 >> (rw,relatime,discard,stripe=128,data=ordered) >> >> >> >> $ imgbase layout >> >> ovirt-node-ng-4.1.1.1-0.20170406.0 >> >> ovirt-node-ng-4.1.1.1-0.20170504.0 >> >> +- ovirt-node-ng-4.1.1.1-0.20170504.0+1 >> >> ovirt-node-ng-4.1.7-0.20171108.0 >> >> +- ovirt-node-ng-4.1.7-0.20171108.0+1 >> >> >> >> >> >> $ rpm -q ovirt-node-ng-image >> >> Das Paket ovirt-node-ng-image ist nicht installiert >> >> >> >> $ nodectl check >> >> Status: OK >> >> Bootloader ... OK >> >> Layer boot entries ... OK >> >> Valid boot entries ... OK >> >> Mount points ... OK >> >> Separate /var ... OK >> >> Discard is used ... OK >> >> Basic storage ... OK >> >> Initialized VG ... OK >> >> Initialized Thin Pool ... OK >> >> Initialized LVs ... OK >> >> Thin storage ... OK >> >> Checking available space in thinpool ... OK >> >> Checking thinpool auto-extend ... OK >> >> vdsmd ... OK >> >> >> ### >> >> >> I can restart my Node and VMs are running, but oVirt Engine tells me no >> update is available. It seems 4.1.7 is installed, but Node still boots the >> old 4.1.1 image. >> >> >> Can i force run the upgrade again or is there another way to fix this? >> >> >> Thanks >> >> Greets >> >> Kilian >> >> >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> >> -- >> >> Lev Veyde >> >> Software Engineer, RHCE | RHCVA | MCITP >> >> Red Hat Israel >> >> <https://www.redhat.com> >> >> lev(a)redhat.com | lveyde(a)redhat.com >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> > -- > > RYAN BARRY > > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHEV HYPERVISOR > > Red Hat NA <https://www.redhat.com/> > > rbarry(a)redhat.com M: +1-651-815-9306 IM: rbarry > <https://red.ht/sig> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users