Hi Didi, thanks for the attention! I created a clean slate test using four identical (Atom J5005 32GB RAM, 1TB SSD) boxes and virgin SSDs. Three-node hosted-engine using oVirt node 4.3.5 (August 5th) image. Tested adding an n+1 compute node with the same current oVirt image, which worked fine (after a long serious of trouble with existing CentOS machines). Since EL7 seems to have equal standing as oVirt nodes, I couldn't quite believe that it fails always. I tried to isolate if there was something specific in my way of setting up the CentOS nodes that caused the failure. So I started with a clean sheet CentOS 7 an another clean SSD. CentOS 1810 ISOs, developer workstation install variant, storage layout identical to the oVirt recommended, updated to the latest patches. Added oVirt repo, cockpit oVirt dashboard, ssl key etc. with all the proper reboots Test 1: Add 4th node as host to three-node cluster. Works like a charm, migrate a running VM back and forth, activate maintenance and remove 4th host Experiment 1: Add epel "yum install epe-release", yum install yumex, pick Cinnamon in Yumex (or basically yum groupinstall cinnamon) Test 2: Try to add 4th node again: Immediate failure with that message in the management engine's deploy log. Experiment 2: yum erase group cinnamon; yum autoremove; reboot Test 3: Add 4th node again: Success! (Adding Cinnamon afterwards, no problem, re-install seems likely to fail) Yes, I saw that bug report. Noticed it was REHL8 and... wasn't sure it would be related. And, like you, I can't really fathom why this would happen: I compared packet versions on both sides, oVirt nodes and CentOS for everyone involved and they seemed identical for everything. I had installed the yum priorities plugin and made sure that the "full" epel repository had prio 99 while the oVirt repos had 1 to ensure that oVirt would win out in case of any potential version conflict... As I understand it, this log on the engine lists actions performed via an ssh connection on the to-be host, and that OTOPI might actually be gathering packages through some time of y miniyum to then deploy on the to-be host in the next step. So I don't know if the Python context of this is the normal Python 2 context of the to-be host, or a temporary one that was actually created on-the-fly etc. Somehow I never found the 200 page oVirt insider bible .-) If you can help me with some context, perhaps I can dig deeper and help filing a better bug report. At the moment it seems so odd and vage, I didn't feel comfortable. Can't see any attach buttons, so I guess I may have to fiddle with my browser's privacy settings..

Hi,

On Thu, Aug 22, 2019 at 3:29 PM <thomas(a)hoberg.net> wrote:

I assume this was successful. Did you check what packages were actually installed? Especially which were updated?

I went for minimal user actions, so things would be easy to repeat. But while "yum groupinstall cinnamon" is only a single command, but it pulls (and removes) the entire suite of Cinnamon desktop apps in one go, and with quite a few dependencies all over the place. I couldn't quite compare everything, but I checked all the obvious oVirt packages, so everything with "*ovirt*, vdsm, otopi, cockpit, gluster etc.: Those had the very same numbers.

Before doing that, did you try disabling/removing full epel repo (only leaving enabled the parts enabled by ovirt-release* package)?

Yes, just disabling the epel-repo won't do the job...

After installing Cinnamon?

...once Cinnamon is installed, installation as a host fails, because Python can't find "rpmUtils". If I remove Cinnamon (yum delete cinnamon; yum autoremove), it works again.

This helps if there is a *conflict*, not sure it does much if epel has a newer version.

Didn't understand "through some time of y miniyum". ovirt-host-deploy,

I wouldn't either, I guess my fingers went into some kind of twist there ;-) it should have ready "through some type of miniyum" From what I understood reading the Python code there, the host deploy package is importing an rpmUtils Python-package (aka miniyum) to ensure that certain rpm-packages are either installed or pulled for the host. And from what I also remember going through the Github sources, this dependency on rpmUtils has been removed at some point

which what is ran on the host at that point, is based on otopi, and otopi has a yum plugin, and a miniyum module that it uses, and these indeed try to install/update packages. This is optional - if you want to prevent that, check "OFFLINE" section in: Yes, and my question was mostly if this is running perhaps in some type of Python chroot()/environment that's distinct from the 'normal' one on the target.

https://github.com/oVirt/ovirt-host-deploy/blob/master/README

I'll have a look at that: Always better to satisfy dependencies early.

On the book:

Such a thing does not exist, and if it did, it will quickly become out-of-date, and quickly get worse over time. If you search around, you actually can find parts of it scattered around, as blog posts, 'deep dive' videos, conference presentations, etc. Part of these is indeed out-of-date :-(, but at least you can rather easily see when they were posted an which version was documented. And of course, you have the source! :-) Yep, source is there, but without some backgrounder, it's a rocky journey... and I have watched quite a few videos already and some blogs. Just don't know how far back I should go, it's ten years, I believe...

I believe the problem here ocurrs in the context of Otopi, and all I have been able to find on Otopi is that while the "human" mode is slightly better than the "machine" mode for interactive use, it's not really meant to be an end-user tool... A concept guide was nowhere to be found.

I'd start with:

1. Check host-deploy logs. You can find them on the engine machine (copied from the host) in /var/log/ovirt-engine/host-deploy. Compare failed and successful ones, especially around 'yum' - it should log the list of packages it's going to update etc.

Yup, looked at that, except that I couldn't quite find that list of packages: It fails trying to satisfy the pre-conditions (missing rpmUtils), before trying to check/install what it needs on the target.

2. Compare 'rpm -qa' between a failed and a working setup. Also 'yum list all'.

Did that to exhaustion but not exhaustively... Honestly, with the work-around (temporarily removing Cinnamon), it's not quite on the critical path any longer... I sure want to solve that puzzle, but I am not sure just when I'll be able to have another go at it.

You mean attach to your email to the mailing list? Not sure you should see, but it's anyway considered better these days to upload somewhere (dropbox, google drive, some pastebin if it's just log snippets) and share a link. This applies to mails to the list. If you open a bug in bugzilla, please do attach everything directly.

I am using the Web-GUI not an e-mail client. I have been looking for some type of widget which allows me to add an attachment there, but only gut a "send" or "cancel" button. I operate my Firefox in paranoid mode, no tracking/blocking ads-fingerprinting etc. so some critical Javascript could fail.

Thanks and best regards, -- Didi

Thank you!!

On Mon, Aug 26, 2019 at 2:13 PM <thomas(a)hoberg.net> wrote:

rpmUtils is part of yum itself, in the yum libraries. miniyum is part of otopi.

It was:

https://bugzilla.redhat.com/show_bug.cgi?id=1724056

But only for 4.4 (the next version).

Otopi seems to generate a tarball on the host that runs the wizard from, which is then forwarded to the hosted-engine VM for execution. In that context a version comparison function (something like "CompareEVR") fails to execute, because that requires rpmUtils to be loaded. To my shame I am not a Python programmer, but it seems to be late-binding, which may be why it's not universally failing at the import of the rpmUtils, but only when Compare..EVR is called. Tracing what's going on in the dynamic context of a Pyhon program executing from an tarball, that's just been sent over an SSH wire to a VM that lies in agony... is a bit steep, especially because it takes an hour to recreate the circumstances. Perhaps I'd volunteer to try 4.4 early instead ;-)

If you want a minimal installation, that already includes exactly what's needed, you can also use ovirt-node :-)

The oVirtNodes work just fine and that's what I fell back on early, when I wasn't sure about the cause. But one usage scenario involves big machine learning compute hosts running Nvidia workloads on V100 GPUs in Docker containers (or basically bare metal), while some support workloads would be managed in an oVirt corner on those same hosts. It basically anticipates where oVirt wants to go, mixing scale-in and scale-out workloads under one management framework. It's also necessary, as many current ML capable GPUs aren't actually supported inside virtual machines for market segmentation reasons.

If you still want to, you can open a bug, and attach all relevant logs, including yum log and something like:

lastid=$(yum history | awk '/|/ && $1 ~ /[0-9]/ {print $1; exit}') for id in $(seq $lastid); do echo === ID $id === yum history info $id done

Very well, you can still use some file upload service. I never used the web interface for the users@ mailing list, no idea how it looks like.

I believe the bug tracker has a native attachment facility: I was just slightly confused that I might see a different UI than everybody else.

Best regards,

Likewise and thanks for your encouraging help so far!

On Wed, Sep 11, 2019 at 8:34 PM <thomas@hoberg.net> wrote:

Hi Didi, I finally figured it out:

Cinnamon pulls python3 as a dependency.

The otopi script from the ovirt-host-deploy.tar that gets transferred from the management engine to the host-to-add prefers python3 using this logic in otopi-functions.py:

get_otopi_python() { for p in "python3" "python" do pyexec=$(find_otopi ${p}) if [ -n "${pyexec}" ]; then echo "${pyexec}" break fi done }

but on CentOS 7 there are no Python3 site-packages where CentOS 8 will be all dnf and python3 I guess.

That means rpmUtils cannot be imported and causes the initialization of miniyum to fail.

With Python3 becoming standard in CentOS 7.7 this could be come a more frequent issue.

That was hard to track down, but yes, in the end the source code helped and then the fact that the otopi scipt could be executed stand-alone.

I will file a bug report and perhaps the documentation could be updated to include a Python3 warning.

Thanks for your support,

Thanks for the report! I'll try to handle it soon. Probably something trivial like changing the order in that line should be enough. We already did that a few times in the past :-( I wonder how come you are the first one reporting it. I guess not many people yet install python3 on CentOS 7. Best regards,

Thomas _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2QUBDSFTWNL27N...

-- Didi

On Thu, Sep 12, 2019 at 10:42 AM <thomas@hoberg.net> wrote:

...

On Wed, Sep 11, 2019 at 8:34 PM <thomas(a)hoberg.net> wrote:

Thanks for the report!

I'll try to handle it soon. Probably something trivial like changing the order in that line should be enough. We already did that a few times in the past :-( I don't know if RHEL/CentOS 8 is completely "python2" clean. If not, you might just create the reverse problem and then there are still other distributions in transition, I guess.

Indeed.

Perhaps a helper function needs to identify the python dialect of the orchestration engine underneath...

Something like this, which we did 3 years ago and quickly reverted, at the time: https://gerrit.ovirt.org/#/c/59831/28/src/bin/otopi

The again I am not sure that Otopi generates the tar ball on the host-to-add or on the management engine: In theory (and soon in practice) these could be crossing dialect boundaries as we'll we a mix of v7/v8 hosts being orchestrated with oVirt for some years perhaps.

Right. And it's generated on the engine, and indeed it should be prepared for both.

...

I wonder how come you are the first one reporting it. I guess not many people yet install python3 on CentOS 7.

Actually I found a previous report and added it as a reference in bugzilla. At the time the issue was closed as WONTFIX, because Python3 was considered irrelevant on CentOS 7...

Indeed. And the reporter was my team mate, not a real user :-), so I could decide it's not that common. I guess I should now apologize for wasting your time instead of fixing it then. Sorry. If Cinnamon requires python3, that's still not that common (running a GUI on a hypervisor host), but at least somewhat more so. I suppose we should be prepared for more packages requiring python3 even on EL7.

https://bugzilla.redhat.com/show_bug.cgi?id=1751324

Bug 1751324

Thanks and best regards, -- Didi