Hi Didi, thanks for the attention! I created a clean slate test using four identical (Atom J5005 32GB RAM, 1TB SSD) boxes and virgin SSDs. Three-node hosted-engine using oVirt node 4.3.5 (August 5th) image. Tested adding an n+1 compute node with the same current oVirt image, which worked fine (after a long serious of trouble with existing CentOS machines). Since EL7 seems to have equal standing as oVirt nodes, I couldn't quite believe that it fails always. I tried to isolate if there was something specific in my way of setting up the CentOS nodes that caused the failure. So I started with a clean sheet CentOS 7 an another clean SSD. CentOS 1810 ISOs, developer workstation install variant, storage layout identical to the oVirt recommended, updated to the latest patches. Added oVirt repo, cockpit oVirt dashboard, ssl key etc. with all the proper reboots Test 1: Add 4th node as host to three-node cluster. Works like a charm, migrate a running VM back and forth, activate maintenance and remove 4th host Experiment 1: Add epel "yum install epe-release", yum install yumex, pick Cinnamon in Yumex (or basically yum groupinstall cinnamon) Test 2: Try to add 4th node again: Immediate failure with that message in the management engine's deploy log. Experiment 2: yum erase group cinnamon; yum autoremove; reboot Test 3: Add 4th node again: Success! (Adding Cinnamon afterwards, no problem, re-install seems likely to fail) Yes, I saw that bug report. Noticed it was REHL8 and... wasn't sure it would be related. And, like you, I can't really fathom why this would happen: I compared packet versions on both sides, oVirt nodes and CentOS for everyone involved and they seemed identical for everything. I had installed the yum priorities plugin and made sure that the "full" epel repository had prio 99 while the oVirt repos had 1 to ensure that oVirt would win out in case of any potential version conflict... As I understand it, this log on the engine lists actions performed via an ssh connection on the to-be host, and that OTOPI might actually be gathering packages through some time of y miniyum to then deploy on the to-be host in the next step. So I don't know if the Python context of this is the normal Python 2 context of the to-be host, or a temporary one that was actually created on-the-fly etc. Somehow I never found the 200 page oVirt insider bible .-) If you can help me with some context, perhaps I can dig deeper and help filing a better bug report. At the moment it seems so odd and vage, I didn't feel comfortable. Can't see any attach buttons, so I guess I may have to fiddle with my browser's privacy settings..

I went for minimal user actions, so things would be easy to repeat. But while "yum groupinstall cinnamon" is only a single command, but it pulls (and removes) the entire suite of Cinnamon desktop apps in one go, and with quite a few dependencies all over the place. I couldn't quite compare everything, but I checked all the obvious oVirt packages, so everything with "*ovirt*, vdsm, otopi, cockpit, gluster etc.: Those had the very same numbers. Yes, just disabling the epel-repo won't do the job... ...once Cinnamon is installed, installation as a host fails, because Python can't find "rpmUtils". If I remove Cinnamon (yum delete cinnamon; yum autoremove), it works again. I wouldn't either, I guess my fingers went into some kind of twist there ;-) it should have ready "through some type of miniyum" From what I understood reading the Python code there, the host deploy package is importing an rpmUtils Python-package (aka miniyum) to ensure that certain rpm-packages are either installed or pulled for the host. And from what I also remember going through the Github sources, this dependency on rpmUtils has been removed at some point Yes, and my question was mostly if this is running perhaps in some type of Python chroot()/environment that's distinct from the 'normal' one on the target. I'll have a look at that: Always better to satisfy dependencies early. On the book: Yep, source is there, but without some backgrounder, it's a rocky journey... and I have watched quite a few videos already and some blogs. Just don't know how far back I should go, it's ten years, I believe... I believe the problem here ocurrs in the context of Otopi, and all I have been able to find on Otopi is that while the "human" mode is slightly better than the "machine" mode for interactive use, it's not really meant to be an end-user tool... A concept guide was nowhere to be found. > I'd start with: > 1. Check host-deploy logs. You can find them on the engine machine > (copied from the host) in /var/log/ovirt-engine/host-deploy. Compare > failed and successful ones, especially around 'yum' - it should log > the list of packages it's going to update etc. Yup, looked at that, except that I couldn't quite find that list of packages: It fails trying to satisfy the pre-conditions (missing rpmUtils), before trying to check/install what it needs on the target. > 2. Compare 'rpm -qa' between a failed and a working setup. Also 'yum list > all'. Did that to exhaustion but not exhaustively... Honestly, with the work-around (temporarily removing Cinnamon), it's not quite on the critical path any longer... I sure want to solve that puzzle, but I am not sure just when I'll be able to have another go at it. > You mean attach to your email to the mailing list? Not sure you should > see, but it's anyway considered better these days to upload somewhere > (dropbox, google drive, some pastebin if it's just log snippets) and > share a link. This applies to mails to the list. If you open a bug in > bugzilla, please do attach everything directly. I am using the Web-GUI not an e-mail client. I have been looking for some type of widget which allows me to add an attachment there, but only gut a "send" or "cancel" button. I operate my Firefox in paranoid mode, no tracking/blocking ads-fingerprinting etc. so some critical Javascript could fail. > Thanks and best regards, > -- > Didi Thank you!!

Otopi seems to generate a tarball on the host that runs the wizard from, which is then forwarded to the hosted-engine VM for execution. In that context a version comparison function (something like "CompareEVR") fails to execute, because that requires rpmUtils to be loaded. To my shame I am not a Python programmer, but it seems to be late-binding, which may be why it's not universally failing at the import of the rpmUtils, but only when Compare..EVR is called. Tracing what's going on in the dynamic context of a Pyhon program executing from an tarball, that's just been sent over an SSH wire to a VM that lies in agony... is a bit steep, especially because it takes an hour to recreate the circumstances. Perhaps I'd volunteer to try 4.4 early instead ;-) The oVirtNodes work just fine and that's what I fell back on early, when I wasn't sure about the cause. But one usage scenario involves big machine learning compute hosts running Nvidia workloads on V100 GPUs in Docker containers (or basically bare metal), while some support workloads would be managed in an oVirt corner on those same hosts. It basically anticipates where oVirt wants to go, mixing scale-in and scale-out workloads under one management framework. It's also necessary, as many current ML capable GPUs aren't actually supported inside virtual machines for market segmentation reasons. I believe the bug tracker has a native attachment facility: I was just slightly confused that I might see a different UI than everybody else. Likewise and thanks for your encouraging help so far!

On Wed, Sep 11, 2019 at 8:34 PM <thomas(a)hoberg.net&gt; wrote: Thanks for the report! I'll try to handle it soon. Probably something trivial like changing the order in that line should be enough. We already did that a few times in the past :-( I wonder how come you are the first one reporting it. I guess not many people yet install python3 on CentOS 7. Best regards, -- Didi

On Thu, Sep 12, 2019 at 10:42 AM <thomas(a)hoberg.net&gt; wrote: Indeed. Something like this, which we did 3 years ago and quickly reverted, at the time: https://gerrit.ovirt.org/#/c/59831/28/src/bin/otopi Right. And it's generated on the engine, and indeed it should be prepared for both. Indeed. And the reporter was my team mate, not a real user :-), so I could decide it's not that common. I guess I should now apologize for wasting your time instead of fixing it then. Sorry. If Cinnamon requires python3, that's still not that common (running a GUI on a hypervisor host), but at least somewhat more so. I suppose we should be prepared for more packages requiring python3 even on EL7. Thanks and best regards, -- Didi