On Wed, Aug 25, 2021 at 2:18 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: [snip] It is not clear the sense of the phrase above "check the status of this host and eventually remediate it, please continue only when the host is listed as 'up'"... Does it refer to the novirt2 host (that is the first I'm installing while nnovirt1 is still in 4.3.10 with a VM running), or novirt1? Because if I go to the engine vm under /var/log/ovirt-engine I see: [root@novmgr ovirt-engine]# cd host-deploy/ [root@novmgr host-deploy]# ll total 348 -rw-r--r--. 1 ovirt ovirt 354888 Aug 25 09:41 ovirt-host-mgmt-ansible-check-20210825094043-novirt1.localdomain.local.log [root@novmgr host-deploy]# So there is only the log file related to the deploy of novirt1 (that I see as up....) no log for novirt2. Gianluca

On Wed, Aug 25, 2021 at 4:34 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Same problem with 4.4.8 async 1. I'm deploying/restoring from novirt2 and the other host (still in 4.3.10) is novirt1. I arrive at [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check actual cluster location] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enable GlusterFS at cluster level] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set VLAN ID at datacenter level] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get active list of active firewalld zones] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Configure libvirt firewalld zone] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Include after_add_host tasks files] [ INFO ] You can now connect to https://novirt2.localdomain.local:6900/ovirt-engine/ and check the status of this host and eventually remediate it, please continue only when the host is listed as 'up' [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create temporary lock file] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Pause execution until /tmp/ansible.5f702qq5_he_setup_lock is removed, delete it once ready to proceed] But then I'm able to connect to local engine web admin UI and novirt1 results up while novirt2 not responsive. Every 3 seconds iinside engine.log I see these 3 lines 2021-08-27 11:05:54,065+02 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to novirt2.localdomain.local/172.19.0.232 2021-08-27 11:05:54,067+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2021-08-27 11:05:54,068+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = novirt2.localdomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='ca9ff6f7-5a7c-4168-9632-998c52f76cfa', vds='Host[novirt2.localdomain.local,ca9ff6f7-5a7c-4168-9632-998c52f76cfa]'})' execution failed: java.net.ConnectException: Connection refused Can anyone telling what I can check? Did you test SHE upgrade from 4.3.10 to 4.4.8 in your check flows? BTW: I'm not using DNS but entries in /etc/hosts Thanks, Gianluca

On Fri, Aug 27, 2021 at 3:25 PM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: No, I didn't do it... I would expect oVirt to manage it... Can I do it now? If I try to do I get: Error while executing action: Cannot switch Host to Maintenance mode. Host still has running VMs on it and is in Non Responsive state. because the temporary engine is on it It seems in 4.2 -> 4.3 update it was not necessary... Also, for example on RHV guides I see: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/... " When upgrading Red Hat Virtualization Manager, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure. " and https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/... " Install RHVH 4.4 or Red Hat Enterprise Linux 8.2 or later on the existing node currently running the Manager virtual machine to use it as the self-hosted engine deployment host. See Installing the Self-hosted Engine Deployment Host for more information. Note It is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure. . . . - The deployment script automatically disables global maintenance mode and calls the HA agent to start the self-hosted engine virtual machine. The upgraded host with the 4.4 self-hosted engine reports that HA mode is active, but the other hosts report that global maintenance mode is still enabled as they are still connected to the old self-hosted engine storage. - Detach the storage domain that hosts the Manager 4.3 machine. For details, see Detaching a Storage Domain from a Data Center <https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/... in the *Administration Guide*. " no mention about ssh fingerprint reissue Any hint on how to do it? Gianluca

On Fri, Aug 27, 2021 at 4:38 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: OK, so next step tried has been (thanks Sandro for the input!): . power down novirt2 where hosted-engine deploy was stuck (and related still local vm), scratching it . install the same host but with name novirt3.localdomain.local and different ip . run the hosted-engine --deploy --restore-from-file=backup.bck . now all goes ok and novirt3 has been added to engine and novirt1 results as up, while novirt2 nonresponsive (it doesn't exist any more...) all flow completes [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool 84f8abd5-31ec-4c62-8130-521bb55c41e6] [ INFO ] changed: [localhost] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20210827182404.conf' [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Hosted Engine successfully deployed [ INFO ] Other hosted-engine hosts have to be reinstalled in order to update their storage configuration. From the engine, host by host, please set maintenance mode and then click on reinstall button ensuring you choose DEPLOY in hosted engine tab. [ INFO ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients. [root@novirt3 ~]# Next step will be to try removing one of the two hosts while still all env in 4.3.10 and then take backup of engine and then install the second as 4.4.8 and see if it goes ok. I'm going to revert the 4.3.10 snapshot consistent env and try... Problems so far before the next test: . After the host deploy of the first 4.4 host (novirt3) I see that still the current config results with novirt1 as the SPM and old hosted engine storage as the master domain (I have iSCSI based SHE) --> all as expected? . the detach of the old engine storage gives at gui events OVFs update was ignored - nothing to update for storage domain 'hosted_storage_old_20210827T173854' Aug 27, 2021, 6:36:03 PM Storage Domain hosted_storage_old_20210827T173854 (Data Center Default) was deactivated and has moved to 'Preparing for maintenance' until it will no longer be accessed by any Host of the Data Center. 8/27/216:36:11 PM and the task seems completed, but continues to remain with the lock so I cannot indeed deactivate.... (even after novirt1 put into maintenance to be updated) It didn't complain that the old hosted engine storage was master and apparently switched another domain (DATA in my case) to master role.... I don't know if related or not . one not required network was not setup automatically on the new host (in practice in this lab I only have ovirtmgmt and this network...), so when I tried to live migrate VM of next host to be updated (novirt1) I got no hosts available due to that.... I went into setup host networks of novirt3 and added the network and then all went fine: I was able to live migrate and install/add the next hosted engine host (that is the last in my case, no ordinary hosts). Also on the second host I had to go through the setup host networks button. BTW: I was then able to update cluster and DC from 4.3 to 4.6 and shutdown/boot the VM after that Gianluca

On Sun, Aug 29, 2021 at 1:13 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: It should. From the DB, before adding it again. Matches on the uuid (search the code for unique_id_out if you want the details). Why? (I didn't follow all this thread, ignoring the rest for now...) Best regards, -- Didi

On Wed, Sep 1, 2021 at 4:26 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: So tried with 4.4.8 async 2 but the same problem [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check actual cluster location] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enable GlusterFS at cluster level] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set VLAN ID at datacenter level] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get active list of active firewalld zones] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Configure libvirt firewalld zone] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Include after_add_host tasks files] [ INFO ] You can now connect to https://novirt2.localdomain.local:6900/ovirt-engine/ and check the status of this host and eventually remediate it, please continue only when the host is listed as 'up' [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create temporary lock file] [ INFO ] changed: [localhost -> localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Pause execution until /tmp/ansible.wy3ichvk_he_setup_lock is removed, delete it once ready to proceed] the host keeps remaining as NoNResponsive in local engine and in engine.log the same 2021-09-10 08:44:51,481+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = novirt2.localdomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='ca9ff6f7-5a7c-4168-9632-998c52f76cfa', vds='Host[novirt2.localdomain.local,ca9ff6f7-5a7c-4168-9632-998c52f76cfa]'})' execution failed: java.net.ConnectException: Connection refused so the initial install/config of novirt2 doesn't start So the scenario is initial 4.3.10 with 2 hosts (novirt1 and novirt2) and 1 she engine (novmgr) iSCSI based storage: hosted_engine storage domain and one data storage domain This is nested env so that through snapshots I can try and repeat steps. novirt1 and novirt2 are two VMS under one oVirt 4.4 env composed by one single host and an external engine the steps: 1 vm running under novirt1 and hosted engine running under novir2 at the beginning . global maintenance . stop engine . backup . shutdown engine vm and scratch novirt2 actually I simulate scenario where I deploy novirt2 on a new hw, that is a clone of novirt2 VM Already tested (in previous version of 4.4.8) that if I go through a different hostname it works As novirt2 and novirt1 (in 4.3) are VMS running on the same hypervisor I see that in their hw details I have the same serial number and the usual random uuid novirt1 uuid B1EF9AFF-D4BD-41A1-B26E-7DD0CC440963 serial number 00fa984c-d5a1-e811-906e-00163566263e novirt2 uuid D584E962-5461-4FA5-AFFA-DB413E17590C serial number 00fa984c-d5a1-e811-906e-00163566263e and the new novirt2 that has a different uuid, being a clone has (from dmidecode) uuid: 10b9031d-a475-4b41-a134-bad2ede3cf11 serial Number: 00fa984c-d5a1-e811-906e-00163566263e Unfortunately I cannot try at the moment the scenario where I deploy the new novirt2 on the same virtual hw, because in the first 4.3 install I configured the OS disk as 50Gb and with this size 4.4.8 complains about insufficient space. And having the snapshot active in preview I cannot resize the disk Eventually I can reinstall 4.3 on an 80Gb disk and try the same, maintaining the same hw ... but this would imply that in general I cannot upgrade using different hw and reusing the same hostnames.... correct? anyway if you want to check generated logs at local engine side and novirt2 side here they are: Contents under /var/log of novmgr (tar.gz format) https://drive.google.com/file/d/1e4WwN4D8GDBpsGqwpwM40MGcLISeOzGO/view?us... Contents under /var/log/of novirt2 (tar.gz format) https://drive.google.com/file/d/1uQxlsbPVclW4xcAbCP8dXyIF2HlLqaR-/view?us... Backup made in 4.3 https://drive.google.com/file/d/19x4cUhXt2NQkmfTNS8AeLOC7mXtj6IYG/view?us... thanks Gianluca

On Sun, Sep 12, 2021 at 10:35 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: I thought that in 4.4.7 there was not this problem if you use the same hostname but with different (real or virtual) hw as the first host during your SHE upgrade from 4.3.10 to 4.4.7. But probably it was not so and I didn't remember correctly.... The automatic mgmt of host adding during the "hosted-engine --deploy --restore-from-file=backup.bck" step if you have different hw and you want to recycle your previous hostname. In the past it often happened to me to combine upgrades of systems with hw refreshing (with standalone hosts, rhcs clusters, also ovirt/rhv from 4.2 to 4.3 if I remember correctly, ecc.) where you re-use an existing hostname on new hardware More than a bug it would be an RFE perhaps.... My env is nested oVirt and my hypervisors are Vms. I notice that in oVirt if you clone a VM it changes the uuid in the clone but it retains the serial number... I confirm that I reverted the snapshots of the 2 VMs used as hypervisors taking them again at initial 4.3 status and remade all the steps, but right after the install of the OS of 4.4.8 oVirt node I created /etc/vdsm/vdsm.id inside novirt2 with the old 4.3 value (the file was not there at that moment) and then all the flow went as expected and I was then able to reach the final 4.4.8 async 2 env with both hosts at 4.4.8, cluster and DC updated to 4.6 compatibility level and no downtime for the VMs inside the env, because I was able to execute live migration after upgrading the first host Perhaps, if you do want to open a bug, it should say something like: My considerations, as explained at the beginning, were to give the chance to reuse the hostname (often the oVirt admin is not responsible for hostname creation/mgmt) if you want to leverage new hw in combination with the upgrade process. Thanks for all the other considerations you put into your answer. Gianluca