[ovirt-users] Re: Ovirt node 4.4.5 failure to upgrade to 4.4.6

On Tue, Jun 8, 2021 at 8:01 AM Guillaume Pavese < guillaume.pavese(a)interactiv-group.com&gt; wrote: > Hello, > > I used the cluster upgrade feature that moves hosts in maintenance one by > one. > This is not a HCI cluster, my storage is on iSCSI multipath > > I managed to fully upgrade the 1st hosts after rebooting and fixing some > network/iSCSI errors. > However, now the second one is stuck at upgrading the ovirt-node layers > too but I can not succeed in upgrading that one. > On this 2nd host, the workaround of removing and reinstalling > ovirt-node-ng-image-update doesn't work. I only get the following error : > > [root@ps-inf-prd-kvm-fr-511 ~]# nodectl check > Status: OK > Bootloader ... OK > Layer boot entries ... OK > Valid boot entries ... OK > Mount points ... OK > Separate /var ... OK > Discard is used ... OK > Basic storage ... OK > Initialized VG ... OK > Initialized Thin Pool ... OK > Initialized LVs ... OK > Thin storage ... OK > Checking available space in thinpool ... OK > Checking thinpool auto-extend ... OK > vdsmd ... OK > > > [root@ps-inf-prd-kvm-fr-511 ~]# nodectl info > bootloader: > default: ovirt-node-ng-4.4.5.1-0.20210323.0 > (4.18.0-240.15.1.el8_3.x86_64) > entries: > ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): > index: 0 > kernel: > /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 > args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 > rd.lvm.lv=onn/swap rhgb quiet > boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard > img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on > root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 > initrd: > /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img > title: ovirt-node-ng-4.4.5.1-0.20210323.0 > (4.18.0-240.15.1.el8_3.x86_64) > blsid: > ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 > layers: > ovirt-node-ng-4.4.5.1-0.20210323.0: > ovirt-node-ng-4.4.5.1-0.20210323.0+1 > current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 > > > [root@ps-inf-prd-kvm-fr-511 ~]# yum remove ovirt-node-ng-image-update > [...] > Removing: > ovirt-node-ng-image-update noarch > 4.4.6.3-1.el8 > @ovirt-4.4 886 M > Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm > > Removed: > ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Complete! > [root@ps-inf-prd-kvm-fr-511 ~]# > > > [root@ps-inf-prd-kvm-fr-511 ~]# yum install ovirt-node-ng-image-update > [...] > Installing: > ovirt-node-ng-image-update noarch > 4.4.6.3-1.el8 > ovirt-4.4 887 M > [...] > ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm > > 23 MB/s | 887 MB 00:39 > Running transaction check > Transaction check succeeded. > Running transaction test > Transaction test succeeded. > Running transaction > Preparing : > Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > *warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) > scriptlet failed, exit status 1* > > *Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update * > Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > > > Installed: > > > ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch > Complete! > [root@ps-inf-prd-kvm-fr-511 ~]# > > [root@ps-inf-prd-kvm-fr-511 ~]# nodectl info > bootloader: > default: ovirt-node-ng-4.4.5.1-0.20210323.0 > (4.18.0-240.15.1.el8_3.x86_64) > entries: > ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): > index: 0 > kernel: > /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 > args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 > rd.lvm.lv=onn/swap rhgb quiet > boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard > img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on > root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 > initrd: > /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img > title: ovirt-node-ng-4.4.5.1-0.20210323.0 > (4.18.0-240.15.1.el8_3.x86_64) > blsid: > ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 > layers: > ovirt-node-ng-4.4.5.1-0.20210323.0: > ovirt-node-ng-4.4.5.1-0.20210323.0+1 > current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 > [root@ps-inf-prd-kvm-fr-511 ~]# > > Is there any way to see where in the POSTIN scriplet the installation > fails? > You can try using dnf's option '--rpmverbosity'. Best regards, > > Guillaume Pavese > Ingénieur Système et Réseau > Interactiv-Group > > > On Fri, Jun 4, 2021 at 8:44 PM Lev Veyde <lveyde(a)redhat.com&gt; wrote: > >> Hi Guillaume, >> >> Have you moved the host to the maintenance before the upgrade (making >> sure that Gluster related options are unchecked)? >> >> Or you started the upgrade directly? >> >> Thanks in advance, >> >> On Thu, Jun 3, 2021 at 10:50 AM wodel youchi <wodel.youchi(a)gmail.com&gt; >> wrote: >> >>> Hi, >>> >>> Is this an hci deployment? >>> >>> If yes : >>> >>> - try to boot using the old version 4.4.5 >>> - verify that your network configuration is still intact >>> - verify that the gluster part is working properly >>> # gluster peer status >>> # gluster volume status >>> >>> - verify the your engine and your host can resolve each other hostname >>> >>> If all is ok try to bring the host at available state >>> >>> >>> Regards. >>> >>> Le mer. 2 juin 2021 08:21, Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; a écrit : >>> >>>> Maybe my problem is in part linked to an issue seen by Jayme earlier, >>>> but then the resolution that worked for him did not succeed for me : >>>> >>>> I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then >>>> upgraded it to Centos-Stream and rebooted >>>> >>>> Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it >>>> failed at the first host. >>>> They are all ovir-node hosts, originally first installed in 4.4.5 >>>> >>>> In Host Event Logs I saw : >>>> >>>> ... >>>> Update of host ps-inf-prd-kvm-fr-510.hostics.fr. >>>> Upgrade packages >>>> Update of host ps-inf-prd-kvm-fr-510.hostics.fr. >>>> Check if image was updated. >>>> Update of host ps-inf-prd-kvm-fr-510.hostics.fr. >>>> Check if image was updated. >>>> Update of host ps-inf-prd-kvm-fr-510.hostics.fr. >>>> Check if image-updated file exists. >>>> Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: >>>> gpav(a)hostics.fr). >>>> >>>> >>>> >>>> ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed >>>> according to yum, >>>> I tried reinstalling it but got errors: "Error in POSTIN scriptlet" : >>>> >>>> Downloading Packages: >>>> [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already >>>> downloaded >>>> ... >>>> Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch >>>> >>>> Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch >>>> >>>> Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch >>>> >>>> warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) >>>> scriptlet failed, exit status 1 >>>> >>>> Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update >>>> --- >>>> Reinstalled: >>>> ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch >>>> >>>> >>>> >>>> nodectl still showed it was on 4.4.5 : >>>> >>>> [root@ps-inf-prd-kvm-fr-510 ~]# nodectl info >>>> bootloader: >>>> default: ovirt-node-ng-4.4.5.1-0.20210323.0 >>>> (4.18.0-240.15.1.el8_3.x86_64) >>>> ... >>>> current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 >>>> >>>> >>>> >>>> I tried to upgrade the Host again from oVirt and this time there was >>>> no error, and the host rebooted. >>>> However, it did not pass active after rebooting and nodectl still >>>> shows that it's 4.4.5 installed. Similar symptoms as OP >>>> >>>> So I removed ovirt-node-ng-image-update, then reinstalled it and >>>> got no error this time. >>>> nodectl info seemed to show that it was installed : >>>> >>>> >>>> [root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info >>>> bootloader: >>>> default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) >>>> ... >>>> current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 >>>> >>>> >>>> However, after reboot the Host was still shown as "unresponsive" >>>> After Marking it as "Manually rebooted", passing it in maintenance >>>> mode and trying to activate it, the Host was automatically fenced. And >>>> still unresponsive after this new reboot. >>>> >>>> I passed it in maintenance mode again, And tried to reinstall it with >>>> "Deploy Hosted Engine" selected >>>> However if failed : "Task Stop services failed to execute." >>>> >>>> In >>>> /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log >>>> : >>>> >>>> "msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service >>>> canceled.\n", "failed" : true, >>>> >>>> "msg" : "Unable to stop service supervdsmd.service: Job for >>>> supervdsmd.service canceled.\n", failed" : true, >>>> >>>> "stderr" : "Error: ServiceOperationError: _systemctlStop >>>> failed\nb'Job for vdsmd.service canceled.\\n' ", >>>> >>>> "stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop >>>> failed", "b'Job for vdsmd.service canceled.\\n' " ], >>>> >>>> >>>> If I try on the Host I get : >>>> >>>> [root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd >>>> Job for vdsmd.service canceled. >>>> >>>> [root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd >>>> ● vdsmd.service - Virtual Desktop Server Manager >>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; >>>> vendor preset: disabled) >>>> Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 >>>> CEST; 7s ago >>>> Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh >>>> --pre-start (code=exited, status=0/SUCCESS) >>>> ... >>>> >>>> Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN >>>> Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? >>>> ... >>>> Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN >>>> Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask >>>> {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': >>>> {'storage> >>>> File: >>>> "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap >>>> >>>> self._bootstrap_inner() >>>> >>>> >>>> >>>> Retrying to manually stop vdsmd a second time then seems to work... >>>> I tried rebooting again, restarting the install always fail at the the >>>> same spot >>>> >>>> What should I try to get this host back up? >>>> >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> Ce message et toutes les pièces jointes (ci-après le “message”) sont >>>> établis à l’intention exclusive de ses destinataires et sont confidentiels. >>>> Si vous recevez ce message par erreur, merci de le détruire et d’en avertir >>>> immédiatement l’expéditeur. Toute utilisation de ce message non conforme a >>>> sa destination, toute diffusion ou toute publication, totale ou partielle, >>>> est interdite, sauf autorisation expresse. L’internet ne permettant pas >>>> d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) >>>> décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse >>>> ou il aurait été modifié. IT, ES, UK. >>>> <https://interactiv-group.com/disclaimer.html> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6S... >>>> >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/TX2CUCG6DDS... >>> >> >> >> -- >> >> Lev Veyde >> >> Senior Software Engineer, RHCE | RHCVA | MCITP >> >> Red Hat Israel >> >> <https://www.redhat.com> >> >> lev(a)redhat.com | lveyde(a)redhat.com >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> > > Ce message et toutes les pièces jointes (ci-après le “message”) sont > établis à l’intention exclusive de ses destinataires et sont confidentiels. > Si vous recevez ce message par erreur, merci de le détruire et d’en avertir > immédiatement l’expéditeur. Toute utilisation de ce message non conforme a > sa destination, toute diffusion ou toute publication, totale ou partielle, > est interdite, sauf autorisation expresse. L’internet ne permettant pas > d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) > décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse > ou il aurait été modifié. IT, ES, UK. > <https://interactiv-group.com/disclaimer.html> > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q3RYE4KCAJ3... > -- Didi