Ovirt node 4.4.5 failure to upgrade to 4.4.6
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me : I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5 In Host Event Logs I saw : ... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr). ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" : Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1 Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch nodectl still showed it was on 4.4.5 : [root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed : [root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot. I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute." In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log : "msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true, "msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true, "stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ", "stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ], If I try on the Host I get : [root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled. [root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ... Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot What should I try to get this host back up? Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>
Hi, Is this an hci deployment? If yes : - try to boot using the old version 4.4.5 - verify that your network configuration is still intact - verify that the gluster part is working properly # gluster peer status # gluster volume status - verify the your engine and your host can resolve each other hostname If all is ok try to bring the host at available state Regards. Le mer. 2 juin 2021 08:21, Guillaume Pavese < guillaume.pavese@interactiv-group.com> a écrit :
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me :
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5
In Host Event Logs I saw :
... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr).
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :
Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
nodectl still showed it was on 4.4.5 :
[root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed :
[root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute."
In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log :
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ],
If I try on the Host I get :
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled.
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot
What should I try to get this host back up?
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6SX53...
Hi Guillaume, Have you moved the host to the maintenance before the upgrade (making sure that Gluster related options are unchecked)? Or you started the upgrade directly? Thanks in advance, On Thu, Jun 3, 2021 at 10:50 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
Is this an hci deployment?
If yes :
- try to boot using the old version 4.4.5 - verify that your network configuration is still intact - verify that the gluster part is working properly # gluster peer status # gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese < guillaume.pavese@interactiv-group.com> a écrit :
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me :
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5
In Host Event Logs I saw :
... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr).
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :
Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
nodectl still showed it was on 4.4.5 :
[root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed :
[root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute."
In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log :
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ],
If I try on the Host I get :
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled.
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot
What should I try to get this host back up?
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6SX53...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TX2CUCG6DDSPUQ...
-- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev@redhat.com | lveyde@redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Hello, I used the cluster upgrade feature that moves hosts in maintenance one by one. This is not a HCI cluster, my storage is on iSCSI multipath I managed to fully upgrade the 1st hosts after rebooting and fixing some network/iSCSI errors. However, now the second one is stuck at upgrading the ovirt-node layers too but I can not succeed in upgrading that one. On this 2nd host, the workaround of removing and reinstalling ovirt-node-ng-image-update doesn't work. I only get the following error : [root@ps-inf-prd-kvm-fr-511 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK [root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 [root@ps-inf-prd-kvm-fr-511 ~]# yum remove ovirt-node-ng-image-update [...] Removing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 @ovirt-4.4 886 M Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm Removed: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]# [root@ps-inf-prd-kvm-fr-511 ~]# yum install ovirt-node-ng-image-update [...] Installing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 ovirt-4.4 887 M [...] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm 23 MB/s | 887 MB 00:39 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch *warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1* *Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update * Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Installed: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]# [root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 [root@ps-inf-prd-kvm-fr-511 ~]# Is there any way to see where in the POSTIN scriplet the installation fails? Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Fri, Jun 4, 2021 at 8:44 PM Lev Veyde <lveyde@redhat.com> wrote:
Hi Guillaume,
Have you moved the host to the maintenance before the upgrade (making sure that Gluster related options are unchecked)?
Or you started the upgrade directly?
Thanks in advance,
On Thu, Jun 3, 2021 at 10:50 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
Is this an hci deployment?
If yes :
- try to boot using the old version 4.4.5 - verify that your network configuration is still intact - verify that the gluster part is working properly # gluster peer status # gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese < guillaume.pavese@interactiv-group.com> a écrit :
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me :
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5
In Host Event Logs I saw :
... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr).
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :
Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
nodectl still showed it was on 4.4.5 :
[root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed :
[root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute."
In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log :
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ],
If I try on the Host I get :
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled.
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot
What should I try to get this host back up?
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6SX53...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TX2CUCG6DDSPUQ...
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
lev@redhat.com | lveyde@redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>
On Tue, Jun 8, 2021 at 8:01 AM Guillaume Pavese < guillaume.pavese@interactiv-group.com> wrote:
Hello,
I used the cluster upgrade feature that moves hosts in maintenance one by one. This is not a HCI cluster, my storage is on iSCSI multipath
I managed to fully upgrade the 1st hosts after rebooting and fixing some network/iSCSI errors. However, now the second one is stuck at upgrading the ovirt-node layers too but I can not succeed in upgrading that one. On this 2nd host, the workaround of removing and reinstalling ovirt-node-ng-image-update doesn't work. I only get the following error :
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
[root@ps-inf-prd-kvm-fr-511 ~]# yum remove ovirt-node-ng-image-update [...] Removing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 @ovirt-4.4 886 M Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
Removed: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]#
[root@ps-inf-prd-kvm-fr-511 ~]# yum install ovirt-node-ng-image-update [...] Installing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 ovirt-4.4 887 M [...] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
23 MB/s | 887 MB 00:39 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch *warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1*
*Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update * Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Installed:
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]#
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 [root@ps-inf-prd-kvm-fr-511 ~]#
Is there any way to see where in the POSTIN scriplet the installation fails?
You can try using dnf's option '--rpmverbosity'. Best regards,
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
On Fri, Jun 4, 2021 at 8:44 PM Lev Veyde <lveyde@redhat.com> wrote:
Hi Guillaume,
Have you moved the host to the maintenance before the upgrade (making sure that Gluster related options are unchecked)?
Or you started the upgrade directly?
Thanks in advance,
On Thu, Jun 3, 2021 at 10:50 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
Is this an hci deployment?
If yes :
- try to boot using the old version 4.4.5 - verify that your network configuration is still intact - verify that the gluster part is working properly # gluster peer status # gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese < guillaume.pavese@interactiv-group.com> a écrit :
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me :
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5
In Host Event Logs I saw :
... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr).
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :
Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
nodectl still showed it was on 4.4.5 :
[root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed :
[root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute."
In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log :
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ],
If I try on the Host I get :
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled.
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot
What should I try to get this host back up?
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6SX53...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TX2CUCG6DDSPUQ...
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
lev@redhat.com | lveyde@redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q3RYE4KCAJ3CPR...
-- Didi
Thank you, I'll try on the 3rd host if the same problem happens (likely) In the meantime I managed to advance a bit on my 2nd host (but I did not get the chance to try your suggestion of '--rpmverbosity') The failure to install ovirt-node-ng-image-update seems to be linked with the errors messages that I reported previously regarding the impossibility to stop vdsmd. On the hosts that were refusing to update or install the rpm, I was also unable to stop vdsmd or supervdsmd : [root@ps-inf-prd-kvm-fr-511 ~]# systemctl stop vdsmd Job for vdsmd.service canceled. [root@ps-inf-prd-kvm-fr-511 ~]# systemctl stop supervdsmd Job for supervdsmd.service canceled. I managed to stop them only after stopping ovirt-ha-broker & ovirt-ha-agent Once these services down, the manual install of ovirt-node-ng-image-update succeeded. iSCSI multipath was not working after reboot, I had to manually rediscover the targets After all that, I could finally do a successful Host Reinstall from oVirt. Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Tue, Jun 8, 2021 at 2:24 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Tue, Jun 8, 2021 at 8:01 AM Guillaume Pavese < guillaume.pavese@interactiv-group.com> wrote:
Hello,
I used the cluster upgrade feature that moves hosts in maintenance one by one. This is not a HCI cluster, my storage is on iSCSI multipath
I managed to fully upgrade the 1st hosts after rebooting and fixing some network/iSCSI errors. However, now the second one is stuck at upgrading the ovirt-node layers too but I can not succeed in upgrading that one. On this 2nd host, the workaround of removing and reinstalling ovirt-node-ng-image-update doesn't work. I only get the following error :
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
[root@ps-inf-prd-kvm-fr-511 ~]# yum remove ovirt-node-ng-image-update [...] Removing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 @ovirt-4.4 886 M Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
Removed: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]#
[root@ps-inf-prd-kvm-fr-511 ~]# yum install ovirt-node-ng-image-update [...] Installing: ovirt-node-ng-image-update noarch 4.4.6.3-1.el8 ovirt-4.4 887 M [...] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
23 MB/s | 887 MB 00:39 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch *warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1*
*Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update * Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Installed:
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch Complete! [root@ps-inf-prd-kvm-fr-511 ~]#
[root@ps-inf-prd-kvm-fr-511 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) entries: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64): index: 0 kernel: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64 args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 rd.lvm.lv=onn/swap rhgb quiet boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 initrd: /boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img title: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) blsid: ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64 layers: ovirt-node-ng-4.4.5.1-0.20210323.0: ovirt-node-ng-4.4.5.1-0.20210323.0+1 current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1 [root@ps-inf-prd-kvm-fr-511 ~]#
Is there any way to see where in the POSTIN scriplet the installation fails?
You can try using dnf's option '--rpmverbosity'.
Best regards,
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
On Fri, Jun 4, 2021 at 8:44 PM Lev Veyde <lveyde@redhat.com> wrote:
Hi Guillaume,
Have you moved the host to the maintenance before the upgrade (making sure that Gluster related options are unchecked)?
Or you started the upgrade directly?
Thanks in advance,
On Thu, Jun 3, 2021 at 10:50 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
Is this an hci deployment?
If yes :
- try to boot using the old version 4.4.5 - verify that your network configuration is still intact - verify that the gluster part is working properly # gluster peer status # gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese < guillaume.pavese@interactiv-group.com> a écrit :
Maybe my problem is in part linked to an issue seen by Jayme earlier, but then the resolution that worked for him did not succeed for me :
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed at the first host. They are all ovir-node hosts, originally first installed in 4.4.5
In Host Event Logs I saw :
... Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Upgrade packages Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image was updated. Update of host ps-inf-prd-kvm-fr-510.hostics.fr. Check if image-updated file exists. Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User: gpav@hostics.fr).
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to yum, I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :
Downloading Packages: [SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already downloaded ... Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update --- Reinstalled: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
nodectl still showed it was on 4.4.5 :
[root@ps-inf-prd-kvm-fr-510 ~]# nodectl info bootloader: default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no error, and the host rebooted. However, it did not pass active after rebooting and nodectl still shows that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no error this time. nodectl info seemed to show that it was installed :
[root@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info bootloader: default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64) ... current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive" After Marking it as "Manually rebooted", passing it in maintenance mode and trying to activate it, the Host was automatically fenced. And still unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with "Deploy Hosted Engine" selected However if failed : "Task Stop services failed to execute."
In /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log :
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed", "b'Job for vdsmd.service canceled.\\n' " ],
If I try on the Host I get :
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd Job for vdsmd.service canceled.
[root@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST; 7s ago Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) ...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? ... Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage> File: "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work... I tried rebooting again, restarting the install always fail at the the same spot
What should I try to get this host back up?
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZB2CLJYXO6SX53...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TX2CUCG6DDSPUQ...
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
lev@redhat.com | lveyde@redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q3RYE4KCAJ3CPR...
-- Didi
-- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>
participants (4)
-
Guillaume Pavese -
Lev Veyde -
wodel youchi -
Yedidyah Bar David