May 2021 - Users - oVirt List Archives

Moving interfaces for switch maintenance
by David White 12 May '21

12 May '21

So I have two switches. All 3 of my HCI oVirt servers are connected to both switches. 1 switch serves the ovirtmgmt network (internal, gluster communication and everything else on that subnet) The other switch serves the "main" front-end network (Private). It turns out that my datacenter plugged the switches into the wrong power supplies, and now needs to move them. My switches have single PSUs, so when we move them, the network will go down. Obviously, I'm just going to move 1 switch at a time. What I'm wondering, though, is there's an "easy" way to force all the traffic for both networks out one of the single interfaces, so that I don't experience any more downtime during this switch maintenance. Should I assign another temporary IP address on each of the interfaces for each server that is in the same subnet as the switch we're going to take down? Sent with ProtonMail Secure Email.

1 0

Issue upgrading from 4.3 (Centos 7) to 4.4 (Centos 8)
by ling＠aliko.com 12 May '21

12 May '21

Hello, I have been trying to upgrade my self-hosted engine from 4.3 to 4.4 but running into issue while performing hosted-engine deploy. Old hypervisor hosts are all running Centos 8 and old ovirt-engine is also running Centos 7. I created a brand new baremental node running Centos 8, Kernel 4.18.0-240.15.1.el8_3.x86_64 and the following engine versions: ovirt-hosted-engine-setup-2.4.9-1.el8.noarch ovirt-hosted-engine-ha-2.4.6-1.el8.noarch ovirt-engine-appliance-4.4-20210323171213.1.el8.x86_64 python3-ovirt-engine-sdk4-4.4.10-1.el8.x86_64 I have many VLANs in my environment. But on this host, I only have these network devices set up (eth0 is the main network, eth1 for storage): # nmcli con NAME UUID TYPE DEVICE ovirtmgmt 02f64861-d992-4e56-8cec-da1906bac09f bridge ovirtmgmt System eth1 bd9e565f-bdc3-4e43-bbd3-5875b9d7fed7 ethernet eth1 virbr0 78e6875d-70f6-4c89-89dd-180dbb9250b1 bridge virbr0 eth0 743b0e26-aae7-44b8-9215-3754a537e90b ethernet eth0 vnet0 bcfead6d-c5b6-4428-9f89-41589735be02 tun vnet0 When I run hosted-engine --deploy --restore-from-file=backup_050321.bck, it hangs after showing: [ INFO ] TASK [ovirt.ovirt.engine_setup : Copy yum configuration file] [ INFO ] changed: [localhost -> ovirt.safari.apple.com] [ INFO ] TASK [ovirt.ovirt.engine_setup : Set 'best' to false] [ INFO ] changed: [localhost -> ovirt.safari.apple.com] [ INFO ] TASK [ovirt.ovirt.engine_setup : Update all packages] virsh shows the VM is in paused state: # virsh list Id Name State ---------------------------------- 1 HostedEngineLocal paused I was able to ssh onto the VM until that point. Do I need to set up all the network connection for all the VLANs before running the deploy script? And how about engine storage domain? I have a new NFS mount ready but it did not ask me about which storage domain to use. Will it ask in later stage? Thanks.

4 12

Something broke & took down multiple VMs for ~20 minutes
by David White 11 May '21

11 May '21

As the subject suggestions, something in oVirt HCI broke. I have no idea what, and it recovered on its own after about 20 minutes or so. I believe that the issue was limited to a single host (although I don't know that for sure), as we had two VMs go completely unresponsive, but a 3rd VM remained operational. For a while during the outage, I was able to log into the oVirt admin web portal, and I noticed at least 1-2 of my hosts (I have 3 hosts) showed the problematic VMs as being problematic inside of oVirt. Reviewing the oVirt Events, I see that this basically started right when the ETL Service Started. There were no events before that point since yesterday, but right when the ETL Service started, it seems like all hell broke loose. oVirt detected "No faulty multipaths" on any of the hosts, but then very quickly started indicating that hosts, vms, and storage targets were unavailable. See my screenshot below. Around 30 - 35 minutes later, it appears that the Hosted Engine terminated due to a storage issue, and auto recovered on a different host. There's a 2nd screenshot beneath the first. Everything came back up shortly before 9am, and has been stable since. In fact, the Volume replication issues that I saw in my environment after I performed maintenance on 1 of my hosts on Friday are no longer present. It appears that the Hosted Engine sees the storage as being perfectly healthy. How do I even begin to figure out what happened, and try to prevent it from happening again? [Screenshot from 2021-04-26 16-36-47.png] [Screenshot from 2021-04-26 16-44-08.png] Sent with ProtonMail Secure Email.

2 7

pool list vm assign user
by Dominique D 11 May '21

11 May '21

Is there a way to know how to see who the vm of a pool assigned to? I am able on the portal to see those who are "logged-in user" but the others VM I don't know to whom they are assigned.

2 3

[ANN] Async oVirt Node release for oVirt 4.4.6
by Sandro Bonazzola 11 May '21

11 May '21

On May 10th 2021 the oVirt project released an async update of oVirt Node (4.4.6.1) Changes: - Updated Advanced Virtualization packages - Updated ovn2.11 and openvswitch2.11 - Updated ansible 2.9.21 ( https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.… ) - Update libssh (fixes CVE-2020-16135) Full diff list: --- ovirt-node-ng-image-4.4.6.manifest-rpm 2021-05-04 17:02:01.839123874 +0200 +++ ovirt-node-ng-image-4.4.6.1.manifest-rpm 2021-05-11 08:39:44.714649170 +0200 @@ -24 +24 @@ -ansible-2.9.20-2.el8.noarch +ansible-2.9.21-2.el8.noarch @@ -89 +89 @@ -corosynclib-3.1.0-4.el8.0.1.x86_64 +corosynclib-3.1.0-5.el8.x86_64 @@ -100 +100 @@ -cups-libs-2.2.6-38.el8.x86_64 +cups-libs-2.2.6-39.el8.x86_64 @@ -130,5 +130,5 @@ -dracut-049-135.git20210121.el8.x86_64 -dracut-config-generic-049-135.git20210121.el8.x86_64 -dracut-live-049-135.git20210121.el8.x86_64 -dracut-network-049-135.git20210121.el8.x86_64 -dracut-squash-049-135.git20210121.el8.x86_64 +dracut-049-136.git20210426.el8.x86_64 +dracut-config-generic-049-136.git20210426.el8.x86_64 +dracut-live-049-136.git20210426.el8.x86_64 +dracut-network-049-136.git20210426.el8.x86_64 +dracut-squash-049-136.git20210426.el8.x86_64 @@ -148,36 +148,36 @@ -fence-agents-all-4.2.1-67.el8.x86_64 -fence-agents-amt-ws-4.2.1-67.el8.noarch -fence-agents-apc-4.2.1-67.el8.noarch -fence-agents-apc-snmp-4.2.1-67.el8.noarch -fence-agents-bladecenter-4.2.1-67.el8.noarch -fence-agents-brocade-4.2.1-67.el8.noarch -fence-agents-cisco-mds-4.2.1-67.el8.noarch -fence-agents-cisco-ucs-4.2.1-67.el8.noarch -fence-agents-common-4.2.1-67.el8.noarch -fence-agents-compute-4.2.1-67.el8.noarch -fence-agents-drac5-4.2.1-67.el8.noarch -fence-agents-eaton-snmp-4.2.1-67.el8.noarch -fence-agents-emerson-4.2.1-67.el8.noarch -fence-agents-eps-4.2.1-67.el8.noarch -fence-agents-heuristics-ping-4.2.1-67.el8.noarch -fence-agents-hpblade-4.2.1-67.el8.noarch -fence-agents-ibmblade-4.2.1-67.el8.noarch -fence-agents-ifmib-4.2.1-67.el8.noarch -fence-agents-ilo-moonshot-4.2.1-67.el8.noarch -fence-agents-ilo-mp-4.2.1-67.el8.noarch -fence-agents-ilo-ssh-4.2.1-67.el8.noarch -fence-agents-ilo2-4.2.1-67.el8.noarch -fence-agents-intelmodular-4.2.1-67.el8.noarch -fence-agents-ipdu-4.2.1-67.el8.noarch -fence-agents-ipmilan-4.2.1-67.el8.noarch -fence-agents-kdump-4.2.1-67.el8.x86_64 -fence-agents-mpath-4.2.1-67.el8.noarch -fence-agents-redfish-4.2.1-67.el8.x86_64 -fence-agents-rhevm-4.2.1-67.el8.noarch -fence-agents-rsa-4.2.1-67.el8.noarch -fence-agents-rsb-4.2.1-67.el8.noarch -fence-agents-sbd-4.2.1-67.el8.noarch -fence-agents-scsi-4.2.1-67.el8.noarch -fence-agents-vmware-rest-4.2.1-67.el8.noarch -fence-agents-vmware-soap-4.2.1-67.el8.noarch -fence-agents-wti-4.2.1-67.el8.noarch +fence-agents-all-4.2.1-68.el8.x86_64 +fence-agents-amt-ws-4.2.1-68.el8.noarch +fence-agents-apc-4.2.1-68.el8.noarch +fence-agents-apc-snmp-4.2.1-68.el8.noarch +fence-agents-bladecenter-4.2.1-68.el8.noarch +fence-agents-brocade-4.2.1-68.el8.noarch +fence-agents-cisco-mds-4.2.1-68.el8.noarch +fence-agents-cisco-ucs-4.2.1-68.el8.noarch +fence-agents-common-4.2.1-68.el8.noarch +fence-agents-compute-4.2.1-68.el8.noarch +fence-agents-drac5-4.2.1-68.el8.noarch +fence-agents-eaton-snmp-4.2.1-68.el8.noarch +fence-agents-emerson-4.2.1-68.el8.noarch +fence-agents-eps-4.2.1-68.el8.noarch +fence-agents-heuristics-ping-4.2.1-68.el8.noarch +fence-agents-hpblade-4.2.1-68.el8.noarch +fence-agents-ibmblade-4.2.1-68.el8.noarch +fence-agents-ifmib-4.2.1-68.el8.noarch +fence-agents-ilo-moonshot-4.2.1-68.el8.noarch +fence-agents-ilo-mp-4.2.1-68.el8.noarch +fence-agents-ilo-ssh-4.2.1-68.el8.noarch +fence-agents-ilo2-4.2.1-68.el8.noarch +fence-agents-intelmodular-4.2.1-68.el8.noarch +fence-agents-ipdu-4.2.1-68.el8.noarch +fence-agents-ipmilan-4.2.1-68.el8.noarch +fence-agents-kdump-4.2.1-68.el8.x86_64 +fence-agents-mpath-4.2.1-68.el8.noarch +fence-agents-redfish-4.2.1-68.el8.x86_64 +fence-agents-rhevm-4.2.1-68.el8.noarch +fence-agents-rsa-4.2.1-68.el8.noarch +fence-agents-rsb-4.2.1-68.el8.noarch +fence-agents-sbd-4.2.1-68.el8.noarch +fence-agents-scsi-4.2.1-68.el8.noarch +fence-agents-vmware-rest-4.2.1-68.el8.noarch +fence-agents-vmware-soap-4.2.1-68.el8.noarch +fence-agents-wti-4.2.1-68.el8.noarch @@ -187 +187 @@ -filesystem-3.8-4.el8.x86_64 +filesystem-3.8-3.el8.x86_64 @@ -193 +193 @@ -freetype-2.9.1-5.el8.x86_64 +freetype-2.9.1-4.el8_3.1.x86_64 @@ -199 +199 @@ -fwupd-1.5.5-3.el8.x86_64 +fwupd-1.5.9-1.el8.x86_64 @@ -210 +210 @@ -glib2-2.56.4-10.el8.x86_64 +glib2-2.56.4-11.el8.x86_64 @@ -269,2 +269,2 @@ -iproute-5.9.0-4.el8.x86_64 -iproute-tc-5.9.0-4.el8.x86_64 +iproute-5.12.0-0.el8.x86_64 +iproute-tc-5.12.0-0.el8.x86_64 @@ -317,2 +317,2 @@ -krb5-libs-1.18.2-9.el8.x86_64 -krb5-workstation-1.18.2-9.el8.x86_64 +krb5-libs-1.18.2-10.el8.x86_64 +krb5-workstation-1.18.2-10.el8.x86_64 @@ -381 +381 @@ -libgcc-8.4.1-1.el8.x86_64 +libgcc-8.4.1-2.1.el8.x86_64 @@ -393 +393 @@ -libgomp-8.4.1-1.el8.x86_64 +libgomp-8.4.1-2.1.el8.x86_64 @@ -409 +409 @@ -libkadm5-1.18.2-9.el8.x86_64 +libkadm5-1.18.2-10.el8.x86_64 @@ -469,2 +469,2 @@ -libssh-0.9.4-2.el8.x86_64 -libssh-config-0.9.4-2.el8.noarch +libssh-0.9.4-3.el8.x86_64 +libssh-config-0.9.4-3.el8.noarch @@ -476 +476 @@ -libstdc++-8.4.1-1.el8.x86_64 +libstdc++-8.4.1-2.1.el8.x86_64 @@ -498,26 +498,26 @@ -libvirt-7.0.0-9.el8s.x86_64 -libvirt-admin-7.0.0-9.el8s.x86_64 -libvirt-bash-completion-7.0.0-9.el8s.x86_64 -libvirt-client-7.0.0-9.el8s.x86_64 -libvirt-daemon-7.0.0-9.el8s.x86_64 -libvirt-daemon-config-network-7.0.0-9.el8s.x86_64 -libvirt-daemon-config-nwfilter-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-interface-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-network-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-nodedev-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-nwfilter-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-qemu-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-secret-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-core-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-disk-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-gluster-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-iscsi-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-iscsi-direct-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-logical-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-mpath-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-rbd-7.0.0-9.el8s.x86_64 -libvirt-daemon-driver-storage-scsi-7.0.0-9.el8s.x86_64 -libvirt-daemon-kvm-7.0.0-9.el8s.x86_64 -libvirt-libs-7.0.0-9.el8s.x86_64 -libvirt-lock-sanlock-7.0.0-9.el8s.x86_64 +libvirt-7.0.0-14.el8s.x86_64 +libvirt-admin-7.0.0-14.el8s.x86_64 +libvirt-bash-completion-7.0.0-14.el8s.x86_64 +libvirt-client-7.0.0-14.el8s.x86_64 +libvirt-daemon-7.0.0-14.el8s.x86_64 +libvirt-daemon-config-network-7.0.0-14.el8s.x86_64 +libvirt-daemon-config-nwfilter-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-interface-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-network-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-nodedev-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-nwfilter-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-qemu-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-secret-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-core-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-disk-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-gluster-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-iscsi-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-iscsi-direct-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-logical-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-mpath-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-rbd-7.0.0-14.el8s.x86_64 +libvirt-daemon-driver-storage-scsi-7.0.0-14.el8s.x86_64 +libvirt-daemon-kvm-7.0.0-14.el8s.x86_64 +libvirt-libs-7.0.0-14.el8s.x86_64 +libvirt-lock-sanlock-7.0.0-14.el8s.x86_64 @@ -533 +533,2 @@ -libxcrypt-4.1.1-5.el8.x86_64 +libxcrypt-4.1.1-6.el8.x86_64 +libxkbcommon-0.9.1-1.el8.x86_64 @@ -618,2 +619,2 @@ -openscap-1.3.4-5.el8.x86_64 -openscap-scanner-1.3.4-5.el8.x86_64 +openscap-1.3.5-2.el8.x86_64 +openscap-scanner-1.3.5-2.el8.x86_64 @@ -626 +627 @@ -openvswitch2.11-2.11.0-50.el8.x86_64 +openvswitch2.11-2.11.3-87.el8s.x86_64 @@ -642 +643 @@ -ovirt-node-ng-image-update-placeholder-4.4.6-1.el8.noarch +ovirt-node-ng-image-update-placeholder-4.4.6.1-1.el8.noarch @@ -650,2 +651,2 @@ -ovirt-release-host-node-4.4.6-1.el8.noarch -ovirt-release44-4.4.6-1.el8.noarch +ovirt-release-host-node-4.4.6.1-1.el8.noarch +ovirt-release44-4.4.6.1-1.el8.noarch @@ -654,2 +655,2 @@ -ovn2.11-2.11.1-39.el8.x86_64 -ovn2.11-host-2.11.1-39.el8.x86_64 +ovn2.11-2.11.1-57.el8s.x86_64 +ovn2.11-host-2.11.1-57.el8s.x86_64 @@ -788 +789 @@ -python3-openvswitch2.11-2.11.0-50.el8.x86_64 +python3-openvswitch2.11-2.11.3-87.el8s.x86_64 @@ -828 +829 @@ -python3-subscription-manager-rhsm-1.28.13-2.el8.x86_64 +python3-subscription-manager-rhsm-1.28.16-1.el8.x86_64 @@ -830 +831 @@ -python3-syspurpose-1.28.13-2.el8.x86_64 +python3-syspurpose-1.28.16-1.el8.x86_64 @@ -836,12 +837,14 @@ -qemu-guest-agent-5.2.0-11.el8s.x86_64 -qemu-img-5.2.0-11.el8s.x86_64 -qemu-kvm-5.2.0-11.el8s.x86_64 -qemu-kvm-block-curl-5.2.0-11.el8s.x86_64 -qemu-kvm-block-gluster-5.2.0-11.el8s.x86_64 -qemu-kvm-block-iscsi-5.2.0-11.el8s.x86_64 -qemu-kvm-block-rbd-5.2.0-11.el8s.x86_64 -qemu-kvm-block-ssh-5.2.0-11.el8s.x86_64 -qemu-kvm-common-5.2.0-11.el8s.x86_64 -qemu-kvm-core-5.2.0-11.el8s.x86_64 -quota-4.04-13.el8.x86_64 -quota-nls-4.04-13.el8.noarch +qemu-guest-agent-5.2.0-16.el8s.x86_64 +qemu-img-5.2.0-16.el8s.x86_64 +qemu-kvm-5.2.0-16.el8s.x86_64 +qemu-kvm-block-curl-5.2.0-16.el8s.x86_64 +qemu-kvm-block-gluster-5.2.0-16.el8s.x86_64 +qemu-kvm-block-iscsi-5.2.0-16.el8s.x86_64 +qemu-kvm-block-rbd-5.2.0-16.el8s.x86_64 +qemu-kvm-block-ssh-5.2.0-16.el8s.x86_64 +qemu-kvm-common-5.2.0-16.el8s.x86_64 +qemu-kvm-core-5.2.0-16.el8s.x86_64 +qemu-kvm-ui-opengl-5.2.0-16.el8s.x86_64 +qemu-kvm-ui-spice-5.2.0-16.el8s.x86_64 +quota-4.04-14.el8.x86_64 +quota-nls-4.04-14.el8.noarch @@ -889 +892 @@ -sos-4.0-11.el8.noarch +sos-4.1-1.el8.noarch @@ -903 +906 @@ -subscription-manager-rhsm-certificates-1.28.13-2.el8.x86_64 +subscription-manager-rhsm-certificates-1.28.16-1.el8.x86_64 @@ -960,0 +964 @@ +xkeyboard-config-2.28-1.el8.noarch @@ -963,0 +968,2 @@ +xmlsec1-1.2.25-4.el8.x86_64 +xmlsec1-openssl-1.2.25-4.el8.x86_64 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: Changing the ovirtmgmt IP address
by Matthew.Stier＠fujitsu.com 10 May '21

10 May '21

Found the answer: Update /var/lib/vdsm/persistence/netconf/nets/ovritmgmt and reboot. From: Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> Sent: Monday, May 10, 2021 3:18 PM To: users(a)ovirt.org Subject: [ovirt-users] Changing the ovirtmgmt IP address Version: 4.3.10 I'm attempting to change the IP address, netmask and gateway of the ovirtmgmt NIC of a host, but everytime I reboot the host, the old address/netmask/gateway re-assert themselves. Where do I need to make the changes, so they will be permanent? I've modified /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt and route-ovirtmgmt, but they don't stick through a reboot.

1 0

Changing the ovirtmgmt IP address
by Matthew.Stier＠fujitsu.com 10 May '21

10 May '21

Version: 4.3.10 I'm attempting to change the IP address, netmask and gateway of the ovirtmgmt NIC of a host, but everytime I reboot the host, the old address/netmask/gateway re-assert themselves. Where do I need to make the changes, so they will be permanent? I've modified /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt and route-ovirtmgmt, but they don't stick through a reboot.

1 0

Re: Unable to provide network acces to second cluster in DC
by Miguel Garcia 10 May '21

10 May '21

Indeed, the network is assigned to the cluster however is not listed at all.

1 0

Changing 1 node Gluster Distributed to replica
by Ernest Clyde Chua 10 May '21

10 May '21

Good day, currently we have a 1 node host that also runs a gluster in 1 node distributed mode and we recently decided to upgrade to a 3 node host which also runs gluster and set a replica count of 3. can someone help me how can i safely change the volume type to replicated

2 5

Sharding decision for oVirt
by levin＠mydream.com.hk 10 May '21

10 May '21

Description of problem: Intermittent VM pause and Qcow image corruption after add new bricks. I'm suffered an issue on image corruption on oVirt 4.3 caused by default gluster ovirt profile, and intermittent VM pause. the problem is similar to #2246 #2254 in glusterfs issue and VM pause issue report in ovirt user group. The gluster vol did not have pending heal object, vol appear in good shape, xfs is healthy, no hardware issue. Sadly few VM have mystery corruption after new bricks added. Afterwards, I try to simulate the problem with or without "cluster.lookup-optimize off" few time, but the problem is not 100% reproducible with lookup-optimize on, I got 1 of 3 attempt that able to reproduce it. It really depend on the workloads and cache status at that moment and the number of object after rebalance as well. Also I tried to disable all sharding features, it ran very solid, write performance increase by far, no corruption, no VM pause when the gluster under stress. So, here is a decision question on shard or not shard. IMO, even recommendation document saying it break large file into smaller chunk that allow healing to complete faster, a larger file can spread over multiple bricks. But there are uncovered issue compared to full large file in this case, I'd like to further deep dive into the reason why recommend shard as default for oVirt? Especially from the reliability and performance perspective, sharding seems losing this end for ovirt/kvm workloads. Is it more appropriate to just tell ovirt user to ensure underlying single bricks shall be large enough to hold the largest chunk instead? Besides, anything i'm overlooked for the shard setting? I'm really doubt to enable sharding on the volume after disaster.

4 12

Re: oVirt deploy new HE Host problem
by Marko Vrgotic 10 May '21

10 May '21

Hi Yedidyah and Strahil, Just to double check, if you received the issue request and log files. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Thursday, 6 May 2021 at 11:43 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: oVirt deploy new HE Host problem It might come handy, here is the complete hosted-engine.conf file [root@ovirt-sj-03 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem fqdn=ovirt-engine.ictv.com vm_disk_id=b019c5fa-8fb5-4bfc-8339-f5b7f590a051 sdUUID=054c43fc-1924-4106-9f80-0f2ac62b9886 console=vnc vmid=66b6d489-ceb8-486a-951a-355e21f13627 conf=/var/run/ovirt-hosted-engine-ha/vm.conf iqn= conf_image_UUID=910f445e-31c0-4441-9c82-720901f7f19b port= network_test=dns vm_disk_vol_id=f1ce8ba6-2d3b-4309-bca0-e6a00ce74c75 storage=10.210.13.64:/hosted_engine gateway=10.210.11.254 ca_subject="C=EN, L=Test, O=Test, CN=Test" ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem connectionUUID=e29cf818-5ee5-46e1-85c1-8aeefa33e95d nfs_version=auto bridge=ovirtmgmt metadata_image_UUID=16b3e5ac-e70b-46e3-bf81-322954fe0b44 mnt_options= domainType=nfs password= vdsm_use_ssl=true tcp_t_port= user= host_id=3 metadata_volume_UUID=b6326e48-a7d2-4cba-af91-441db9f353c2 spUUID=00000000-0000-0000-0000-000000000000 conf_volume_UUID=c518f937-60fe-4fed-a54c-db11328bb507 portal= lockspace_image_UUID=e08188be-f733-4d5c-9222-a4b4e2228955 lockspace_volume_UUID=081f81c5-b2b2-46d5-9f82-9d9041ccc108 tcp_t_address= ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Thursday, 6 May 2021 at 11:20 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: oVirt deploy new HE Host problem Hi Strahil and Yedidyah, As agreed, short summary: Deploy new HE host fails Pre Deploy state: * Host1 and Host3 are current HE HA pool * Host1 and Host3 are unaware of Host2 (check below) * I am trying to add Host2 to HE HA pool * Host2 is fully reinstalled – clean OS * Host2 is added to oVirt as regular Host * Host2 is currently in Maintenance mode (waiting for Reinstall with HE Deploy) [root@ovirt-sj-03 ~]# hosted-engine --vm-status --== Host ovirt-sj-01.ictv.com (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : d15bb877 local_conf_timestamp : 3103909 Host timestamp : 3103909 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3103909 (Thu May 6 01:42:06 2021) host-id=1 score=3400 vm_conf_refresh_time=3103909 (Thu May 6 01:42:06 2021) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host ovirt-sj-03.ictv.com (id: 3) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 Engine status : {"health": "good", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : 15801717 local_conf_timestamp : 3106395 Host timestamp : 3106395 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3106395 (Thu May 6 01:42:13 2021) host-id=3 score=3400 vm_conf_refresh_time=3106395 (Thu May 6 01:42:13 2021) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False Deployment: I have attached logs from Host2 and Engine – if anything is missing, please let me know. Kindly awaiting your reply. You might notice slight time change on Host2 logs – that’s after re-provisioning I did not set correct timezone ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

Importing VM fails with "No space left on device"
by j.velasco＠outlook.com 09 May '21

09 May '21

Hello List, I am facing the following issue when I try to import a VM from a KVM host to my oVirt (4.4.5.11-1.el8). The importing I done throuth GUI using the option of KVM provider. -- Log1: # cat /var/log/vdsm/import-57f84423-56cb-4187-86e2-f4208348e1f5-20210507T124121.log [ 0.0] preparing for copy [ 0.0] Copying disk 1/1 to /rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/cb63ffc9-07ee-4323-9e8a-378be31ae3f7/e7e69cbc-47bf-4557-ae02-ca1c53c8423f Traceback (most recent call last): File "/usr/libexec/vdsm/kvm2ovirt", line 23, in <module> kvm2ovirt.main() File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 277, in main handle_volume(con, diskno, src, dst, options) File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 228, in handle_volume download_disk(sr, estimated_size, None, dst, options.bufsize) File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 169, in download_disk op.run() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 57, in run res = self._run() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 163, in _run self._write_chunk(count) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 188, in _write_chunk n = self._dst.write(v) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/file.py", line 88, in write return util.uninterruptible(self._fio.write, buf) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/util.py", line 20, in uninterruptible return func(*args) OSError: [Errno 28] No space left on device -- Log2: # cat /var/log/vdsm/vdsm.log 2021-05-07 10:29:49,813-0500 DEBUG (v2v/57f84423) [root] START thread <Thread(v2v/57f84423, started daemon 140273162123008)> (func=<bound method ImportVm._run of <vdsm.v2v.ImportVm object at 0x7f946051c5c0>>, args=(), kwargs={}) (concurrent:258) 2021-05-07 10:29:49,813-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' starting import (v2v:880) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') moving from state preparing -> state preparing (task:624) 2021-05-07 10:29:49,814-0500 INFO (v2v/57f84423) [vdsm.api] START prepareImage(sdUUID='cc9fae8e-b714-44cf-9dac-3a83a15b0455', spUUID='24d9d2fa-98f9-11eb-aea7-00163e09cc71', imgUUID='226cc137-1992-4246-9484-80a1bfb5e9f7', leafUUID='847bc460-1b54-4756-8ced-4b969c399900', allowIllegal=False) from=internal, task_id=58a7bdc0-0f7e-4307-92ba-040f1a272721 (api:48) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Trying to register resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' for lock type 'shared' (resourceManager:474) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' is free. Now locking as 'shared' (1 active user) (resourceManager:531) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.ResourceManager.Request] (ResName='00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455', ReqID='b2ebef3c-8b3d-4429-b9da-b6f3af2c9ac4') Granted request (resourceManager:221) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') _resourcesAcquired: 00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455 (shared) (task:856) 2021-05-07 10:29:49,814-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') ref 1 aborting False (task:1008) 2021-05-07 10:29:49,815-0500 DEBUG (v2v/57f84423) [common.commands] /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /sbin/lvm lvs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360060e8007dfc8000030dfc80000113f$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags cc9fae8e-b714-44cf-9dac-3a83a15b0455 (cwd None) (commands:153) 2021-05-07 10:29:49,900-0500 DEBUG (v2v/57f84423) [common.commands] SUCCESS: <err> = b''; <rc> = 0 (commands:185) 2021-05-07 10:29:49,902-0500 DEBUG (v2v/57f84423) [storage.LVM] lvs reloaded (lvm:759) 2021-05-07 10:29:49,904-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-23 /usr/bin/dd iflag=direct skip=2096 bs=512 if=/dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/metadata count=1 (cwd None) (commands:211) 2021-05-07 10:29:49,916-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] SUCCESS: <err> = b'1+0 records in\n1+0 records out\n512 bytes copied, 0.000228452 s, 2.2 MB/s\n'; <rc> = 0 (commands:224) 2021-05-07 10:29:49,916-0500 DEBUG (v2v/57f84423) [storage.Misc] err: [b'1+0 records in', b'1+0 records out', b'512 bytes copied, 0.000228452 s, 2.2 MB/s'], size: 512 (misc:114) 2021-05-07 10:29:49,917-0500 INFO (v2v/57f84423) [storage.LVM] Activating lvs: vg=cc9fae8e-b714-44cf-9dac-3a83a15b0455 lvs=['847bc460-1b54-4756-8ced-4b969c399900'] (lvm:1738) 2021-05-07 10:29:49,917-0500 DEBUG (v2v/57f84423) [common.commands] /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /sbin/lvm lvchange --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360060e8007dfc8000030dfc80000113f$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --autobackup n --available y cc9fae8e-b714-44cf-9dac-3a83a15b0455/847bc460-1b54-4756-8ced-4b969c399900 (cwd None) (commands:153) 2021-05-07 10:29:50,034-0500 DEBUG (v2v/57f84423) [common.commands] SUCCESS: <err> = b''; <rc> = 0 (commands:185) 2021-05-07 10:29:50,035-0500 INFO (v2v/57f84423) [storage.StorageDomain] Creating image run directory '/run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7' (blockSD:1362) 2021-05-07 10:29:50,035-0500 INFO (v2v/57f84423) [storage.fileUtils] Creating directory: /run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7 mode: None (fileUtils:201) 2021-05-07 10:29:50,036-0500 INFO (v2v/57f84423) [storage.StorageDomain] Creating symlink from /dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/847bc460-1b54-4756-8ced-4b969c399900 to /run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900 (blockSD:1367) 2021-05-07 10:29:50,037-0500 DEBUG (v2v/57f84423) [common.commands] /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /sbin/lvm lvs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360060e8007dfc8000030dfc80000113f$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags cc9fae8e-b714-44cf-9dac-3a83a15b0455 (cwd None) (commands:153) 2021-05-07 10:29:50,119-0500 DEBUG (v2v/57f84423) [common.commands] SUCCESS: <err> = b''; <rc> = 0 (commands:185) 2021-05-07 10:29:50,121-0500 DEBUG (v2v/57f84423) [storage.LVM] lvs reloaded (lvm:759) 2021-05-07 10:29:50,122-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-23 /usr/bin/dd iflag=direct skip=2096 bs=512 if=/dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/metadata count=1 (cwd None) (commands:211) 2021-05-07 10:29:50,135-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] SUCCESS: <err> = b'1+0 records in\n1+0 records out\n512 bytes copied, 0.000212608 s, 2.4 MB/s\n'; <rc> = 0 (commands:224) 2021-05-07 10:29:50,135-0500 DEBUG (v2v/57f84423) [storage.Misc] err: [b'1+0 records in', b'1+0 records out', b'512 bytes copied, 0.000212608 s, 2.4 MB/s'], size: 512 (misc:114) 2021-05-07 10:29:50,136-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-23 /usr/bin/dd iflag=direct skip=2096 bs=512 if=/dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/metadata count=1 (cwd None) (commands:211) 2021-05-07 10:29:50,143-0500 DEBUG (v2v/57f84423) [storage.Misc.excCmd] SUCCESS: <err> = b'1+0 records in\n1+0 records out\n512 bytes copied, 0.000248951 s, 2.1 MB/s\n'; <rc> = 0 (commands:224) 2021-05-07 10:29:50,143-0500 DEBUG (v2v/57f84423) [storage.Misc] err: [b'1+0 records in', b'1+0 records out', b'512 bytes copied, 0.000248951 s, 2.1 MB/s'], size: 512 (misc:114) 2021-05-07 10:29:50,143-0500 DEBUG (v2v/57f84423) [root] /usr/bin/taskset --cpu-list 0-23 /usr/bin/qemu-img info --output json -U /rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900 (cwd None) (commands:211) 2021-05-07 10:29:50,161-0500 DEBUG (v2v/57f84423) [root] SUCCESS: <err> = b''; <rc> = 0 (commands:224) 2021-05-07 10:29:50,162-0500 INFO (v2v/57f84423) [storage.StorageDomain] Creating symlink from /run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7 to /rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7 (blockSD:1332) 2021-05-07 10:29:50,162-0500 DEBUG (v2v/57f84423) [storage.StorageDomain] path to image directory already exists: /rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7 (blockSD:1338) 2021-05-07 10:29:50,163-0500 INFO (v2v/57f84423) [vdsm.api] FINISH prepareImage return={'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900', 'info': {'type': 'block', 'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900'}, 'imgVolumesInfo': [{'domainID': 'cc9fae8e-b714-44cf-9dac-3a83a15b0455', 'imageID': '226cc137-1992-4246-9484-80a1bfb5e9f7', 'volumeID': '847bc460-1b54-4756-8ced-4b969c399900', 'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900', 'leasePath': '/dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/leases', 'leaseOffset': 108003328}]} from=internal, task_id=58a7bdc0-0f7e-4307-92ba-040f1a272721 (api:54) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') finished: {'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900', 'info': {'type': 'block', 'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900'}, 'imgVolumesInfo': [{'domainID': 'cc9fae8e-b714-44cf-9dac-3a83a15b0455', 'imageID': '226cc137-1992-4246-9484-80a1bfb5e9f7', 'volumeID': '847bc460-1b54-4756-8ced-4b969c399900', 'path': '/rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900', 'leasePath': '/dev/cc9fae8e-b714-44cf-9dac-3a83a15b0455/leases', 'leaseOffset': 108003328}]} (task:1210) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') moving from state finished -> state finished (task:624) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.ResourceManager.Owner] Owner.releaseAll resources %s (resourceManager:742) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Trying to release resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' (resourceManager:546) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Released resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' (0 active users) (resourceManager:564) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' is free, finding out if anyone is waiting for it. (resourceManager:570) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] No one is waiting for resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455', Clearing records. (resourceManager:578) 2021-05-07 10:29:50,163-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='58a7bdc0-0f7e-4307-92ba-040f1a272721') ref 0 aborting False (task:1008) 2021-05-07 10:29:50,164-0500 INFO (v2v/57f84423) [root] Storing import log at: '/var/log/vdsm/import/import-57f84423-56cb-4187-86e2-f4208348e1f5-20210507T102950.log' (v2v:436) 2021-05-07 10:29:50,170-0500 DEBUG (v2v/57f84423) [root] /usr/bin/taskset --cpu-list 0-23 /usr/bin/nice -n 19 /usr/bin/ionice -c 3 /usr/libexec/vdsm/kvm2ovirt --uri qemu+tcp://root@172.16.0.61/system --bufsize 1048576 --source /var/lib/libvirt/images/vm_powervp-si.qcow2 --dest /rhev/data-center/mnt/blockSD/cc9fae8e-b714-44cf-9dac-3a83a15b0455/images/226cc137-1992-4246-9484-80a1bfb5e9f7/847bc460-1b54-4756-8ced-4b969c399900 --storage-type volume --vm-name vm_powervp-si --allocation sparse (cwd None) (v2v:1511) 2021-05-07 10:29:50,175-0500 DEBUG (v2v/57f84423) [root] /usr/bin/taskset --cpu-list 0-23 /usr/bin/nice -n 19 /usr/bin/ionice -c 3 tee /var/log/vdsm/import/import-57f84423-56cb-4187-86e2-f4208348e1f5-20210507T102950.log (cwd None) (v2v:1511) 2021-05-07 10:29:50,274-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copying disk 1/1 (v2v:912) 2021-05-07 10:29:50,277-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 0/100 (v2v:921) 2021-05-07 10:29:51,277-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 0/100 (v2v:921) 2021-05-07 10:29:52,277-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 0/100 (v2v:921) 2021-05-07 10:30:14,283-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 10/100 (v2v:921) 2021-05-07 10:30:15,283-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 10/100 (v2v:921) 2021-05-07 10:30:16,283-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 10/100 (v2v:921) 2021-05-07 10:30:39,288-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 20/100 (v2v:921) 2021-05-07 10:30:40,288-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 20/100 (v2v:921) 2021-05-07 10:30:46,281-0500 INFO (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' copy disk 1 progress 100/100 (v2v:921) 2021-05-07 10:30:46,407-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') moving from state preparing -> state preparing (task:624) 2021-05-07 10:30:46,407-0500 INFO (v2v/57f84423) [vdsm.api] START teardownImage(sdUUID='cc9fae8e-b714-44cf-9dac-3a83a15b0455', spUUID='24d9d2fa-98f9-11eb-aea7-00163e09cc71', imgUUID='226cc137-1992-4246-9484-80a1bfb5e9f7', volUUID=None) from=internal, task_id=ecbcd983-2f33-45a4-b962-7fc4b9342822 (api:48) 2021-05-07 10:30:46,407-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Trying to register resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' for lock type 'shared' (resourceManager:474) 2021-05-07 10:30:46,407-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' is free. Now locking as 'shared' (1 active user) (resourceManager:531) 2021-05-07 10:30:46,407-0500 DEBUG (v2v/57f84423) [storage.ResourceManager.Request] (ResName='00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455', ReqID='1744a1a5-d543-4528-be79-c752bce08263') Granted request (resourceManager:221) 2021-05-07 10:30:46,408-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') _resourcesAcquired: 00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455 (shared) (task:856) 2021-05-07 10:30:46,408-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') ref 1 aborting False (task:1008) 2021-05-07 10:30:46,408-0500 INFO (v2v/57f84423) [storage.StorageDomain] Removing image run directory '/run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7' (blockSD:1386) 2021-05-07 10:30:46,408-0500 INFO (v2v/57f84423) [storage.fileUtils] Removing directory: /run/vdsm/storage/cc9fae8e-b714-44cf-9dac-3a83a15b0455/226cc137-1992-4246-9484-80a1bfb5e9f7 (fileUtils:182) 2021-05-07 10:30:46,408-0500 DEBUG (v2v/57f84423) [common.commands] /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /sbin/lvm lvs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360060e8007dfc8000030dfc80000113f$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags cc9fae8e-b714-44cf-9dac-3a83a15b0455 (cwd None) (commands:153) 2021-05-07 10:30:46,510-0500 DEBUG (v2v/57f84423) [common.commands] SUCCESS: <err> = b''; <rc> = 0 (commands:185) 2021-05-07 10:30:46,511-0500 DEBUG (v2v/57f84423) [storage.LVM] lvs reloaded (lvm:759) 2021-05-07 10:30:46,512-0500 INFO (v2v/57f84423) [storage.LVM] Deactivating lvs: vg=cc9fae8e-b714-44cf-9dac-3a83a15b0455 lvs=['847bc460-1b54-4756-8ced-4b969c399900'] (lvm:1746) 2021-05-07 10:30:46,512-0500 DEBUG (v2v/57f84423) [common.commands] /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /sbin/lvm lvchange --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360060e8007dfc8000030dfc80000113f$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --autobackup n --available n cc9fae8e-b714-44cf-9dac-3a83a15b0455/847bc460-1b54-4756-8ced-4b969c399900 (cwd None) (commands:153) 2021-05-07 10:30:46,629-0500 DEBUG (v2v/57f84423) [common.commands] SUCCESS: <err> = b''; <rc> = 0 (commands:185) 2021-05-07 10:30:46,630-0500 INFO (v2v/57f84423) [vdsm.api] FINISH teardownImage return=None from=internal, task_id=ecbcd983-2f33-45a4-b962-7fc4b9342822 (api:54) 2021-05-07 10:30:46,630-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') finished: None (task:1210) 2021-05-07 10:30:46,630-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') moving from state finished -> state finished (task:624) 2021-05-07 10:30:46,630-0500 DEBUG (v2v/57f84423) [storage.ResourceManager.Owner] Owner.releaseAll resources %s (resourceManager:742) 2021-05-07 10:30:46,630-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Trying to release resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' (resourceManager:546) 2021-05-07 10:30:46,631-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Released resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' (0 active users) (resourceManager:564) 2021-05-07 10:30:46,631-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] Resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455' is free, finding out if anyone is waiting for it. (resourceManager:570) 2021-05-07 10:30:46,631-0500 DEBUG (v2v/57f84423) [storage.ResourceManager] No one is waiting for resource '00_storage.cc9fae8e-b714-44cf-9dac-3a83a15b0455', Clearing records. (resourceManager:578) 2021-05-07 10:30:46,631-0500 DEBUG (v2v/57f84423) [storage.TaskManager.Task] (Task='ecbcd983-2f33-45a4-b962-7fc4b9342822') ref 0 aborting False (task:1008) 2021-05-07 10:30:46,631-0500 ERROR (v2v/57f84423) [root] Job '57f84423-56cb-4187-86e2-f4208348e1f5' failed (v2v:869) 2021-05-07 10:30:46,635-0500 DEBUG (v2v/57f84423) [root] FINISH thread <Thread(v2v/57f84423, stopped daemon 140273162123008)> (concurrent:261) -- Details of the enviroment: # df -Ph Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev tmpfs 32G 4.0K 32G 1% /dev/shm tmpfs 32G 26M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0+1 584G 11G 573G 2% / /dev/mapper/onn-home 1014M 40M 975M 4% /home /dev/mapper/onn-tmp 1014M 40M 975M 4% /tmp /dev/sda2 1014M 479M 536M 48% /boot /dev/mapper/onn-var 30G 3.2G 27G 11% /var /dev/sda1 599M 6.9M 592M 2% /boot/efi /dev/mapper/onn-var_log 8.0G 498M 7.6G 7% /var/log /dev/mapper/onn-var_crash 10G 105M 9.9G 2% /var/crash /dev/mapper/onn-var_log_audit 2.0G 84M 2.0G 5% /var/log/audit tmpfs 6.3G 0 6.3G 0% /run/user/0 /dev/mapper/da3e3aff--0bfc--42cd--944f--f6145c50134a-master 976M 1.3M 924M 1% /rhev/data-center/mnt/blockSD/da3e3aff-0bfc-42cd-944f-f6145c50134a/master /dev/mapper/onn-lv_iso 12G 11G 1.6G 88% /rhev/data-center/mnt/_dev_mapper_onn-lv__iso 172.19.1.80:/exportdomain 584G 11G 573G 2% /rhev/data-center/mnt/172.19.1.80:_exportdomain * Inodes available = 99%. # qemu-img info /var/lib/libvirt/images/vm_powervp-si.qcow2 image: /var/lib/libvirt/images/vm_powervp-si.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 4.2G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true

2 1

hosted-engine /var full
by Pascal D 09 May '21

09 May '21

I have an issue with my 4.3 hosted-engine. The /var is full. The directory taking 20G is /var/opt/rh/rh-postgresql10/lib/pgsql/data/base/data/16398 some of those files have size over 1GB. Can they be safely removed? TIA

3 7

No space left on /var/tmp
by Pietro Pesce 09 May '21

09 May '21

Hello from the documents i see "Appliance requires at least 5GB of free space for /var/tmp as mentioned in the documentation." so , /var have 20G free, why during the task of extraction of appliance the /var was fulled up? and the installation fail. # rpm -qa|grep ovirt-engine-appliance ovirt-engine-appliance-4.3-20200603.1.0.2.el7.x86_64 # rpm -qa|grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.3.12-1.0.1.el7.noarch thanks a lot PP

1 0

Fail if MAC address structure is incorrect
by Pietro Pesce 09 May '21

09 May '21

Hello i try to deploy self hosted engine, but recive this error: [ INFO ] TASK [ovirt.hosted_engine_setup : Fail if MAC address structure is incorrect] [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'not he_vm_mac_addr | regex_search( \"^[a-fA-F0-9][02468aAcCeE](:[a-fA-F0-9]{2}){5}$\" )' failed. The error was: Unexpected templating type error occurred on ({% if not he_vm_mac_addr | regex_search( \"^[a-fA-F0-9][02468aAcCeE](:[a-fA-F0-9]{2}){5}$\" ) %} True {% else %} False {% endif %}): expected string or buffer\n\nThe error appears to be in '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/pre_checks/validate_mac_address.yml': line 14, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n register: he_vm_mac_addr\n - name: Fail if MAC address structure is incorrect\n ^ here\n"} on the MAC field there is 00:16:3e:62:be:0e # rpm -qa|grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.3.13-1.0.1.el7.noarch # rpm -qa|grep ovirt-engine-appliance ovirt-engine-appliance-4.3-20200603.1.0.2.el7.x86_64 # rpm -qa|grep -i oracle-ovirt-release-el7 oracle-ovirt-release-el7-1.0-3.el7.x86_64 # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.9 (Maipo) Best Regards

1 0

Glusterfs and vm's
by eevans＠digitaldatatechs.com 08 May '21

08 May '21

I have researched and applied several tweaks to Gluster to improve performance but on the vm side, depending on distribution, you can "tweak" the vm for better performance on Gluster as well; yum install tuned.noarch tuned-utils-systemtap.noarch tuned-utils.noarch tuned-gtk.noarch -y (GTK is for a gui if you have one.) tuned-adm profile virtual-guest Also, you can increase or decrease cache or swappiness on the host to fit your vm's, for workstations or servers. My cache is set to 2048 MB cache and swappiness of 10. Thanks for the help you have given me. Eric This helps RHEL and CentOS machines utilize glusterfs and actually speeds teh vm up. I hope this will help someone. If you want the URL for the article, just ask. Also, you can increase or decrease cache or swappiness to fit your vm's, for workstations or servers.

2 1

Cannot find a valid baseurl for repo: ovirt-4
by Timothy87 08 May '21

08 May '21

Any solution for this (Cannot find a valid baseurl for repo: ovirt-4).

1 0

oVirt virtual machines migration 3.6.3 to 4.4.5
by jean-marie.perron＠viseo.com 07 May '21

07 May '21

Hello, We have a project to migrate from an old oVirt infrastructure (3.6.3) to a new one (4.4.5). The new infrastructure is already in place and we want to migrate the virtual machines. We tried to import the virtual machines via a domain with an NFS mount from the old oVirt (import pre-configured domain). With this domain we only see VM disks and not full virtual machines for import. Is there a way to transfer the virtual machines from the old oVirt to the new one by an NFS mount then import into oVirt Engine or a "copy/paste" virtual machine files then import. Thanks a lot, JMP

1 0

oVirt 2021 Spring survey questions
by Sandro Bonazzola 07 May '21

07 May '21

Hi, it's about the usual time of the year when we ask the community to provide feedback with a survey. Any questions you'd like to be asked? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

9 16

[ANN] Async release for oVirt 4.4.6
by Sandro Bonazzola 07 May '21

07 May '21

on May 7th 2021 the oVirt project released an async update to the following packages: - ovirt-engine-extension-aaa-ldap-1.4.3 - ovirt-engine-4.4.6.7 Fixing the following bugs: - Bug 1896779 <https://bugzilla.redhat.com/show_bug.cgi?id=1896779> - Unable to access LDAP server using IPv6 when both engine and LDAP are configured with IPv6 interfaces only, but DNS records for LDAP contains also IPv4 address - Bug 1940138 <https://bugzilla.redhat.com/show_bug.cgi?id=1940138> - AAA - IPV6 detection needs to be disabled for LDAP hosts to be recognized - Bug 1941541 <https://bugzilla.redhat.com/show_bug.cgi?id=1941541> - ldap setup fails to login with connection error when use dns is specified - Bug 1950209 <https://bugzilla.redhat.com/show_bug.cgi?id=1950209> - Leaf images used by the VM is deleted by the engine during snapshot merge - Bug 1957595 <https://bugzilla.redhat.com/show_bug.cgi?id=1957595> - Failed to import VM from KVM - Bug 1956677 <https://bugzilla.redhat.com/show_bug.cgi?id=1956677> - Snapshot merge fails with null pointer exception. We are planning an oVirt Node async once libvirt fix for Bug 1953577 <https://bugzilla.redhat.com/show_bug.cgi?id=1953577> will be available from CentOS Virtualization SIG. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

libvirt crashes / update missing
by Rik Theys 07 May '21

07 May '21

Hi, On our hosts, we periodically see libvirt crashes as described in https://access.redhat.com/solutions/5694061. https://access.redhat.com/errata/RHSA-2021:1125 (issued April 7) is the advisory that contains a fix for this issue. This bug should be fixed in libvirt 6.6.0-13.2 in the "Advanced Virtualization" repo. When I look at the ovirt-4.4-advanced-virtualization repo on my hosts, it does not (yet) contain this update. Are updates from the Advanced Virtualization repo not automatically rebuild for CentOS 8.3? On a CentOS Stream 8 machine, libvirt is version libvirt-6.0.0-35.module_el8.5.0+746+bbd5d70c.src.rpm. Installing centos-release-advanced-virtualization adds the Advanced Virtualization repo file, which then also brings in the 6.6.0-13 version. When I look at a CentOS 8 stream mirror, for example http://mirror.kinamo.be/centos/8-stream/virt/x86_64/, I do see the virt repos with a newer libvirt, but this is not the repo the centos-release-advanced-virtualization package is using? What is the recommended setup to use to have this bug fix available? Regards, Rik -- Rik Theys System Engineer KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee +32(0)16/32.11.07 ---------------------------------------------------------------- <<Any errors in spelling, tact or fact are transmission errors>>

1 0

oVirt 4.4.6 is now generally available
by Lev Veyde 07 May '21

07 May '21

The oVirt project is excited to announce the general availability of oVirt 4.4.6 , as of May 4th, 2021. This release unleashes an altogether more powerful and flexible open source virtualization solution that encompasses hundreds of individual changes and a wide range of enhancements across the engine, storage, network, user interface, and analytics, as compared to oVirt 4.3. Important notes before you install / upgrade Please note that oVirt 4.4 only supports clusters and data centers with compatibility version 4.2 and above. If clusters or data centers are running with an older compatibility version, you need to upgrade them to at least 4.2 (4.3 is recommended). Please note that in RHEL 8 / CentOS 8 several devices that worked on EL7 are no longer supported. For example, the megaraid_sas driver is removed. If you use Enterprise Linux 8 hosts you can try to provide the necessary drivers for the deprecated hardware using the DUD method (See the users’ mailing list thread on this at https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NDSVUZSESOXEFJ… ) Rebase on CentOS Stream Starting with oVirt 4.4.6 both oVirt Node and oVirt Engine Appliance are based on CentOS Stream. You can still install oVirt 4.4.6 on Red Hat Enterprise Linux 8.3, CentOS Linux 8.3 or equivalent but in order to use cluster level 4.6 you’ll have to wait for 8.4 to be available. Please note that existing oVirt Nodes updating to 4.4.6 will automatically be based on CentOS Stream. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. What’s new in oVirt 4.4.6 Release? This update is the sixth in a series of stabilization updates to the 4.4 series. This release is available now on x86_64 architecture for: - Red Hat Enterprise Linux 8.3 - CentOS Linux (or similar) 8.3 - CentOS Stream 8 This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: - Red Hat Enterprise Linux 8.3 - CentOS Linux (or similar) 8.3 - oVirt Node NG (based on CentOS Stream 8) - CentOS Stream oVirt Node and Appliance have been updated, including: - oVirt 4.4.6: https://www.ovirt.org/release/4.4.6/ - CentOS Stream 8 - Ansible 2.9.20: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.… - Advanced Virtualization 8.4 See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS Stream 8 - oVirt Node NG is already available for CentOS Stream 8 Additional resources: - Read more about the oVirt 4.4.6 release highlights: https://www.ovirt.org/release/4.4.6/ - Get more oVirt project updates on Twitter: https://twitter.com/ovirt - Check out the latest project news on the oVirt blog: https://blogs.ovirt.org/ [1] https://www.ovirt.org/release/4.4.6/ [2] https://resources.ovirt.org/pub/ovirt-4.4/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

3 2

host install failed when adding 4.4.6 node
by dhanaraj.ramesh＠yahoo.com 07 May '21

07 May '21

I installed new ovirt node with 4.4.6 6 and the same version ovirt engine is running as stand alone manager in cent os 8. When I try to add the node to engine getting Host installation failed and below is the log from engine. node has default route and able to access the cockpit 021-05-06 20:12:49,111-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Running command: InstallVdsCommand internal: false. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2021-05-06 20:12:49,195-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:12:49,222-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDS 2021-05-06 20:12:49,222-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Before Installation host 3e4d4299-0425-4c86-9c79-930c1f43f78b, renlovkvma01.ren.broadcom.net 2021-05-06 20:12:49,227-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37578a41 2021-05-06 20:12:49,234-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] FINISH, SetVdsStatusVDSCommand, return: , log id: 37578a41 2021-05-06 20:12:49,244-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Opening ssh-copy-id session on host renlovkvma01.ren.broadcom.net 2021-05-06 20:12:49,284-06 ERROR [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Can not run fence action on host 'renlovkvma01.ren.broadcom.net', no suitable proxy host was found. 2021-05-06 20:12:49,302-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_START_MIGHT_FAIL(9,010), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Start this host using Power-Management are expected to fail. 2021-05-06 20:12:49,304-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_STOP_MIGHT_FAIL(9,011), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Stop this host using Power-Management are expected to fail. 2021-05-06 20:12:49,309-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: KDUMP_DETECTION_NOT_CONFIGURED_ON_VDS(617), Kdump integration is enabled for host renlovkvma01.ren.broadcom.net, but kdump is not configured properly on host. 2021-05-06 20:12:49,323-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: USER_UPDATE_VDS(43), Host renlovkvma01.ren.broadcom.net configuration was updated by admin@internal-authz. 2021-05-06 20:12:49,577-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Executing ssh-copy-id command on host renlovkvma01.ren.broadcom.net 2021-05-06 20:12:50,424-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook execution has started on host renlovkvma01.ren.broadcom.net. 2021-05-06 20:12:54,018-06 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Host installation failed for host '3e4d4299-0425-4c86-9c79-930c1f43f78b', 'renlovkvma01.ren.broadcom.net': null 2021-05-06 20:12:54,019-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290371f3 2021-05-06 20:12:54,023-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] FINISH, SetVdsStatusVDSCommand, return: , log id: 290371f3 2021-05-06 20:12:54,032-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_INSTALL_FAILED(505), Host renlovkvma01.ren.broadcom.net installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. 2021-05-06 20:12:54,040-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Lock freed to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:14:45,462-06 INFO [org.ovirt.engine.core.bll.IsOvirtCockpitSSOStartedQuery] (default task-1) [792e24b7-7d44-460b-9204-63fa87db9b5e] The ovirt-cockpit-sso service is not running, pid file: '/var/run/ovirt-cockpit-sso/ovirt-cockpit-sso.pid'. 2021-05-06 20:16:02,350-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:16:02,358-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Running command: SyncNetworkProviderCommand internal: true. 2021-05-06 20:16:02,471-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-1) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:16:02,609-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Lock freed to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:18:14,084-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-6) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:18:14,134-06 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-6) [621ba285] Running command: CreateUserSessionCommand internal: false. 2021-05-06 20:18:14,144-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [621ba285] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '10.33.184.100' using session 'dYs+hblptTN7LSrrcmfTdFOz1eekPWKZiJXnyLJW2NOfRhZF4bDnFdMX736S0xTlxe76NmzOYDl+p5qS9WAVgA==' logged in. 2021-05-06 20:18:14,873-06 INFO [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-7) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt' -- 404 2021-05-06 20:18:22,413-06 INFO [org.ovirt.engine.core.bll.IsOvirtCockpitSSOStartedQuery] (default task-6) [af0d630e-a5f8-4626-a94c-945c5477e87b] The ovirt-cockpit-sso service is not running, pid file: '/var/run/ovirt-cockpit-sso/ovirt-cockpit-sso.pid'. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineScheduledThreadPool' is using 0 threads out of 1, 100 threads waiting for tasks. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineThreadMonitoringThreadPool' is using 1 threads out of 1, 0 threads waiting for tasks. 2021-05-06 20:19:33,626-06 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Backup check started. 2021-05-06 20:19:33,630-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), There is no full backup available, please run engine-backup to prevent data loss in case of corruption. 2021-05-06 20:19:33,630-06 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Backup check completed. 2021-05-06 20:19:33,661-06 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] Attempting to update VMs/Templates Ovf. 2021-05-06 20:20:53,590-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Running command: InstallVdsCommand internal: false. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2021-05-06 20:20:53,604-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:20:53,618-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDS 2021-05-06 20:20:53,618-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Before Installation host 3e4d4299-0425-4c86-9c79-930c1f43f78b, renlovkvma01.ren.broadcom.net 2021-05-06 20:20:53,619-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e0aa36a 2021-05-06 20:20:53,623-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] FINISH, SetVdsStatusVDSCommand, return: , log id: 6e0aa36a 2021-05-06 20:20:53,628-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Opening ssh-copy-id session on host renlovkvma01.ren.broadcom.net 2021-05-06 20:20:53,677-06 ERROR [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Can not run fence action on host 'renlovkvma01.ren.broadcom.net', no suitable proxy host was found. 2021-05-06 20:20:53,679-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_START_MIGHT_FAIL(9,010), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Start this host using Power-Management are expected to fail. 2021-05-06 20:20:53,681-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_STOP_MIGHT_FAIL(9,011), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Stop this host using Power-Management are expected to fail. 2021-05-06 20:20:53,689-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: KDUMP_DETECTION_NOT_CONFIGURED_ON_VDS(617), Kdump integration is enabled for host renlovkvma01.ren.broadcom.net, but kdump is not configured properly on host. 2021-05-06 20:20:53,695-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: USER_UPDATE_VDS(43), Host renlovkvma01.ren.broadcom.net configuration was updated by admin@internal-authz. 2021-05-06 20:20:53,833-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Executing ssh-copy-id command on host renlovkvma01.ren.broadcom.net 2021-05-06 20:20:54,553-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook execution has started on host renlovkvma01.ren.broadcom.net. 2021-05-06 20:20:57,584-06 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Host installation failed for host '3e4d4299-0425-4c86-9c79-930c1f43f78b', 'renlovkvma01.ren.broadcom.net': null 2021-05-06 20:20:57,585-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 640c7f84 2021-05-06 20:20:57,589-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] FINISH, SetVdsStatusVDSCommand, return: , log id: 640c7f84 2021-05-06 20:20:57,592-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_INSTALL_FAILED(505), Host renlovkvma01.ren.broadcom.net installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. 2021-05-06 20:20:57,595-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Lock freed to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:21:02,618-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Lock Acquired to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:21:02,625-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Running command: SyncNetworkProviderCommand internal: true. 2021-05-06 20:21:02,741-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-6) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:21:02,876-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Lock freed to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' [root@renlove01 ~]# ^C [root@renlove01 ~]# cd /var/log/ovirt-engine/host-deploy/ [root@renlove01 host-deploy]# ls [root@renlove01 host-deploy]#

1 0

Re: oVirt deploy new HE Host problem
by Marko Vrgotic 06 May '21

06 May '21

It might come handy, here is the complete hosted-engine.conf file [root@ovirt-sj-03 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem fqdn=ovirt-engine.ictv.com vm_disk_id=b019c5fa-8fb5-4bfc-8339-f5b7f590a051 sdUUID=054c43fc-1924-4106-9f80-0f2ac62b9886 console=vnc vmid=66b6d489-ceb8-486a-951a-355e21f13627 conf=/var/run/ovirt-hosted-engine-ha/vm.conf iqn= conf_image_UUID=910f445e-31c0-4441-9c82-720901f7f19b port= network_test=dns vm_disk_vol_id=f1ce8ba6-2d3b-4309-bca0-e6a00ce74c75 storage=10.210.13.64:/hosted_engine gateway=10.210.11.254 ca_subject="C=EN, L=Test, O=Test, CN=Test" ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem connectionUUID=e29cf818-5ee5-46e1-85c1-8aeefa33e95d nfs_version=auto bridge=ovirtmgmt metadata_image_UUID=16b3e5ac-e70b-46e3-bf81-322954fe0b44 mnt_options= domainType=nfs password= vdsm_use_ssl=true tcp_t_port= user= host_id=3 metadata_volume_UUID=b6326e48-a7d2-4cba-af91-441db9f353c2 spUUID=00000000-0000-0000-0000-000000000000 conf_volume_UUID=c518f937-60fe-4fed-a54c-db11328bb507 portal= lockspace_image_UUID=e08188be-f733-4d5c-9222-a4b4e2228955 lockspace_volume_UUID=081f81c5-b2b2-46d5-9f82-9d9041ccc108 tcp_t_address= ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Thursday, 6 May 2021 at 11:20 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: oVirt deploy new HE Host problem Hi Strahil and Yedidyah, As agreed, short summary: Deploy new HE host fails Pre Deploy state: * Host1 and Host3 are current HE HA pool * Host1 and Host3 are unaware of Host2 (check below) * I am trying to add Host2 to HE HA pool * Host2 is fully reinstalled – clean OS * Host2 is added to oVirt as regular Host * Host2 is currently in Maintenance mode (waiting for Reinstall with HE Deploy) [root@ovirt-sj-03 ~]# hosted-engine --vm-status --== Host ovirt-sj-01.ictv.com (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : d15bb877 local_conf_timestamp : 3103909 Host timestamp : 3103909 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3103909 (Thu May 6 01:42:06 2021) host-id=1 score=3400 vm_conf_refresh_time=3103909 (Thu May 6 01:42:06 2021) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host ovirt-sj-03.ictv.com (id: 3) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 Engine status : {"health": "good", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : 15801717 local_conf_timestamp : 3106395 Host timestamp : 3106395 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3106395 (Thu May 6 01:42:13 2021) host-id=3 score=3400 vm_conf_refresh_time=3106395 (Thu May 6 01:42:13 2021) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False Deployment: I have attached logs from Host2 and Engine – if anything is missing, please let me know. Kindly awaiting your reply. You might notice slight time change on Host2 logs – that’s after re-provisioning I did not set correct timezone ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

oVirt deploy new HE Host problem
by Marko Vrgotic 06 May '21

06 May '21

Hi Strahil and Yedidyah, As agreed, short summary: Deploy new HE host fails Pre Deploy state: * Host1 and Host3 are current HE HA pool * Host1 and Host3 are unaware of Host2 (check below) * I am trying to add Host2 to HE HA pool * Host2 is fully reinstalled – clean OS * Host2 is added to oVirt as regular Host * Host2 is currently in Maintenance mode (waiting for Reinstall with HE Deploy) [root@ovirt-sj-03 ~]# hosted-engine --vm-status --== Host ovirt-sj-01.ictv.com (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : d15bb877 local_conf_timestamp : 3103909 Host timestamp : 3103909 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3103909 (Thu May 6 01:42:06 2021) host-id=1 score=3400 vm_conf_refresh_time=3103909 (Thu May 6 01:42:06 2021) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host ovirt-sj-03.ictv.com (id: 3) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 Engine status : {"health": "good", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : 15801717 local_conf_timestamp : 3106395 Host timestamp : 3106395 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3106395 (Thu May 6 01:42:13 2021) host-id=3 score=3400 vm_conf_refresh_time=3106395 (Thu May 6 01:42:13 2021) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False Deployment: I have attached logs from Host2 and Engine – if anything is missing, please let me know. Kindly awaiting your reply. You might notice slight time change on Host2 logs – that’s after re-provisioning I did not set correct timezone ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 05 May '21

05 May '21

In my opinion, reinstalling the engine will never give an insight how the situation went so bad. Currently, at least the HE can live migrate between host1 & host3 and most probably will power up properly in case of serious problems. Yet, it's quite strange why the setup of host2 failsand on top of that the symbolic links to the relevant storage are missing. Marco, let's have a separate thread for the host2 issue. Best Regards,Strahil Nikolov

1 0

Ovirt 4.4.6 - Automate disable of default reboot of host add
by Don Dupuis 05 May '21

05 May '21

Hi I install alot of hosts via python sdk. My scripts were working perfectly in 4.3. Now in 4.4 the hosts reboot by default on installation and I don't want this. I have tried in python and in ansible, but now luck. Can someone guide me on the best way to achieve this? Thanks Don

3 7

Re: Import VM with Xen Based in Ovirt
by Guillaume Pavese 05 May '21

05 May '21

Hello, You can do 'virsh pool-create' on the same directory as your currently running Xen VMs' disks. That will make the disks visible to libvirt (prerequisite for them to be exportable to oVirt). If libvirt doesn't see your stopped VMs when doing 'virsh list --all' you need to add the <UUID> parameter to their definition file - For that, as explain, first have your vm running, then do 'virsh dumpxml vmname' You will see something like <domain type='xen' id='10'> <name>vmname</name> <uuid>7ecf52a5-3657-fc7a-b2e2-66edd4cf1b6c</uuid> - Then, shutdown your vm and copy the UUID from the dump you just did in your vm definition file : In /etc/xen/vmname put the missing line with the following format: uuid = "7ecf52a5-3657-fc7a-b2e2-66edd4cf1b6c" The vm should now be visible by libvirt when doing 'virsh list --all' Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Sun, May 2, 2021 at 6:47 PM IT (Hema Arun) <HemaA(a)emiratesnbd.com> wrote: > Hi Guillaume, > > > > Thank you for the reply back. I am so glad that I received a reply from > someone. > > I would like to try this option but I am not much familiar with virsh. I > am having Oracle VM which is based on Xen. I don’t have a pool listed in “'virsh > pool-list'" > > > > My existing OracleVM has already the vmdisks folder mounted as a > repository. When I do “virsh pool-create" from the Host, any idea will it > conflict the existing repository? > > > > Also to list the stopped VMs, what entry is to be added in the XML file. > Can you help me with that. > > > > Thanks a lot > > > > Regards, > > *Hema Arun* > > Senior Technology Engineer > > Group Information Technology - DevOps Chapter – *Linux and MiddleWare > Elastic Compute Platform * > > Teams: @hemaa > > > > *From:* Guillaume Pavese <guillaume.pavese(a)interactiv-group.com> > *Sent:* Wednesday, April 28, 2021 4:24 PM > *To:* IT (Hema Arun) <HemaA(a)emiratesnbd.com> > *Cc:* hemaa--- via Users <users(a)ovirt.org>; Strahil Nikolov < > hunter86_bg(a)yahoo.com> > *Subject:* Re: [ovirt-users] Re: Import VM with Xen Based in Ovirt > > > > CAUTION: This email originated from outside the Emirates NBD Group. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > > I successfully imported xem vm to oVirt with the GUI > > You need libvirt to be able to list the powered-off vm on the xen host > first > > > > What was missing for me at first was : > > > > 1) You need a libvirt pool referencing the path to VMs disks. > > Verify that's not already the case with 'virsh pool-list', > > > > Otherwise create it : > > # cat vm-pool.xml > <pool type="dir"> > <name>vm-pool</name> > <target> > <path>/path/to/vmdisks</path> > </target> > </pool> > > # virsh pool-create vm-pool.xml > # virsh pool-list ## Should list the newly created pool > > > > 2) Then each VM need an UUID present in its xml definition file. > > If not present, you can start the vm, do 'virsh dumpxml domainname' and > get the UUID for the running VM, > > then power it off and edit the xml to add this UUID line, that will make > it visible to virsh when powered off. > > You can check now with : 'virsh list --all' > > > > 3) For a successful import I found that it's also necessary to > > - install a non xen kernel in the vm, > > - and have disk listed by UUIDs in /etc/fstab (maybe not necessary this > last one) > > > > > > Good luck, > > > > > > Guillaume Pavese > Ingénieur Système et Réseau > > Interactiv-Group > > > > > > On Wed, Apr 28, 2021 at 7:38 PM Strahil Nikolov via Users <users(a)ovirt.org> > wrote: > > I think that you have to export the BM as OVA or convert it to KVM before > being able to import via oVirt > > > > Best Regards, > > Strahil Nikolov > > On Wed, Apr 28, 2021 at 12:36, hemaa--- via Users > > <users(a)ovirt.org> wrote: > > Any leads please? > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/WB7KTXNCLD27R… > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/IIFFJYN3LBR3W… > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> > > > > Ce message et toutes les pièces jointes (ci-après le “message”) sont > établis à l’intention exclusive de ses destinataires et sont confidentiels. > Si vous recevez ce message par erreur, merci de le détruire et d’en avertir > immédiatement l’expéditeur. Toute utilisation de ce message non conforme a > sa destination, toute diffusion ou toute publication, totale ou partielle, > est interdite, sauf autorisation expresse. L’internet ne permettant pas > d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) > décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse > ou il aurait été modifié. IT, ES, UK. > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Finteracti…> > DISCLAIMER: "This e-mail message including any of its attachments is > intended solely for the addressee(s) and may contain privileged > information. If you are not the addressee or you have received this email > message in error, please notify the sender who will remove your details > from its database. You are not authorized to read, copy, disseminate, > distribute or use this e-mail message or any attachment to it in any manner > and must delete the email and destroy any hard copies of it. This e-mail > message does not contain financial instructions or commitments of any kind. > Any views expressed in this message are those of the individual sender and > do not necessarily reflect the views of Emirates NBD PJSC, or any other > related subsidiaries, entities or persons." > -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>

1 0

Re: Unable to migrate Engine to another HE Host
by Yedidyah Bar David 05 May '21

05 May '21

On Wed, May 5, 2021 at 3:18 PM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > Status Update: > > > > During migration between Host1 and Host3, links are being created and I see no Errors as before in the agent/broker logs. > > > > Having all links in place, and having waited 24hours, for potential Engine updates, I tried to deploy HE on Host2: > > Part1: > > From oVirt UI – HostedEngine Deploy > Imeddiately on Host2 started seeing messages like “ Is the HostedEngine deployed ?” > hosted-engine.conf got populated only with host_id and ca_path > it failed > > Part2: > > I noticed that during deployment no links in /var/rund/vdsm/storage/<hosted_storage_id> were created > Copied hosted-engine.conf from Host1 to Host2 > Replace the host_id with correct value > Reran the deployment > Noticed that /var/rund/vdsm/storage/<hosted_storage_id> two links got created, one of them being the link to metadata_image > Host1 and Host3 hosted-engine –vm-status was showing Host2 but with status unknown/stale data > Deployment failed > > Part3: > > Since link to conf_image was not created in first phase, I added it manually > Populated the hosted-engine.conf > Reran the deployment > Same result > Hosted-engine.conf on Host2 would end up with only host_id and ca_path values and deployment would fail > > > > At this point, I cleaned up all hosted-engie remains from Host2 using ovirt-hosted-engine-cleanup and removed metadata of Host2 from host1 and Host3 > > > > I am out of ideas. It seems that Host1 and Host3 are happily operating, but I am unable to add any other hosts to HE pool. > > > > Please assist if you have any ideas. Look, Marko - I admit it seems to me like you simply enjoy debugging this yourself... 1. Did you try to reinstall the OS on host2? If not, is there any reason not to? Other than the (legitimate!) wish to "understand what's broken and fix just that"? Would reinstallation take a lot of time/work? You can also do a full backup beforehand and then compare later, to see what the differences were. 2. It's very hard to help you by only guessing around. If you have a concrete issue, such as "I add a host with 'Deploy Hosted Engine' and this fails", then please provide all relevant logs. 3. If you still want to continue debugging by yourself, fine - the code is open, and at least I personally try quite hard to make it easy, in the relatively small parts of the code I touched, to search around it even without having a complete picture of its structure, mainly by making texts inside logs be "unique enough" so that you can easily find them in the code. > > I am open to executing restore – but I would much rather like to at least discover where or what is the problem, before moving to planning restore. Good luck and best regards, -- Didi

2 1

Engine dm-1 disk crashed
by marcel d'heureuse 05 May '21

05 May '21

Hi, We shutdown the complete ovirt system, if a poweroutage occurs and we don't want to make an hard power off. So we stop all vms, put the storages in maintenance and shut down the engine, disconnect storages and shut down the nodes. On power up, sometimes it works fine but in some cases the engine need a xfs_repair session to start. What we make wrong? Have you an idea? Br Marcel

3 9

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 05 May '21

05 May '21

Status Update: During migration between Host1 and Host3, links are being created and I see no Errors as before in the agent/broker logs. Having all links in place, and having waited 24hours, for potential Engine updates, I tried to deploy HE on Host2: Part1: * From oVirt UI – HostedEngine Deploy * Imeddiately on Host2 started seeing messages like “ Is the HostedEngine deployed ?” * hosted-engine.conf got populated only with host_id and ca_path * it failed Part2: * I noticed that during deployment no links in /var/rund/vdsm/storage/<hosted_storage_id> were created * Copied hosted-engine.conf from Host1 to Host2 * Replace the host_id with correct value * Reran the deployment * Noticed that /var/rund/vdsm/storage/<hosted_storage_id> two links got created, one of them being the link to metadata_image * Host1 and Host3 hosted-engine –vm-status was showing Host2 but with status unknown/stale data * Deployment failed Part3: * Since link to conf_image was not created in first phase, I added it manually * Populated the hosted-engine.conf * Reran the deployment * Same result * Hosted-engine.conf on Host2 would end up with only host_id and ca_path values and deployment would fail At this point, I cleaned up all hosted-engie remains from Host2 using ovirt-hosted-engine-cleanup and removed metadata of Host2 from host1 and Host3 I am out of ideas. It seems that Host1 and Host3 are happily operating, but I am unable to add any other hosts to HE pool. Please assist if you have any ideas. I am open to executing restore – but I would much rather like to at least discover where or what is the problem, before moving to planning restore. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Wednesday, 5 May 2021 at 12:16 To: Strahil Nikolov <hunter86_bg(a)yahoo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Both hosts have same agent.conf: ❯ cat <(ssh root(a)ovirt-sj-01.ictv.com 'cat /etc/ovirt-hosted-engine-ha/agent.conf') [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 ❯ cat <(ssh root(a)ovirt-sj-03.ictv.com 'cat /etc/ovirt-hosted-engine-ha/agent.conf') [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 And its same on the good platform: ❯ ssh root(a)ovirt-hv-01.avinity.tv 'cat /etc/ovirt-hosted-engine-ha/agent.conf' [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Date: Wednesday, 5 May 2021 at 00:08 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Yes, /etc/ovirt-hosted-engine-ha/agent.conf should be enough. Best Regards, Strahil Nikolov

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 05 May '21

05 May '21

Yes,/etc/ovirt-hosted-engine-ha/agent.conf should be enough. Best Regards,Strahil Nikolov

2 1

oVirt 4.4.7 development
by Michal Skrivanek 04 May '21

04 May '21

Hi all, we are starting to work on oVirt 4.4.7 and as you probably noticed we are slowly transitioning away from CentOS. We added CentOS Stream option for regular hosts, and the current 4.4.6 Node and Appliance are already built from Stream. We also have some initial patches for alternative RHEL clones other than CentOS, however we can’t really support them until they provide the required dependencies. We are not ditching CentOS just yet, but we can no longer accommodate the long delay it takes CentOS team to produce a next minor version, hence for oVirt 4.4.7 we will require the operating system to be at the RHEL 8.4 level - that’s a CentOS 8.4 if that gets released in time or the actual RHEL 8.4 or any 8.4 clone like Alma, Rocky, OL with the right dependencies (probably the ones from Stream) or CentOS Stream. In the meantime we're going to be moving the existing CI resources to test with CentOS Stream to unblock development of new features. If you would want to help with testing other OSes patches are welcome to add support for them Thanks, michal

2 2

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 04 May '21

04 May '21

Ok – These are the symlinks in place: 1. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_conf_image> 2. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_metadata_image> 3. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_lockspace_image> 4. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_vmdisk_id_image> - this one exist only on the current Host 5. /var/run/vdsm/storage/<hosted_storage_id>/<one more_link_I_have_no_clue what_it_is_for> Migration issue – seem to be solved at part 3: Part1: * Try to migrate HE from Host1 to Host3 – migration fails * Analysis showed that Host3 was missing the symlink num4 – which should be created during migration * I created symlink num4 and that migration works but HE on donor host has status “down_unexpected” instead of “down” Part2: * Next thing I noticed was that symlink num1 was missing on both Hosts * Added symlink num1 to both Hosts and reran the migration * Migration done successfully from Host1 to Host3 and HE on donor host had status “down” Part3: * Migrating back from Host3 to Host1 * Noticed that at beginning of migration the symlink num4 gets created on destination Host (Host1 in this case) * This time the symlink num4 on destination was not deleted and migration went through Part4: * Repeated migration from Host1 to Host3 * All went as expected. New HE Host deploy issue: * Current conclusion is that the new HE Host deploy was failing as the symlink num1 was not existing on running Hosts ( 1 and 3) so it was reported as missing/non-existent * Since the link is now added and it persists – I will try to deploy new HA Host I will keep you posted. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Tuesday, 4 May 2021 at 13:47 To: Strahil Nikolov <hunter86_bg(a)yahoo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil, Creating link towards conf_image on shared storage, did help in regards to HE migrating with donor not ending up with down_expected but simply down. Regarding “Can you paste the contents of the ha daemons' config where the shared storage should be defined?”, are you referring to /etc/ovirt-hosted-engine-ha/agent.conf, var/lib/ovirt-hosted-engine-ha/broker.conf and ha.conf ? If not, please give me specific files, so I can provide configs. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Date: Tuesday, 4 May 2021 at 09:21 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** It's quite interesting why the link is not created when it should ... Can you paste the contents of the ha daemons' config where the shared storage should be defined? You can create the link if needed, but it won't solve the problem. Best Regards, Strahil Nikolov

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 04 May '21

04 May '21

It's quite interesting why the link is not created when it should ...Can you paste the contents of the ha daemons' config where the shared storage should be defined? You can create the link if needed, but it won't solve the problem. Best Regards,Strahil Nikolov

2 1

OLVM: error deleting snapshot live
by Renevey Christian 04 May '21

04 May '21

Hi, With last OLVM update installed, we are unable to delete snapshot without stopping the vm (live merge error) Vdsm.log: 2021-05-04 09:44:28,389+0200 ERROR (jsonrpc/6) [virt.vm] (vmId='dba27e34-f8f3-4d9b-a6fd-95ba92c2970e') Live merge failed (job: 36f515b5-fb9e-4e29-b5f8-5b7c062cef5d) (vm:5957) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5955, in merge bandwidth, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 719, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /rhev/data-center/mnt/blockSD/47c2336e-1936-4b28-b759-efcfe8a19b58/images/269f76c4-2a63-4cdf-a2da-0f56f4acc464/c317c53b-ad79-421f-9263-f1f19c81a9a7 which is already in use 2021-05-04 09:44:28,412+0200 INFO (jsonrpc/6) [api.virt] FINISH merge return={'status': {'message': 'Merge failed', 'code': 52}} from=::ffff:192.168.1.110,38834, flow_id=390f0b25-f364-4adc-b7b3-b732ee0e0213, vmId=dba27e34-f8f3-4d9b-a6fd-95ba92c2970e (api:54) 2021-05-04 09:44:28,413+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 52) in 0.58 seconds (__init__:312) 2021-05-04 09:44:29,124+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.1.110,38834 (api:48) Any idea whats wrong ? Rgds

2 1

How to use Ovirt node 4.4.5.0 with local storage
by danilosevilla＠gmail.com 04 May '21

04 May '21

Hi, I'm totally new with Ovirt. I have a production server in which I planned to install Ovirt. I need to use their local storage for Virtualization, but it seems that only have chance to storage in NAS or other network or fiber storage. Can I use the local storage for virtualization?. If yes, how can I do that. Thanks.

2 1

error id 980
by ozmen62＠hotmail.com 04 May '21

04 May '21

Hi, First of all, thanks for this great list and the people who try to help eachother. This time my problem is about engine OVF file or engine storage setting. How can i came with this idea, because every 2-3 hours engine try to migrate other host(From host1 to host2) "Invalid status on Data Center B300. Setting status to Non Responsive." when i check hosts log , see these ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM stopped on localhost ovirt-ha-agent ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore ERROR Unable to extract HEVM OVF i believe there is some kind of fc storage path or config problem. My storage setting is good. there is no error about them Is there anyone who experienced about that to solve it thanks

2 5

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Investigation so far has showed, that symlink for conf_volume/conf_image is missing on Hosts 1 and 3. According to the ovirt-engine deep dive, connection/access to conf_volume is required for adding new HE Hosts as it holds its configuration template. This is most likely the reason why I could not add new Host and why the Host1 and Host3 were occasionally complaining the the conf_image_uuid is missing. I also checked the DB tb images – the conf_image and conf_volume entries exist. Question: Considering DB entries seem correct, am I “safe” to manually create the symlink on Host1 and Host3 , from global maintenance mode, set correct permissions and than exit maintenance mode? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 12:44 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Cluster level 4.5
by Don Dupuis 03 May '21

03 May '21

What version of libvirt is required for a host to be put in this cluster level? I am using CentOS 8.3 and cpu is Cascade Lake Server. It says that my host is only compatible with cluster version 4.2,4.3 and 4.4. I am doing a new install of Ovirt 4.4.5. I have tried to update libvirt version but have run into issues. Currently installed libvirt is libvirt-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64. Don

2 1

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 03 May '21

03 May '21

As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing.Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. Best Regards,Strahil Nikolov On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic<M.Vrgotic(a)activevideo.com> wrote:  Dear oVirt, I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: On Engine: 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) On source Host: 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run time.time(), migrationParams, machineParams File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration self._migration_flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: operation aborted: migration out job: canceled by client I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. This is production system, so I cannot just move on to upgrading/deploying to 4.4. Additionally – : - Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? - What is the effect or reinitialize-lockspace? Kindly awaiting your reply. Happy to provide any additional information needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com w: www.activevideo.com ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FRTRNTSGPLQOS…

3 2

new engine restore from backup
by ozmen62＠hotmail.com 03 May '21

03 May '21

Hi, I've built e new engine and tried to restore from backup after run engine-setup it returns; [ ERROR ] Failed to execute stage 'Misc configuration': function getdwhhistorytimekeepingbyvarname(unknown) does not exist LINE 2: select * from GetDwhHistoryTimekeepingByVarName( redhat page says seploy new engine, but already done it. old and new engine are same version Do you have any idea how can i restore engine. There is no another backup

2 4

Trouble migrating the websocket-proxy / Trying to make VM-Portal publicly available
by tivon.haeberlein＠secges.de 02 May '21

02 May '21

Hi community, I hope you can help me on this one. My oVirt cluster is deployed on an internal network (10.1.30.0/24). At the moment I am trying to make the VM-Portal publicly available so people can administer their vms with the portal and noVNC. I have read the oVirt Documentation (as well as trying a couple of things) and found that the best way to accomplish this is by installing a proxy server which proxies HTTP and HTTPS to the engine. To get the noVNC working I have to install the ovirt-engine-websocket-proxy on the proxy host as well and set it up. For the proxy host I'm using centos 8. For the HTTP and HTTPS proxy I'm using haproxy (which is working perfectly already). For the noVNC Proxy I'm trying to follow the documentation (https://www.ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engin…) but am facing a couple of issues. Issues start with removing the websocket proxy from the engine machine. I put the cluster into global-maintenance-mode and run "engine-cleanup" as told in the docs: [root@engine ~]# engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ INFO ] Hosted Engine HA is in Global Maintenance mode. All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502182730-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed As you can see the output differs a bit from the docs. Instead of being asked "Do you want to remove all components? (Yes, No) [Yes]: No" I'm being notified that all data will be lost when continuing. Of course I do not want to loose all data so I abort the cleanup. I then tried to just install the websocket-proxy on the proxy server and ran into the next issue. On the proxy server I am also following the docs and running "yum install ovirt-engine-websocket-proxy" followed by "engine-setup": [root@proxy ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- Configure WebSocket Proxy on this host (Yes, No) [Yes]: --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] DNF No matches found for the following disable plugin patterns: versionlock [ INFO ] No product updates found ### I've cut a bit of the output to keep it shorter ### [ INFO ] Stage: Transaction setup [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ ERROR ] Failed to execute stage 'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans' [ INFO ] DNF Performing DNF transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502183159-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed As you can see it failed at stage "'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans'". Here I did try to disable SElinux as well as enforcing it (as set at default) without any success. Am I on the right approach to make the VM-Portal publicly avalible by doing this or is there a better / more supported way? Can you help me out on getting this running or pointing me in the right direction (if this is the right approach)? Thank you guys! Best regards, Tivon I'm new to the community but will help to keep oVirt growing and support where I can.

1 0

How to assign a server disk and nic profile via REST API?
by ovirt.org＠nevim.eu 02 May '21

02 May '21

Hello to everybody, for about an hour I have stuck my nose in the ovirt and rhev documentation, but I still can't understand how to use the REST API to change the disk profile and nothing the server profile. Ideally, how to set it for them when setting up a VM. Thank you so much for the advice.

2 7