- Users - oVirt List Archives

Unable to live migrate a VM from 4.4.2 to 4.4.3 CentOS Linux host
by Gianluca Cecchi 23 Dec '20

23 Dec '20

Hello, I was able to update an external CentOS Linux 8.2 standalone engine from 4.4.2 to 4.4.3 (see dedicated thread). Then I was able to put into maintenance one 4.4.2 host (CentOS Linux 8.2 based, not ovirt node ng) and run: [root@ov301 ~]# dnf update Last metadata expiration check: 0:27:11 ago on Wed 11 Nov 2020 08:48:04 PM CET. Dependencies resolved. ====================================================================================================================== Package Arch Version Repository Size ====================================================================================================================== Installing: kernel x86_64 4.18.0-193.28.1.el8_2 BaseOS 2.8 M kernel-core x86_64 4.18.0-193.28.1.el8_2 BaseOS 28 M kernel-modules x86_64 4.18.0-193.28.1.el8_2 BaseOS 23 M ovirt-ansible-collection noarch 1.2.1-1.el8 ovirt-4.4 276 k replacing ovirt-ansible-engine-setup.noarch 1.2.4-1.el8 replacing ovirt-ansible-hosted-engine-setup.noarch 1.1.8-1.el8 Upgrading: ansible noarch 2.9.15-2.el8 ovirt-4.4-centos-ovirt44 17 M bpftool x86_64 4.18.0-193.28.1.el8_2 BaseOS 3.4 M cockpit-ovirt-dashboard noarch 0.14.13-1.el8 ovirt-4.4 3.5 M ioprocess x86_64 1.4.2-1.el8 ovirt-4.4 37 k kernel-tools x86_64 4.18.0-193.28.1.el8_2 BaseOS 3.0 M kernel-tools-libs x86_64 4.18.0-193.28.1.el8_2 BaseOS 2.8 M libiscsi x86_64 1.18.0-8.module_el8.2.0+524+f765f7e0 AppStream 89 k nftables x86_64 1:0.9.3-12.el8_2.1 BaseOS 311 k ovirt-hosted-engine-ha noarch 2.4.5-1.el8 ovirt-4.4 325 k ovirt-hosted-engine-setup noarch 2.4.8-1.el8 ovirt-4.4 227 k ovirt-imageio-client x86_64 2.1.1-1.el8 ovirt-4.4 21 k ovirt-imageio-common x86_64 2.1.1-1.el8 ovirt-4.4 155 k ovirt-imageio-daemon x86_64 2.1.1-1.el8 ovirt-4.4 15 k ovirt-provider-ovn-driver noarch 1.2.32-1.el8 ovirt-4.4 27 k ovirt-release44 noarch 4.4.3-1.el8 ovirt-4.4 17 k python3-ioprocess x86_64 1.4.2-1.el8 ovirt-4.4 33 k python3-nftables x86_64 1:0.9.3-12.el8_2.1 BaseOS 25 k python3-ovirt-engine-sdk4 x86_64 4.4.6-1.el8 ovirt-4.4 560 k python3-perf x86_64 4.18.0-193.28.1.el8_2 BaseOS 2.9 M python3-pyasn1 noarch 0.4.6-3.el8 ovirt-4.4-centos-opstools 140 k python3-pyasn1-modules noarch 0.4.6-3.el8 ovirt-4.4-centos-opstools 151 k qemu-img x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 1.0 M qemu-kvm x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 118 k qemu-kvm-block-curl x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 129 k qemu-kvm-block-gluster x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 131 k qemu-kvm-block-iscsi x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 136 k qemu-kvm-block-rbd x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 130 k qemu-kvm-block-ssh x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 131 k qemu-kvm-common x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 1.2 M qemu-kvm-core x86_64 15:4.2.0-29.el8.6 ovirt-4.4-advanced-virtualization 3.4 M selinux-policy noarch 3.14.3-41.el8_2.8 BaseOS 615 k selinux-policy-targeted noarch 3.14.3-41.el8_2.8 BaseOS 15 M spice-server x86_64 0.14.2-1.el8_2.1 AppStream 404 k tzdata noarch 2020d-1.el8 BaseOS 471 k vdsm x86_64 4.40.35.1-1.el8 ovirt-4.4 1.4 M vdsm-api noarch 4.40.35.1-1.el8 ovirt-4.4 106 k vdsm-client noarch 4.40.35.1-1.el8 ovirt-4.4 24 k vdsm-common noarch 4.40.35.1-1.el8 ovirt-4.4 136 k vdsm-hook-ethtool-options noarch 4.40.35.1-1.el8 ovirt-4.4 9.8 k vdsm-hook-fcoe noarch 4.40.35.1-1.el8 ovirt-4.4 10 k vdsm-hook-openstacknet noarch 4.40.35.1-1.el8 ovirt-4.4 18 k vdsm-hook-vhostmd noarch 4.40.35.1-1.el8 ovirt-4.4 17 k vdsm-hook-vmfex-dev noarch 4.40.35.1-1.el8 ovirt-4.4 11 k vdsm-http noarch 4.40.35.1-1.el8 ovirt-4.4 15 k vdsm-jsonrpc noarch 4.40.35.1-1.el8 ovirt-4.4 31 k vdsm-network x86_64 4.40.35.1-1.el8 ovirt-4.4 331 k vdsm-python noarch 4.40.35.1-1.el8 ovirt-4.4 1.3 M vdsm-yajsonrpc noarch 4.40.35.1-1.el8 ovirt-4.4 40 k Installing dependencies: NetworkManager-ovs x86_64 1:1.22.14-1.el8 ovirt-4.4-copr:copr.fedorainfracloud.org:networkmanager:NetworkManager-1.22 144 k Transaction Summary ====================================================================================================================== Install 5 Packages Upgrade 48 Packages Total download size: 116 M After reboot I can activate the host (strange that I see many pop up messages about "finished activating host") and the host is shown as OS Version: RHEL - 8.2 - 2.2004.0.2.el8 OS Description: CentOS Linux 8 (Core) Kernel Version: 4.18.0 - 193.28.1.el8_2.x86_64 KVM Version: 4.2.0 - 29.el8.6 LIBVIRT Version: libvirt-6.0.0-25.2.el8 VDSM Version: vdsm-4.40.35.1-1.el8 SPICE Version: 0.14.2 - 1.el8_2.1 GlusterFS Version: [N/A] CEPH Version: librbd1-12.2.7-9.el8 Open vSwitch Version: [N/A] Nmstate Version: nmstate-0.2.10-1.el8 Kernel Features: MDS: (Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable), L1TF: (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable), SRBDS: (Not affected), MELTDOWN: (Mitigation: PTI), SPECTRE_V1: (Mitigation: usercopy/swapgs barriers and __user pointer sanitization), SPECTRE_V2: (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling), ITLB_MULTIHIT: (KVM: Mitigation: Split huge pages), TSX_ASYNC_ABORT: (Not affected), SPEC_STORE_BYPASS: (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) VNC Encryption: Disabled FIPS mode enabled: Disabled while another host still in 4.4.2: OS Version: RHEL - 8.2 - 2.2004.0.2.el8 OS Description: CentOS Linux 8 (Core) Kernel Version: 4.18.0 - 193.19.1.el8_2.x86_64 KVM Version: 4.2.0 - 29.el8.3 LIBVIRT Version: libvirt-6.0.0-25.2.el8 VDSM Version: vdsm-4.40.26.3-1.el8 SPICE Version: 0.14.2 - 1.el8 GlusterFS Version: [N/A] CEPH Version: librbd1-12.2.7-9.el8 Open vSwitch Version: [N/A] Nmstate Version: nmstate-0.2.10-1.el8 Kernel Features: MDS: (Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable), L1TF: (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable), SRBDS: (Not affected), MELTDOWN: (Mitigation: PTI), SPECTRE_V1: (Mitigation: usercopy/swapgs barriers and __user pointer sanitization), SPECTRE_V2: (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling), ITLB_MULTIHIT: (KVM: Mitigation: Split huge pages), TSX_ASYNC_ABORT: (Not affected), SPEC_STORE_BYPASS: (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) VNC Encryption: Disabled FIPS mode enabled: Disabled But if I try to move away VMs from the 4.4.2 host to the 4.4.3 one I get error: Failed to migrate VM c8client to Host ov301 . Trying to migrate to another Host. (btw: there is no other active host; there is a ov300 host that is in maintenance) No available host was found to migrate VM c8client to. It seems the root error in engine.log is: 2020-11-11 21:44:42,487+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] Migration of VM 'c8client' to host 'ov301' failed: VM destroyed during the startup. On target host in /var/log/libvirt/qemu/c8clinet.log I see: 2020-11-11 20:44:40.981+0000: shutting down, reason=failed In target vdsm.log 2020-11-11 21:44:39,958+0100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.migrationCreate took more than 1.00 seconds to succeed: 1.97 (__init__:316) 2020-11-11 21:44:40,230+0100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=cb51fd4a-09d3-4d77-821b-391da2467487 (api:48) 2020-11-11 21:44:40,231+0100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'fa33df49-b09d-4f86-9719-ede649542c21': {'code': 0, 'lastCheck': '4.1', 'delay': '0.000836715', 'valid': True, 'version': 4, 'acquired': True, 'actual': True}} from=internal, task_id=cb51fd4a-09d3-4d77-821b-391da2467487 (api:54) 2020-11-11 21:44:41,929+0100 INFO (jsonrpc/5) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.4.192.32,52266, vmId=c95da734-7ed1-4caa-bacb-3fa24f4efb56 (api:48) 2020-11-11 21:44:41,930+0100 INFO (jsonrpc/5) [virt.vm] (vmId='c95da734-7ed1-4caa-bacb-3fa24f4efb56') Release VM resources (vm:4666) 2020-11-11 21:44:41,930+0100 INFO (jsonrpc/5) [virt.vm] (vmId='c95da734-7ed1-4caa-bacb-3fa24f4efb56') Stopping connection (guestagent:444) 2020-11-11 21:44:41,930+0100 INFO (jsonrpc/5) [vdsm.api] START teardownImage(sdUUID='fa33df49-b09d-4f86-9719-ede649542c21', spUUID='ef17cad6-7724-4cd8-96e3-9af6e529db51', imgUUID='ff10a405-cc61-4d00-a83f-3ee04b19f381', volUUID=None) from=::ffff:10.4.192.32,52266, task_id=177461c0-83d6-4c90-9c5c-3cc8ee9150c7 (api:48) It seems that during the host update the OVN configuration has not been maintained. Right now all my active VMs are with at least a vnic on OVN so I cannot test the scenario of migrating a VM without OVN based vnic. In fact on engine I see only the currently active host in 4.4.2 (ov200) and another host that is in maintenance (it is still in 4.3.10; I wanted to update to 4.4.2 but I realized that 4.4.3 has been out...): [root@ovmgr1 ovirt-engine]# ovn-sbctl show Chassis "6a46b802-5a50-4df5-b1af-e73f58a57164" hostname: "ov200.mydomain" Encap geneve ip: "10.4.192.32" options: {csum="true"} Port_Binding "2ae7391b-4297-4247-a315-99312f6392e6" Port_Binding "c1ec60a4-b4f3-4cb5-8985-43c086156e83" Port_Binding "174b69f8-00ed-4e25-96fc-7db11ea8a8b9" Port_Binding "66359e79-56c4-47e0-8196-2241706329f6" Port_Binding "ccbd6188-78eb-437b-9df9-9929e272974b" Chassis "ddecf0da-4708-4f93-958b-6af365a5eeca" hostname: "ov300.mydomain" Encap geneve ip: "10.4.192.33" options: {csum="true"} [root@ovmgr1 ovirt-engine]# Any hint about the reason of losing OVN config for ov301 and the correct procedure to get it again and persiste future updates? NOTE: this was a cluster in 4.3.10 and I updated it to 4.4.2 and I noticed that the OVN config was not retained and I had to run on hosts: [root@ov200 ~]# vdsm-tool ovn-config engine_ip ov200_ip_on_mgmt Using default PKI files Created symlink /etc/systemd/system/multi-user.target.wants/openvswitch.service → /usr/lib/systemd/system/openvswitch.service. Created symlink /etc/systemd/system/multi-user.target.wants/ovn-controller.service → /usr/lib/systemd/system/ovn-controller.service. [root@ov200 ~]# Now it seems the problem persists... Why do I have to run each time? Gianluca

1 3

Get Host Capabilities failed: Internal JSON-RPC error: {'reason': 'internal error: Duplicate key'}
by tommy 23 Dec '20

23 Dec '20

Hi,everyone: I got this error in my ovirt env: VDSM ooengh1.tltd.com command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': 'internal error: Duplicate key'} Systemctl message is: Dec 23 20:48:48 ooengh1.tltd.com vdsm[2431]: ERROR Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1371, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 93, in get machinetype.compatible_cpu_models()) File "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 43, in __call__ value = self.func(*args) File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 142, in compatible_cpu_models all_models = domain_cpu_models(c, arch, cpu_mode) File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 97, in domain_cpu_models domcaps = conn.getDomainCapabilities(None, arch, None, virt_type, 0) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3844, in getDomainCapabilities if ret is None: raise libvirtError ('virConnectGetDomainCapabilities() failed', conn=self) libvirtError: internal error: Duplicate key AnyOne can help me ? Thanks!

1 0

Cannot ping from a nested VM
by wodel youchi 23 Dec '20

23 Dec '20

Hi, We have an HCI plateforme based upon 4.4.3 version. We activated the use of nested kvm and we reinstalled all the nodes to be able to use it. Then we created a new nested HCI on top of the physical HCI to test ansible deployment. The deployment went well until the phase which adds the two other nodes (the two other virtualized hypervisors), but the VM-Manager couldn't add them. After investigation, we found that the nested VM-Manager could not communicate except with it's first virtualized hypervisor. To make the nested VM-Manager capable of communicating with the outside world, we had to stop firewalld on the physical node of the HCI. Any ideas? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 2

Upgrade to 4.4.4
by Jonathan Baecker 22 Dec '20

22 Dec '20

Hello, I'm running here a upgrade from 4.4.3 to latest 4.4.4, on a 3 node self hosted cluster. The engine upgrade went fine and now I'm on host upgrades. When I check there the updates it shows only *ovirt-node-ng-image-update-4.4.4-1.el8.noarch.rpm*. For that I have run manual updates on each host, with maintenance mode -> yum update -> reboot. When I run now on the engine *cat /etc/redhat-release *it show: *CentOS Linux release 8.3.2011* But on my nodes it shows still: *CentOS Linux release 8.2.2004 (Core)* How can this be? Best regards Jonathan

1 2

Is there a proper way to migrate an oVirt cluster to a new network?
by Kristian Petersen 21 Dec '20

21 Dec '20

Is there a way to properly migrate an oVirt cluster from one network to another properly? Especially when it comes to the self-hosted manager and its storage. -- Kristian Petersen System Administrator BYU Dept. of Chemistry and Biochemistry

1 0

how to gracefully stop engine and the HA before maintain the two host-engine servers ?
by tommy 20 Dec '20

20 Dec '20

When I stop one server, the other one auto start the engine vm, and I can not perform the maintaince. Thanks!

3 3

oVirt can not connect to KVM Libvirtd
by tommy 20 Dec '20

20 Dec '20

Hi,everyone! butI tried using oVirt to Manage the KVM Libvirtd Server, like this: But,it got error like this: VDSM host1 command GetVmsNamesFromExternalProviderVDS failed: Cannot recv data: Host key verification failed.: Connection reset by peer. Should I configure SSH Host Key ? where and how to config it ? Thanks!

1 1

OVA export issues
by Jonathan Baecker 19 Dec '20

19 Dec '20

Hello, I have here an older server, one host with external engine, 4.4.4-1.el8 installation, with a slow 2Gbit nfs storage. When I export OVAs from smaller VMs (under 40GB) it works but with bigger one, around 95GB I have problems. I have already change the export target to a local folder, which makes the process faster and the GUI shows no errors, but when I extract the vm.ovf file, from the exported archive, it is not complete. It missing a closing part from the xml definition. The end stops here: *</ovf:E* By my test before, I was able to add the closing part by hand and import the OVA, but then the VM had no disk attached. I don't know if the size is the problem, my backup script which export the VMs to the export domain works normal. I only know, that the error I can reproduce on minimum 2 VMs. The last part of the export log looks like: /var/log/ovirt-engine/ansible-runner-service.log:2020-12-19 12:02:52,610 - runner_service.services.playbook - DEBUG - cb_event_handler event_data={'uuid': '2ac62a85-7efa-4d14-a941-c1880bd016fd', 'counter': 35, 'stdout': 'changed: [onode-2.example.org]', 'start_line': 34, 'end_line': 35, 'runner_ident': '9ac880ce-4219-11eb-bfd8-5254000e4c2c', 'event': 'runner_on_ok', 'pid': 1039034, 'created': '2020-12-19T17:02:52.606639', 'parent_uuid': '5254000e-4c2c-ed4e-fdf7-000000000022', 'event_data': {'playbook': 'ovirt-ova-export.yml', 'playbook_uuid': 'ee4a698c-f639-49c8-8fa9-af2778f0862d', 'play': 'all', 'play_uuid': '5254000e-4c2c-ed4e-fdf7-000000000007', 'play_pattern': 'all', 'task': 'Rename the OVA file', 'task_uuid': '5254000e-4c2c-ed4e-fdf7-000000000022', 'task_action': 'command', 'task_args': '', 'task_path': '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-ova-export-post-pack/tasks/main.yml:2', 'role': 'ovirt-ova-export-post-pack', 'host': 'onode-2.discovery.intern', 'remote_addr': 'onode-2.discovery.intern', 'res': {'cmd': ['mv', '/mnt/intern/win2016-01.ova.tmp', '/mnt/intern/win2016-01.ova'], 'stdout': '', 'stderr': '', 'rc': 0, 'start': '2020-12-19 12:02:52.560917', 'end': '2020-12-19 12:02:52.572358', 'delta': '0:00:00.011441', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'mv "/mnt/intern/win2016-01.ova.tmp" "/mnt/intern/win2016-01.ova"', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], '_ansible_no_log': False}, 'start': '2020-12-19T17:02:51.954144', 'end': '2020-12-19T17:02:52.605820', 'duration': 0.651676, 'event_loop': None, 'uuid': '2ac62a85-7efa-4d14-a941-c1880bd016fd'}} Any ideas? Best regards Jonathan

2 1

Glusternetwork
by Ariez Ahito 18 Dec '20

18 Dec '20

can someone help me try to assign gluster network this is my current setup hosted-engine with stand alone gluster ovirt host 1 host 2 host 3 (and so on) eno1 192.168.0.10 ovirtmgmt enfs20 vm network dmz_1 dmz_2 dmz3 (and so on) enfs21 <--- i want to assign this physical network with gluster netowrk gluster1 gluster 2 (and so on) eno1 192.168.0.11 eno2 i followed the instruction in the offical documentation Create the logical network for gluster traffic Log in to the engine Browse to the engine and log in using the administrative credentials you configured in Chapter 8, Deploy the Hosted Engine using the Cockpit UI. Create a logical network for gluster traffic Click the Networks tab and then click New. The New Logical Network wizard appears. On the General tab of the wizard, provide a Name for the new logical network, and uncheck the VM Network checkbox. On the Cluster tab of the wizard, uncheck the Required checkbox. Click OK to create the new logical network. Enable the new logical network for gluster Click the Networks tab and select the new logical network. Click the Clusters sub-tab and then click Manage Network. The Manage Network dialogue appears. In the Manage Network dialogue, check the Migration Network and Gluster Network checkboxes. Click OK to save. Attach the gluster network to the host Click the Hosts tab and select the host. Click the Network Interfaces subtab and then click Setup Host Networks. Drag and drop the newly created network to the correct interface. Ensure that the Verify connectivity checkbox is checked. Ensure that the Save network configuration checkbox is checked. Click OK to save. so when i got to network> then click the logical network i create (gluster_net) cluster tab > network status is down? then click manage network button and assign migration network and gluster network

1 1

assigning logical networks to physical interface
by Ariez Ahito 18 Dec '20

18 Dec '20

Hi guys could someone help me or guide me on how to assign a logical network to a physical. our last ovirt engine was version 4.0 but with this latest release 4.4 seems other options are gone missing now.

1 1