February 2022 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

Re: [ovirt-users] Re: [=EXTERNAL=] Re: help using nvme/tcp storage with cinderlib and Managed Block Storage
by Nir Soffer 31 Mar '22

31 Mar '22

On Wed, Feb 23, 2022 at 6:24 PM Muli Ben-Yehuda <muli(a)lightbitslabs.com> wrote: > > Thanks for the detailed instructions, Nir. I'm going to scrounge up some hardware. > By the way, if anyone else would like to work on NVMe/TCP support, for NVMe/TCP target you can either use Lightbits (talk to me offline for details) or use the upstream Linux NVMe/TCP target. Lightbits is a clustered storage system while upstream is a single target, but the client side should be close enough for vdsm/ovirt purposes. I played with NVMe/TCP a little bit, using qemu to create a virtual NVMe disk, and export it using the kernel on one VM, and consume it on another VM. https://futurewei-cloud.github.io/ARM-Datacenter/qemu/nvme-of-tcp-vms/ One question about device naming - do we always get the same name of the device in all hosts? To support VM migration, every device must have unique name in the cluster. With multipath we always have unique name, since we disable "friendly names", so we always have: /dev/mapper/{wwid} With rbd we also do not use /dev/rbdN but a unique path: /dev/rbd/poolname/volume-vol-id How do we ensure cluster-unique device path? If os_brick does not handle it, we can to do in ovirt, for example: /run/vdsm/mangedvolumes/{uuid} -> /dev/nvme7n42 but I think this should be handled in cinderlib, since openstack have the same problem with migration. Nir > > Cheers, > Muli > -- > Muli Ben-Yehuda > Co-Founder and Chief Scientist @ http://www.lightbitslabs.com > LightOS: The Special Storage Sauce For Your Cloud > > > On Wed, Feb 23, 2022 at 4:55 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >> >> On Wed, Feb 23, 2022 at 4:20 PM Muli Ben-Yehuda <muli(a)lightbitslabs.com> wrote: >> > >> > Thanks, Nir and Benny (nice to run into you again, Nir!). I'm a neophyte in ovirt and vdsm... What's the simplest way to set up a development environment? Is it possible to set up a "standalone" vdsm environment to hack support for nvme/tcp or do I need "full ovirt" to make it work? >> >> It should be possible to install vdsm on a single host or vm, and use vdsm >> API to bring the host to the right state, and then attach devices and run >> vms. But I don't know anyone that can pull this out since simulating what >> engine is doing is hard. >> >> So the best way is to set up at least one host and engine host using the >> latest 4.5 rpms, and continue from there. Once you have a host, building >> vdsm on the host and upgrading the rpms is pretty easy. >> >> My preferred setup is to create vms using virt-manager for hosts, engine >> and storage and run all the vms on my laptop. >> >> Note that you must have some traditional storage (NFS/iSCSI) to bring up >> the system even if you plan to use only managed block storage (MBS). >> Unfortunately when we add MBS support we did have time to fix the huge >> technical debt so you still need a master storage domain using one of the >> traditional legacy options. >> >> To build a setup, you can use: >> >> - engine vm: 6g ram, 2 cpus, centos stream 8 >> - hosts vm: 4g ram, 2 cpus, centos stream 8 >> you can start with one host and add more hosts later if you want to >> test migration. >> - storage vm: 2g ram, 2 cpus, any os you like, I use alpine since it >> takes very little >> memory and its NFS server is fast. >> >> See vdsm README for instructions how to setup a host: >> https://github.com/oVirt/vdsm#manual-installation >> >> For engine host you can follow: >> https://ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engine_us… >> >> And after that this should work: >> >> dnf install ovirt-engine >> engine-setup >> >> Accepting all the defaults should work. >> >> When you have engine running, you can add a new host with >> the ip address or dns name of you host(s) vm, and engine will >> do everything for you. Note that you must install the ovirt-release-master >> rpm on the host before you add it to engine. >> >> Nir >> >> > >> > Cheers, >> > Muli >> > -- >> > Muli Ben-Yehuda >> > Co-Founder and Chief Scientist @ http://www.lightbitslabs.com >> > LightOS: The Special Storage Sauce For Your Cloud >> > >> > >> > On Wed, Feb 23, 2022 at 4:16 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >> >> >> >> On Wed, Feb 23, 2022 at 2:48 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: >> >> > >> >> > So I started looking in the logs and tried to follow along with the >> >> > code, but things didn't make sense and then I saw it's ovirt 4.3 which >> >> > makes things more complicated :) >> >> > Unfortunately because GUID is sent in the metadata the volume is >> >> > treated as a vdsm managed volume[2] for the udev rule generation and >> >> > it prepends the /dev/mapper prefix to an empty string as a result. >> >> > I don't have the vdsm logs, so I am not sure where exactly this fails, >> >> > but if it's after [4] it may be possible to workaround it with a vdsm >> >> > hook >> >> > >> >> > In 4.4.6 we moved the udev rule triggering the volume mapping phase, >> >> > before starting the VM. But it could still not work because we check >> >> > the driver_volume_type in[1], and I saw it's "driver_volume_type": >> >> > "lightos" for lightbits >> >> > In theory it looks like it wouldn't take much to add support for your >> >> > driver in a future release (as it's pretty late for 4.5) >> >> >> >> Adding support for nvme/tcp in 4.3 is probably not feasible, but we will >> >> be happy to accept patches for 4.5. >> >> >> >> To debug such issues vdsm log is the best place to check. We should see >> >> the connection info passed to vdsm, and we have pretty simple code using >> >> it with os_brick to attach the device to the system and setting up the udev >> >> rule (which may need some tweaks). >> >> >> >> Nir >> >> >> >> > [1] https://github.com/oVirt/vdsm/blob/500c035903dd35180d71c97791e0ce4356fb77ad… >> >> > >> >> > (4.3) >> >> > [2] https://github.com/oVirt/vdsm/blob/b42d4a816b538e00ea4955576a5fe762367be787… >> >> > [3] https://github.com/oVirt/vdsm/blob/b42d4a816b538e00ea4955576a5fe762367be787… >> >> > [4] https://github.com/oVirt/vdsm/blob/b42d4a816b538e00ea4955576a5fe762367be787… >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Wed, Feb 23, 2022 at 12:44 PM Muli Ben-Yehuda <muli(a)lightbitslabs.com> wrote: >> >> > > >> >> > > Certainly, thanks for your help! >> >> > > I put cinderlib and engine.log here: http://www.mulix.org/misc/ovirt-logs-20220223123641.tar.gz >> >> > > If you grep for 'mulivm1' you will see for example: >> >> > > >> >> > > 2022-02-22 04:31:04,473-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-10) [36d8a122] Command 'HotPlugDiskVDSCommand(HostName = client1, HotPlugDiskVDSParameters:{hostId='fc5c2860-36b1-4213-843f-10ca7b35556c', vmId='e13f73a0-8e20-4ec3-837f-aeacc082c7aa', diskId='d1e1286b-38cc-4d56-9d4e-f331ffbe830f', addressMap='[bus=0, controller=0, unit=2, type=drive, target=0]'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = Failed to bind /dev/mapper/ on to /var/run/libvirt/qemu/21-mulivm1.mapper.: Not a directory, code = 45 >> >> > > >> >> > > Please let me know what other information will be useful and I will prove. >> >> > > >> >> > > Cheers, >> >> > > Muli >> >> > > >> >> > > On Wed, Feb 23, 2022 at 11:14 AM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: >> >> > >> >> >> > >> Hi, >> >> > >> >> >> > >> We haven't tested this, and we do not have any code to handle nvme/tcp >> >> > >> drivers, only iscsi and rbd. Given the path seen in the logs >> >> > >> '/dev/mapper', it looks like it might require code changes to support >> >> > >> this. >> >> > >> Can you share cinderlib[1] and engine logs to see what is returned by >> >> > >> the driver? I may be able to estimate what would be required (it's >> >> > >> possible that it would be enough to just change the handling of the >> >> > >> path in the engine) >> >> > >> >> >> > >> [1] /var/log/ovirt-engine/cinderlib/cinderlib//log >> >> > >> >> >> > >> On Wed, Feb 23, 2022 at 10:54 AM <muli(a)lightbitslabs.com> wrote: >> >> > >> > >> >> > >> > Hi everyone, >> >> > >> > >> >> > >> > We are trying to set up ovirt (4.3.10 at the moment, customer preference) to use Lightbits (https://www.lightbitslabs.com) storage via our openstack cinder driver with cinderlib. The cinderlib and cinder driver bits are working fine but when ovirt tries to attach the device to a VM we get the following error: >> >> > >> > >> >> > >> > libvirt: error : cannot create file '/var/run/libvirt/qemu/18-mulivm1.dev/mapper/': Is a directory >> >> > >> > >> >> > >> > We get the same error regardless of whether I try to run the VM or try to attach the device while it is running. The error appears to come from vdsm which passes /dev/mapper as the prefered device? >> >> > >> > >> >> > >> > 2022-02-22 09:50:11,848-0500 INFO (vm/3ae7dcf4) [vdsm.api] FINISH appropriateDevice return={'path': '/dev/mapper/', 'truesize': '53687091200', 'apparentsize': '53687091200'} from=internal, task_id=77f40c4e-733d-4d82-b418-aaeb6b912d39 (api:54) >> >> > >> > 2022-02-22 09:50:11,849-0500 INFO (vm/3ae7dcf4) [vds] prepared volume path: /dev/mapper/ (clientIF:510) >> >> > >> > >> >> > >> > Suggestions for how to debug this further? Is this a known issue? Did anyone get nvme/tcp storage working with ovirt and/or vdsm? >> >> > >> > >> >> > >> > Thanks, >> >> > >> > Muli >> >> > >> > >> >> > >> > _______________________________________________ >> >> > >> > Users mailing list -- users(a)ovirt.org >> >> > >> > To unsubscribe send an email to users-leave(a)ovirt.org >> >> > >> > Privacy Statement: https://www.ovirt.org/privacy-policy.html >> >> > >> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> >> > >> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/I3PAG5HMBHUOJ… >> >> > >> >> >> > > >> >> > > Lightbits Labs >> >> > > Lead the cloud-native data center transformation by delivering scalable and efficient software defined storage that is easy to consume. >> >> > > >> >> > > This message is sent in confidence for the addressee only. It may contain legally privileged information. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality, advise the sender immediately of any error in transmission and delete the email from their systems. >> >> > > >> >> > > >> >> > _______________________________________________ >> >> > Users mailing list -- users(a)ovirt.org >> >> > To unsubscribe send an email to users-leave(a)ovirt.org >> >> > Privacy Statement: https://www.ovirt.org/privacy-policy.html >> >> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> >> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DKFOCYQA6E4N3… >> >> >> > >> > Lightbits Labs >> > Lead the cloud-native data center transformation by delivering scalable and efficient software defined storage that is easy to consume. >> > >> > This message is sent in confidence for the addressee only. It may contain legally privileged information. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality, advise the sender immediately of any error in transmission and delete the email from their systems. >> > >> > >> > > Lightbits Labs > Lead the cloud-native data center transformation by delivering scalable and efficient software defined storage that is easy to consume. > > This message is sent in confidence for the addressee only. It may contain legally privileged information. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality, advise the sender immediately of any error in transmission and delete the email from their systems. > >

6 25

[ANN] Schedule for oVirt 4.5.0
by Sandro Bonazzola 25 Feb '22

25 Feb '22

The oVirt development team leads are pleased to inform that the schedule for oVirt 4.5.0 has been finalized. The key dates follows: * Feature Freeze - String Freeze - Alpha release: 2022-03-15 * Alpha release test day: 2022-03-17 * Code freeze - Beta release: 2022-03-29 * Beta release test day: 2022-03-31 * General Availability release: 2022-04-12 A release management draft page has been created at: https://www.ovirt.org/release/4.5.0/ If you're willing to help testing the release during the test days please join the oVirt development mailing list at https://lists.ovirt.org/archives/list/devel@ovirt.org/ and report your feedback there. Instructions for installing oVirt 4.5.0 Alpha and oVirt 4.5.0 Beta for testing will be added to the release page https://www.ovirt.org/release/4.5.0/ when the corresponding version will be released. Professional Services, Integrators and Backup vendors: please plan a test session against your additional services, integrated solutions, downstream rebuilds, backup solution accordingly. If you're not listed here: https://ovirt.org/community/user-stories/users-and-providers.html consider adding your company there. If you're willing to help updating the localization for oVirt 4.5.0 please follow https://ovirt.org/develop/localization.html If you're willing to help promoting the oVirt 4.5.0 release you can submit your banner proposals for the oVirt home page and for the social media advertising at https://github.com/oVirt/ovirt-site/issues As an alternative please consider submitting a case study as in https://ovirt.org/community/user-stories/user-stories.html Feature owners: please start planning a presentation of your feature for oVirt Youtube channel: https://www.youtube.com/c/ovirtproject Do you want to contribute to getting ready for this release? Read more about oVirt community at https://ovirt.org/community/ and join the oVirt developers https://ovirt.org/develop/ Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo(a)redhat.com Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.

2 5

Re: [ovirt-users] Re: [=EXTERNAL=] Re: help using nvme/tcp storage with cinderlib and Managed Block Storage
by Benny Zlotnik 24 Feb '22

24 Feb '22

Switching to +devel Browsing the code a bit, I think a good starting point would be https://github.com/oVirt/vdsm/blob/500c035903dd35180d71c97791e0ce4356fb77ad… - if volume_type not in ("rbd", "iscsi"): + if volume_type not in ("rbd", "iscsi", "lightos"): And it's possible that it would be all that's required to make it work on ovirt >=4.5 https://github.com/oVirt/vdsm/blob/500c035903dd35180d71c97791e0ce4356fb77ad… The path resolver might warn, but it looks like it would return /dev/nvme0n so the generated udev rule should come out correct I'm happy to assist if it turns out to be more challenging (and if not) :)

2 1

[Call to Action] oVirt 4.5 localization updated in zanata
by Sandro Bonazzola 24 Feb '22

24 Feb '22

Hi, https://zanata.ovirt.org/ has been updated with new text to be translated. If you want to help with this, please follow https://ovirt.org/develop/localization.html The ovirt-web-ui and ovirt-engine-ui-extensions projects lack Italian, Czech and Turkish translation. The oVirt engine project needs some touch on all the translations but especially on Italian, Czech, Russian and Turkish Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo(a)redhat.com Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.

1 0

engine-setup failed on absent external_truststore
by Shmuel Melamud 23 Feb '22

23 Feb '22

Hi! Tried to reinstall the current master in a clean directory. engine-setup failed with the following error: [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Creating CA: /home/smelamud/engine-root/master/etc/pki/ovirt-engine/ca.pem [ INFO ] Creating CA: /home/smelamud/engine-root/master/etc/pki/ovirt-engine/qemu-ca.pem [ INFO ] Updating OVN SSL configuration [ INFO ] Updating OVN timeout configuration [ ERROR ] Failed to execute stage 'Misc configuration': [Errno 2] No such file or directory: '/home/smelamud/engine-root/master/var/lib/ovirt-engine/external_truststore' How can this be solved? Shmuel

2 4

Vdsm: Failing network tests
by Milan Zamazal 17 Feb '22

17 Feb '22

Hi, for example here: https://github.com/oVirt/vdsm/runs/5220340052?check_suite_focus=true Could you please look at it and fix it or disable the given test quickly so that we can merge patches? Thanks, Milan

2 2

finish github migration
by Michal Skrivanek 09 Feb '22

09 Feb '22

Hi, we're about to finish migrating all the projects to GitHub. If you're still missing anything, let us know....otherwise we plan to switch off completely the gerrit replication this week Thanks, michal

1 1

hosted engine deployment (v4.4.10) - TASK Check engine VM health - fatal FAILED
by Charles Stellen 08 Feb '22

08 Feb '22

Dear Ovirt Hackers, (CZ: pokud to pomuze - muzeme i cesky) we are dealing with hosted engine deployment issue on a fresh AMD EPYC servers: and we are ready to donate hardware to Ovirt community after we pass this issue ( :-) ) 0/ base infra: - 3 identical physical servers (produced in 2021-4Q) - fresh, clean and recent version of centos 8 stream installed (@^minimal-environment) - servers are interconnected with cisco switch, each other are network visible, all with nice internet access (NAT) 1/ storage: - all 3 servers/nodes host nice and clean glusterfs (v9.5) and volume "vol-images01" is ready for VM images - ovirt hosted engine deployment procedure: - easily accept mentioned glusterfs storage domain - mount it during "hosted-engine --deploy" with no issue - all permissions are set correctly at all glustrfs nodes ("chown vdsm.kvm vol-images01") - no issue with storage domain at all 2/ ovirt - hosted engine deployment: - all 3 servers successfully deployed recent ovirt version with standart procedure (on top of minimal install of centos 8 stream): dnf -y install ovirt-host virt-host-validate: PASS ALL - at first server we continue with: dnf -y install ovirt-engine-appliance hosted-engine --deploy (pure commandline - so no cockpit is used) DEPLOYMENT ISSUE: - during "hosted-engine --deploy" procedure - hosted engine becomes temporairly accessible at:https://server01:6900/ovirt-engine/ - with request to manualy set "ovirtmgmt" virtual nic - Hosts > server01 > Network Interfaces > [SETUP HOST NETWORKS] "ovirtmgmt" dropped to eno1 - [OK] - than All pass fine - and host "server01" becomes Active - back to commandline to Continue with deployment "Pause execution until /tmp/ansible.jksf4_n2_he_setup_lock is removed" by removing the lock file - deployment than pass all steps_until_ "[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check engine VM health]" ISSUE DETAILS: new VM becomes not accessible in the final stage - as it should be reachable at its final IP: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fail if Engine IP is different from engine's he_fqdn resolved IP] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM IP address is while the engine's he_fqdn ovirt-engine.mgmt.pss.local resolves to 10.210.1.101. If you are using DHCP, check your DHCP reservation configuration"} - problem is, that even if we go with "Static" IP (provided during answering procedure) or with "DHCP" way (with properly set DHCP and DNS server responding with correct IP for both WE STUCK THERE WE TRYIED: - no success to connect to terminal/vnc of running VM "HostedEngine" to figure out the internal network issue any suggestion howto "connect" into newly deployed UP and RUNNING HostedEngine VM? to figure out eventually manualy fix the internal network issue? Thank You all for your help Charles Stellen PS: we are advanced in Ovirt deployment (from version 4.0), also we are advanced in GNU/Linux KVM based virtualisation for 10+ years, so any suggests or any details requested - WE ARE READY to provide online debuging or direct access to servers is not a problem PPS: after we pass this deployment - and after decomissioning procedure - we are ready to provide older HW to Ovirt community

1 1