- Devel - oVirt List Archives

Silent errors in storage code on Vdsm startup
by Milan Zamazal 06 Feb '19

06 Feb '19

Hi, I'm trying to run Vdsm with Python 3, which exposes various Python 3 incompatibilities in Vdsm code. I'm currently semi-stuck on Vdsm crashing very shortly after startup with: Panic: Error initializing IRS Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/vdsmd.py", line 95, in serve_clients irs = Dispatcher(HSM()) File "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 46, in __init__ self._exposeFunctions(obj) File "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 56, in _exposeFunctions if hasattr(funcObj, _EXPORTED_ATTRIBUTE) and callable(funcObj): File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 85, in __getattr__ raise se.StoragePoolNotConnected vdsm.storage.exception.StoragePoolNotConnected: Storage pool not connected: () That wouldn't be a problem by itself, errors are expected in Python 3 experiments. However the problem is that this is often the *only* error I receive. I've found out that HSM() call spawns threads, which apparently don't have chance to report their errors before the panic above kills Vdsm. For instance, there was a string-byte mismatch in running iscsiadm scanning commands. Only after I added time.sleep to the end of HSM.__init__, I got an error + traceback from that and could see what's wrong. That worries me a bit, because it might mean that if some initialization external commands are not fast enough, Vdsm may not be able to start. After fixing some problems, I get the panic again, even with the sleep at the end of HSM.__init__, and again without any other error. It's difficult to proceed in such a situation -- any tips what to look at or to look for are welcome. FWIW, my vdsm.log now ends with INFO (MainThread) [storage.check] Starting check service (check:91) DEBUG (check/loop) [root] START thread <Thread(check/loop, started daemon 140180434425600)> (func=<bound method EventLoop.run_forever of <EventLoop running=True closed=False at 0x140181096264704>>, args=(), kwargs={}) (concurrent:193) INFO (check/loop) [storage.asyncevent] Starting <EventLoop running=True closed=False at 0x140181096264704> (asyncevent:125) DEBUG (hsm/init) [storage.SamplingMethod] Returning last result (misc:386) DEBUG (hsm/init) [storage.SamplingMethod] Trying to enter sampling method (vdsm.storage.hba.<function rescan at 0x7f7e7a597e18>) (misc:376) DEBUG (hsm/init) [storage.SamplingMethod] Got in to sampling method (misc:379) DEBUG (hsm/init) [storage.HBA] Starting scan (hba:58) DEBUG (hsm/init) [storage.HBA] Scan finished (hba:64) DEBUG (hsm/init) [storage.SamplingMethod] Returning last result (misc:386) DEBUG (hsm/init) [root] /usr/bin/taskset --cpu-list 0-0 /sbin/udevadm settle --timeout=5 (cwd None) (commands:200) DEBUG (hsm/init) [root] SUCCESS: <err> = b''; <rc> = 0 (commands:221) DEBUG (hsm/init) [storage.SamplingMethod] Returning last result (misc:386) DEBUG (hsm/init) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 /usr/bin/sudo -n /sbin/lvm pvs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size,mda_used_count (cwd None) (commands:200) DEBUG (hsm/init) [storage.Misc.excCmd] SUCCESS: <err> = b''; <rc> = 0 (commands:221) DEBUG (hsm/init) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 /usr/bin/sudo -n /sbin/lvm vgs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) (commands:200) DEBUG (hsm/init) [storage.Misc.excCmd] SUCCESS: <err> = b''; <rc> = 0 (commands:221) DEBUG (hsm/init) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 /usr/bin/sudo -n /sbin/lvm lvs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags (cwd None) (commands:200) DEBUG (hsm/init) [storage.Misc.excCmd] SUCCESS: <err> = b''; <rc> = 0 (commands:221) DEBUG (hsm/init) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 /usr/bin/sudo -n /sbin/lvm vgs --config 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) (commands:200) DEBUG (hsm/init) [storage.Misc.excCmd] SUCCESS: <err> = b''; <rc> = 0 (commands:221) INFO (hsm/init) [storage.HSM] FINISH HSM init succeeded in 0.43 seconds (hsm:410) DEBUG (hsm/init) [storage.HSM] FINISH thread <Thread(hsm/init, stopped daemon 140180442818304)> (concurrent:196) and is completely silent after that. Thanks, Milan

2 4

package versioning problem causing failure on CQ ovirt-master on all projects
by Dafna Ron 06 Feb '19

06 Feb '19

Hi, Please note that ovirt-ansible-hosted-engine-setup has a versioning problem with the package and is causing bootstrap to fail for upgrade suite [1] This is effecting all projects, its been reported to the developers and should be fixed as soon as possible. you can view CQ status here: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… [1] http://pastebin.test.redhat.com/708086 Thanks, Dafna

6 16

Plans for ovirt-guest-agent
by Sandro Bonazzola 05 Feb '19

05 Feb '19

Hi, I understood that most of the ovirt-guest-agent logic moved to qemu-guest-agent. We are currently shipping ovirt-guest-agent for Fedora, EL7 and Windows, all of them using Python 2. Most of the bugs around ovirt-guest-agent have been closed wontfix or deferred with comments like "The guest agent is going away in 4.3 in favor of the qemu guest agent, so this will not be resolved" but we still have the ovirt guest agent around in oVirt 4.3.0. What's the plan for 4.3 updates and 4.4? Is it safe to not install ovirt-guest-agent but just qemu-guest-agent? Wich version of qemu-guest-agent is required to completely replace ovirt-guest-agent? Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

Proposing Yedidyah Bar David (didi) as a integration maintainer for oVirt Engine
by Sandro Bonazzola 05 Feb '19

05 Feb '19

Hi, Didi is maintaining OTOPI and wrote with me most of the code in engine-setup back into oVirt 3.3 release. He maintained that code across the following releases along with the backup and restore tool and the rename tool. He's the most active contributor within integration team for ovirt-engine related code. I would like to propose Didi as integration maintainer for oVirt Engine. Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

5 5

propose Ravi Nori as a frontend maintainer for ovirt-engine infra
by Greg Sheremeta 05 Feb '19

05 Feb '19

Hi all, I'd like to propose Ravi Nori as a frontend maintainer for ovirt-engine infra team. While the infra team typically hasn't had a frontend maintainer, Ravi has strong knowledge of all of the frontend infra pieces, including everything related to SSO, welcome page, and GWT in admin portal. Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

5 5

[ANN] oVirt 4.3.0 is now generally available
by Sandro Bonazzola 04 Feb '19

04 Feb '19

The oVirt Project is excited to announce the general availability of oVirt 4.3.0, as of February 4th, 2019 This release unleashes an altogether more powerful and flexible open source virtualization solution that encompasses over four hundreds individual changes and a wide range of enhancements across the engine, storage, network, user interface, and analytics on top of oVirt 4.2 series. What's new in oVirt 4.3.0? * Q35 chipset, support booting using UEFI and Secure Boot * Skylake-server and AMD EPYC support * New smbus driver in windows guest tools * Improved support for v2v * OVA export / import of Templates * Full support for live migration of High Performance VMs * Microsoft Failover clustering support (SCSI Persistent Reservation) for Direct LUN disks * Hundreds of bug fixes on top of oVirt 4.2 series * New VM portal details page (see a preview here: https://imgur.com/a/ExINpci) * New Cluster upgrade UI * OVN security groups * IPv6 (static host addresses) * Support of Neutron from RDO OpenStack 13 as external network provider * Support of using Skydive from RDO OpenStack 14 as Tech Preview * Support for 3.6 and 4.0 data centers, clusters and hosts were removed * Now using PostgreSQL 10 * New metrics support using rsyslog instead of fluentd This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. If you’re managing more than one oVirt instance, OpenShift Origin or RDO we also recommend to try ManageIQ <http://manageiq.org/>. In such case, please ensure to take the qc2 image and not the ova image. Notes: - oVirt Appliance is already available for CentOS 7 and Fedora 28 - oVirt Node NG is already available for CentOS 7 and Fedora 28 [2] - oVirt Windows Guest Tools iso is already available [2] Additional Resources: * Read more about the oVirt 4.3.0 release highlights: http://www.ovirt.org/release/4.3.0/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.0/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

OST fails with LagoDeployError: ... failed with status 1 on lago-basic-suite-master-host-0
by Nir Soffer 04 Feb '19

04 Feb '19

I had 2 OST failure today with the same infrastructure failure, before running any test: + install_cmd='yum install --nogpgcheck --downloaddir=/dev/shm -y' + yum install --nogpgcheck --downloaddir=/dev/shm -y ntp net-snmp ovirt-engine ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*' otopi-debug-plugins cronie Error: Package: 7:device-mapper-event-1.02.149-10.el7_6.3.x86_64 (alocalsync) Requires: device-mapper = 7:1.02.149-10.el7_6.3 Installed: 7:device-mapper-1.02.149-10.el7_6.2.x86_64 (installed) device-mapper = 7:1.02.149-10.el7_6.2 ret=1 + echo 'install failed with status 1' + exit 1 - STDERR Links to failed builds: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_man… - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_man… The last successful build was at Feb 1: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4004/ The trend show that all builds failed since Feb 3 9:02 am. https://jenkins.ovirt.org/job/ovirt-system-tests_manual/buildTimeTrend This issue is blocking 4.3 work on 4k support. Nir

2 3

Re: [rhev-devel] u/s errors during update
by Roni Eliezer 03 Feb '19

03 Feb '19

Adding devel(a)ovirt.org Thx Roni On Sun, Feb 3, 2019 at 2:31 PM Eyal Edri <eedri(a)redhat.com> wrote: > you might want to ask on devel(a)ovirt.org and not rhev-devel which is > downstream and might be less monitored. > > On Sun, Feb 3, 2019 at 11:10 AM Roni Eliezer <reliezer(a)redhat.com> wrote: > >> Hi >> I got the following errors (see below) when I try to upgrade 4.3 u/s >> Any idea? >> >> --> Finished Dependency Resolution >> Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 >> (@centos-ovirt43-testing) >> Requires: librte_efd.so.1()(64bit) >> Removing: dpdk-17.11-15.el7.x86_64 (@rhel-extras) >> librte_efd.so.1()(64bit) >> Updated By: dpdk-18.11-2.el7_6.x86_64 (rhel-extras) >> Not found >> Available: dpdk-17.11-7.el7.x86_64 (rhel-extras) >> librte_efd.so.1()(64bit) >> Available: dpdk-17.11-11.el7.x86_64 (rhel-extras) >> librte_efd.so.1()(64bit) >> Available: dpdk-17.11-13.el7.x86_64 (rhel-extras) >> librte_efd.so.1()(64bit) >> Available: dpdk-2.0.0-8.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-2.2.0-2.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-2.2.0-3.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-16.11.2-4.el7.x86_64 (rhel-extras) >> Not found >> Error: Package: 1:openvswitch-ovn-common-2.10.1-1.el7.x86_64 >> (@centos-ovirt43-testing) >> Requires: librte_pmd_softnic.so.1()(64bit) >> Removing: dpdk-17.11-15.el7.x86_64 (@rhel-extras) >> librte_pmd_softnic.so.1()(64bit) >> Updated By: dpdk-18.11-2.el7_6.x86_64 (rhel-extras) >> Not found >> Available: dpdk-17.11-7.el7.x86_64 (rhel-extras) >> librte_pmd_softnic.so.1()(64bit) >> Available: dpdk-17.11-11.el7.x86_64 (rhel-extras) >> librte_pmd_softnic.so.1()(64bit) >> Available: dpdk-17.11-13.el7.x86_64 (rhel-extras) >> librte_pmd_softnic.so.1()(64bit) >> Available: dpdk-2.0.0-8.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-2.2.0-2.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-2.2.0-3.el7.x86_64 (rhel-extras) >> Not found >> Available: dpdk-16.11.2-4.el7.x86_64 (rhel-extras) >> Not found >> >> > > -- > > Eyal edri > > > MANAGER > > RHV/CNV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) >

1 0

oVirt on Fedora 28 status update
by Nir Soffer 01 Feb '19

01 Feb '19

Last time we discussed this here, we had only sanlock issue: https://bugzilla.redhat.com/1593853 The bug was fixed upstream about 2 month ago, but the Fedora package was not available. The package is not available yet in Fedora, but we have a build here: https://koji.fedoraproject.org/koji/buildinfo?buildID=1182539 You can install sanlock from this build using: dnf upgrade https://kojipkgs.fedoraproject.org//packages/sanlock/3.6.0/4.fc28/x86_64/py… \ https://kojipkgs.fedoraproject.org//packages/sanlock/3.6.0/4.fc28/x86_64/sa… \ https://kojipkgs.fedoraproject.org//packages/sanlock/3.6.0/4.fc28/x86_64/sa… Hopefully the package will pushed soon to updates-testing repo. With this you can use enable selinux as god intended. But if you update your host to kernel 4.20.4-100, multipath is broken. All multipath devices are not available, and your hosts will probably become non-operational since they report the iSCSI/FC storage domains in problem. The issue was reported here: https://lkml.org/lkml/2018/11/5/398 And we have this Fedora 29 bug: https://bugzilla.redhat.com/1669235 Ben explains it is: The kernel is switching over to use block multiqueue instead of the old request queue. Part of doing this is removing support for the old request queue from device-mapper. Another part is to remove support for the old request queue from the scsi layer. For some reason, the first part got into this fedora kernel, but the second part didn't. It seems to me that since the fedora kernel has removed support for non-blk-mq based devices, I should have been compiled with CONFIG_SCSI_MQ_DEFAULT=y To fix this issue you need to add the scsi_mod.use_blk_mq=Y option to the kernel command line: grubby --args=scsi_mod.use_blk_mq=Y --update-kernel /boot/vmlinuz-4.20.4-100.fc28.x86_64 After reboot, your multipath devices will appear again. Cheers, Nir

2 3

[Ovirt] [CQ weekly status] [01-02-2019]
by Dafna Ron 01 Feb '19

01 Feb '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#2) We have no current failures. Last failed job was for ovirt-ansible-hosted-engine-setup due to build-artiacts job on Jan 25. the issue is worked on by Simon Tiraboschi *CQ-Master:* GREEN (#1) There are no current issues in master CQ During the week we had 3 regressions for ovirt-engine 1. Domain version 5 feature was merged but its was not yet supported on vdsm side: https://gerrit.ovirt.org/#/c/96178/ - storage: Added V5 domain format definition. Date failure in CQ: 24-01-2019 test failing: 002_bootstrap.add_master_storage_domain Fix: https://gerrit.ovirt.org/#/c/97262/ - storage: Make default domain verion V4 for 4.3 2. CQ was still failing on create domain v5 after the fix was merged to fix regression #1 https://gerrit.ovirt.org/#/c/96178/ - storage: Added V5 domain format definition. Date failure in CQ: 26-01-2019 test failing: 002_bootstrap.add_secondary_storage_domains Fix: https://gerrit.ovirt.org/#/c/97408/ - storage: V5 is not a latest domain version yet ** first patch fixed the domain create for manual create but via the API the version was available which is why we were still failing on CQ. 3. Added callbacks for MBS disks causing regular disks to use callbacks unnecessarily https://gerrit.ovirt.org/#/c/96815/ - core: set VmDevices properly when creating and attaching We saw this regression once #1 and #2 regressions were fixed. Date first failure detected: 30-01-2019 Fix: https://gerrit.ovirt.org/#/c/97428/ - core: partial revert of 4d439006 Ovirt-engine passed on 31-01-2019 Current running jobs for 4.2 [1] and master [2] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0