- Devel - oVirt List Archives

[VDSM] Flaky protocol detector test
by Nir Soffer 17 Feb '19

17 Feb '19

Recently this test cause too many random failures. I guess this is the known issue with handling SUBSCRIBE in protocol detector. I posted https://gerrit.ovirt.org/c/97099/ To mark it as broken until we have time to dig deeper ====================================================================== FAIL: test_detect_slow_client_concurrency(True) (protocoldetector_test.AcceptorTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/protocoldetector_test.py", line 167, in test_detect_slow_client_concurrency self.check_concurrently(self.check_slow_client, use_ssl) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/protocoldetector_test.py", line 266, in check_concurrently self.assertTrue(all(done)) AssertionError: False is not true -------------------- >> begin captured logging << -------------------- 2019-01-19 20:47:16,021 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Creating socket (host='127.0.0.1', port=0, family=2, socketype=1, proto=6) (protocoldetector:225) 2019-01-19 20:47:16,022 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at 127.0.0.1:46009 (protocoldetector:183) 2019-01-19 20:47:16,023 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Adding detector <protocoldetector_test.Echo object at 0x7f1bb77f8dd0> (protocoldetector:210) 2019-01-19 20:47:16,024 DEBUG (MainThread) [vds.MultiProtocolAcceptor] Adding detector <protocoldetector_test.Uppercase object at 0x7f1bb77f8c90> (protocoldetector:210) 2019-01-19 20:47:16,031 INFO (Thread-61) [ProtocolDetector.AcceptorImpl] Accepted connection from 127.0.0.1:51950 (protocoldetector:61) 2019-01-19 20:47:16,047 INFO (Thread-61) [ProtocolDetector.AcceptorImpl] Accepted connection from 127.0.0.1:51952 (protocoldetector:61) 2019-01-19 20:47:16,048 INFO (Thread-61) [ProtocolDetector.AcceptorImpl] Accepted connection from 127.0.0.1:51954 (protocoldetector:61) 2019-01-19 20:47:16,065 INFO (Thread-61) [ProtocolDetector.AcceptorImpl] Accepted connection from 127.0.0.1:51956 (protocoldetector:61) 2019-01-19 20:47:16,068 INFO (Thread-61) [ProtocolDetector.AcceptorImpl] Accepted connection from 127.0.0.1:51958 (protocoldetector:61) 2019-01-19 20:47:16,069 DEBUG (Thread-61) [ProtocolDetector.Detector] Using required_size=9 (protocoldetector:89) 2019-01-19 20:47:16,097 DEBUG (Thread-61) [ProtocolDetector.Detector] Using required_size=9 (protocoldetector:89) 2019-01-19 20:47:16,099 DEBUG (Thread-61) [ProtocolDetector.Detector] Using required_size=9 (protocoldetector:89) 2019-01-19 20:47:16,107 DEBUG (Thread-61) [ProtocolDetector.Detector] Using required_size=9 (protocoldetector:89) 2019-01-19 20:47:16,109 DEBUG (Thread-61) [ProtocolDetector.Detector] Using required_size=9 (protocoldetector:89) 2019-01-19 20:47:16,574 INFO (Thread-61) [ProtocolDetector.Detector] Detected protocol echo from 127.0.0.1:51950 (protocoldetector:125) 2019-01-19 20:47:17,570 DEBUG (Thread-61) [ProtocolDetector.Detector] Timed out while waiting for data (protocoldetector:94) 2019-01-19 20:47:17,571 DEBUG (Thread-61) [ProtocolDetector.Detector] Timed out while waiting for data (protocoldetector:94) 2019-01-19 20:47:17,576 DEBUG (Thread-61) [ProtocolDetector.Detector] Timed out while waiting for data (protocoldetector:94) 2019-01-19 20:47:17,578 DEBUG (Thread-61) [ProtocolDetector.Detector] Timed out while waiting for data (protocoldetector:94) --------------------- >> end captured logging << ---------------------

2 7

GitHub now has draft PRs
by Greg Sheremeta 17 Feb '19

17 Feb '19

GitHub now has draft PRs! https://github.blog/2019-02-14-introducing-draft-pull-requests/ For those of us using github instead of gerrit, please submit draft PRs early :) Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

2 1

[ OST Failure Report ] [ oVirt Master/4.2 (ovirt-vmconsole) ] [ 15-02-2019 ] [ TEST NAME ]
by Dafna Ron 16 Feb '19

16 Feb '19

Hi, We are failing to deploy hosts in upgrade suites on both master and 4.2 for project ovirt-vmconsole. it seems we are missing packages for selinux-policy. Root cause identified by CQ as: https://gerrit.ovirt.org/#/c/97704/ - spec: clean up and reorganize can you please take a look at this issue? Sandro requested a bug so I opened one: https://bugzilla.redhat.com/show_bug.cgi?id=1677630 Error: 019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] 2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod method['method']() File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages self.processTransaction() File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction if self._miniyum.buildTransaction(): File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction raise yum.Errors.YumBaseError(msg) YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] 2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] Thanks, Dafna

3 8

[Ovirt] [CQ weekly status] [15-02-2019]
by Dafna Ron 15 Feb '19

15 Feb '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: RED (#1) 1. We had 4.2 failure due to selinux-policy package not downloaded automatically by ost as a dependency to vmconsoule package. The package stopped downloading after this change in vconsole: https://gerrit.ovirt.org/#/c/97704/ - clean up and reorganize I reported the issue to the list after discussing it with Sandro and after a discussion with the developer Sandro approved merging a patch to CI that would manually add the package: https://gerrit.ovirt.org/#/c/97785/ I am re-running the project's last patch now. - Action item for Galit to check why OST is not detecting the package if the project deps are ok. *CQ-Master:* RED (#1) 1. The same failure in vmconsole was causing upgrade suite to fail on master. Current running jobs for 4.2 [1] and master [2] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

[VDSM] Running the new storage tests on your laptop
by Nir Soffer 15 Feb '19

15 Feb '19

I want to share our new block storage tests, running on your laptop, from your editor, creating real block storage domain with real logical volumes. One catch, these tests require root - there is no way to create devices without root. To make it easy to run as root only the tests that need root, they are marked with "root" mark. Here an example of running the root tests for block storage domain: $ sudo ~/.local/bin/tox -e storage-py27 tests/storage/blocksd_test.py -- -m root And for lvm: $ sudo ~/.local/bin/tox -e storage-py27 tests/storage/lvm_test.py -- -m root To run all storage tests that require root: $ sudo ~/.local/bin/tox -e storage-py27 -- -m root tests/storage Another issue - after running the tests as root, you need to fix ownership of some files in .tox/, tests/htmlcov*, and /var/tmp/vdsm. You can: $ sudo chown -R $USER:$USER .tox tests /var/tmp/vdsm We will improve this later. Note that I'm running tox installed as user: $ pip install --user tox This gets most recent tox with minimal breakage of system python. With the new tests, our code coverage is now 57%: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/2888/artifact/check… - blockSD: 47% https://jenkins.ovirt.org/job/vdsm_standard-check-patch/2888/artifact/check… - lvm: 74% https://jenkins.ovirt.org/job/vdsm_standard-check-patch/2888/artifact/check… These tests are rather slow, all the root tests take 26 seconds. But OST takes more then 40 minutes, and cover less code in this area. OST coverage for lvm mudle: 71% https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4071/artifact/expor… To get debug the tests, you can use new option in recent pytest --log-cli-level: $ sudo ~/.local/bin/tox -e storage-py27 tests/storage/blocksd_test.py -- -m root --log-cli-level=info Here is an example output from a test creating a storage domain (use --log-cli-leve=debug if this is not verbose enough) ------------------------------------------------------------------------------- live log call ------------------------------------------------------------------------------- blockSD.py 1034 INFO sdUUID=d4d7649d-4849-4413-bdc2-b7b84f239092 domainName=loop-domain domClass=1 vgUUID=3OJX6U-UDLc-VFtg-2cRO-q3kR-UH2g-Nvf78I storageType=3 version=3, block_size=512, alignment=1048576 blockSD.py 600 INFO size 512 MB (metaratio 262144) lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=metadata, size=512m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - metadata blockSD.py 522 INFO Create: SORT MAPPING: ['/dev/loop2'] lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=inbox, size=16m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - inbox lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=outbox, size=16m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - outbox lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=ids, size=8m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - ids lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=leases, size=2048m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - leases lvm.py 1168 INFO Creating LV (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, lv=master, size=1024m, activate=True, contiguous=False, initialTags=(), device=None) lvm.py 1198 WARNING Could not change ownership of one or more volumes in vg (d4d7649d-4849-4413-bdc2-b7b84f239092) - master lvm.py 1333 INFO Deactivating lvs: vg=d4d7649d-4849-4413-bdc2-b7b84f239092 lvs=['master'] blockdev.py 84 INFO Zeroing device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/metadata (size=41943040) utils.py 454 INFO Zero device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/metadata: 0.00 seconds blockdev.py 84 INFO Zeroing device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/inbox (size=1024000) utils.py 454 INFO Zero device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/inbox: 0.02 seconds blockdev.py 84 INFO Zeroing device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/outbox (size=1024000) utils.py 454 INFO Zero device /dev/d4d7649d-4849-4413-bdc2-b7b84f239092/outbox: 0.02 seconds lvm.py 1438 INFO Changing VG tags (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, delTags=[], addTags=['MDT_LEASETIMESEC=60', 'MDT_IOOPTIMEOUTSEC=10', 'MDT_LEASERETRIES=3', 'MDT_LOCKRENEWALINTERVALSEC=5', 'MDT_SDUUID=d4d7649d-4849-4413-bdc2-b7b84f239092', 'MDT_ROLE=Regular', 'MDT_POOL_UUID=', 'MDT_PV0=pv:loop2&44&uuid:wzupcF-uQME-3PIa-4WNf-THJJ-57MS-3UIdxD&44&pestart:0&44&pecount:157&44&mapoffset:0', 'MDT_CLASS=Data', 'MDT__SHA_CKSUM=ee58868dee52c4cc128f0ee89a0c382de1fe6419', 'MDT_LOGBLKSIZE=512', 'MDT_VGUUID=3OJX6U-UDLc-VFtg-2cRO-q3kR-UH2g-Nvf78I', 'MDT_PHYBLKSIZE=512', 'MDT_DESCRIPTION=loop-domain', 'MDT_TYPE=ISCSI', 'MDT_VERSION=3', 'MDT_LOCKPOLICY=']) lvm.py 1438 INFO Changing VG tags (vg=d4d7649d-4849-4413-bdc2-b7b84f239092, delTags=['RHAT_storage_domain_UNREADY'], addTags=['RHAT_storage_domain']) lvm.py 1325 INFO Activating lvs: vg=d4d7649d-4849-4413-bdc2-b7b84f239092 lvs=['master'] PASSED [ 50%] Nir

2 2

Reviews
by Germano Veit Michel 15 Feb '19

15 Feb '19

Hello, I've pinged this a few times before, sorry for doing it again. If there is interest to incorporate these changes in ovirt to make troubleshooting LSM/Snapshots easier please review these changes, otherwise we can abandon them. https://gerrit.ovirt.org/#/q/topic:snapshot-tools https://gerrit.ovirt.org/#/q/topic:dump-chains-sqlite Thanks,

2 1

Failed dependencies Ovirt 4.3 on clean install Centos 7.6
by Erick Perez 15 Feb '19

15 Feb '19

Centos 7.6 (minimal install ISO) UEFI boot yum -y update reboot [root@ovirt01] yum install cockpit-ovirt-dashboard Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (ovirt-4.3-centos-ovirt43) Requires: librte_mbuf.so.3()(64bit) Available: dpdk-17.11-13.el7.x86_64 (extras) librte_mbuf.so.3()(64bit) Available: dpdk-17.11-15.el7.x86_64 (extras) librte_mbuf.so.3()(64bit) Installed:dpdk-18.11-2.el7_6.x86_64 (@extras) ~librte_mbuf.so.4()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest [root@ovirt01] Note: The above message appears for librte_mbuf/mempool/pmd and several others. It seems openvswitch specifically need DPDK v17 and not v18. Please clarify. thanks,

3 3

oVirt Glance repo not accessible blocked OST
by Tian Xu 15 Feb '19

15 Feb '19

Hi Experts, I am trying to run ovirt-system-test in my local lab, which need proxy to access internet. ovirt cannot work with proxy to access ovirt glance repository, whose URL is: http://glance.ovirt.org:9292, a related bugzilla was filed by someone and in open status: https://bugzilla.redhat.com/show_bug.cgi?id=1362433 Some ovirt system tests depend on ovirt glance repository for VM images, these tests either skipped or failed, then subsequent VM related tests cannot be run. Here is the test file: https://github.com/oVirt/ovirt-system-tests/blob/master/basic-suite-4.2/tes…, and the verify_glance_import failed, then other test blocked too. Below is exception message and details test log see attachment. Running test scenario 004_basic_sanity.py <?xml version="1.0" encoding="UTF-8"?> <testsuite name="nosetests" tests="3" errors="1" failures="0" skip="1"> <testcase classname="004_basic_sanity" name="add_blank_vms" time="2.473"/> <testcase classname="004_basic_sanity" name="verify_glance_import" time="0.050"> <skipped type="unittest.case.SkipTest" message="Glance is not available"><![CDATA[SkipTest: Glance is not available ]]></skipped> </testcase> <testcase classname="004_basic_sanity" name="add_vm1_from_template" time="0.266"> <error type="exceptions.IndexError" message="list index out of range"><![CDATA[Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/mnt/zz/bak/ovirt-system-tests-master/basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 748, in add_vm1_from_template )[0] IndexError: list index out of range ]]></error> </testcase> </testsuite> Thanks， Xu

2 3

Fwd: [ovirt-users] Ovirt Cluster completely unstable
by Sandro Bonazzola 14 Feb '19

14 Feb '19

Any suggestion from Gluster team on how to get back to a stable system in a very short loop? opened https://bugzilla.redhat.com/show_bug.cgi?id=1677160 to track this on gluster side. ---------- Forwarded message --------- From: <dscott(a)umbctraining.com> Date: gio 14 feb 2019 alle ore 00:26 Subject: [ovirt-users] Ovirt Cluster completely unstable To: <users(a)ovirt.org> I'm abandoning my production ovirt cluster due to instability. I have a 7 host cluster running about 300 vms and have been for over a year. It has become unstable over the past three days. I have random hosts both, compute and storage disconnecting. AND many vms disconnecting and becoming unusable. 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts running 3.12.5. I submitted a bugzilla bug and they immediately assigned it to the storage people but have not responded with any meaningful information. I have submitted several logs. I have found some discussion on problems with instability with gluster 3.12.5. I would be willing to upgrade my gluster to a more stable version if that's the culprit. I installed gluster using the ovirt gui and this is the version the ovirt gui installed. Is there an ovirt health monitor available? Where should I be looking to get a resolution the problems I'm facing. _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IEXC… -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

disk cache issues
by Hetz Ben Hamo 14 Feb '19

14 Feb '19

Hi, After digging around and finding a bit of info about viodiskcache - I understand that if the user enable it - then the VM cannot be live migrated. Umm, unless the op decides to do a live migration including changing storage - I don't understand why the live migration is disabled. If the VM will only be live migrated between nodes, then the storage is the same, nothing is saved locally on the node's hard disk, so what is the reason to disable live migration? Thanks, Hetz

3 2