- Devel - oVirt List Archives

[CQ Failure Report] [oVirt Master - ovirt-engine] [16/9/18] [basic-suite.el7.x86_64]
by Ehud Yonasi 17 Sep '18

17 Sep '18

Hey everyone, We have a failure in CQ on the test of verify_add_hosts. Link to error: https://pastebin.com/Bwqn19Pn Link to the CQ job: http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change…

2 1

vdsm has been tagged (v4.20.40)
by Milan Zamazal 17 Sep '18

17 Sep '18

Vdsm 4.20.40 for oVirt 4.2.7 rc1

1 0

ovirt-imageio for s390x - seems to need 4k support
by Nir Soffer 14 Sep '18

14 Sep '18

I posted a patch[1] to add s390x build for ovirt-imageio, but the build reveal that direct I/O using 512 bytes alignment does not work on the test slave (or maybe this is a platform issue?). See failing tests: https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/414/consol… *00:05:34.273* [check-patch.fc28.s390x] func = <built-in method readinto of _io.FileIO object at 0x3ff850a2048>*00:05:34.273* [check-patch.fc28.s390x] args = (<mmap.mmap object at 0x3ff85238618>,)*00:05:34.273* [check-patch.fc28.s390x] e = IOError(22, 'Invalid argument') Increasing the default block size fixes these tests, but cause other test to fail, because the code is not ready for 4k yet: https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/415/consol… So adding s390x build requires more work. I open a bug[2] for this work. We should have 4k support in imageio in 4.3, but I'm not sure we will have time to work on this before that, unless someone else wants to contribute :-) [1] https://gerrit.ovirt.org/c/94095/ [2] https://bugzilla.redhat.com/1624963 Nir

2 1

[ACTION REQUIRED] ovirt-engine-nodejs update
by Greg Sheremeta 13 Sep '18

13 Sep '18

Hi, If you have a project that uses ovirt-engine-nodejs to build, would you please take a moment to test that it builds with the latest 8.x, 8.11.4? https://jenkins.ovirt.org/job/ovirt-engine-nodejs_master_check-patch-el7-x8… [We needed to update it for ovirt-web-ui because the latest patternfly requires node >= 8.9.] Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 3

issue in vdsm monitoring
by Dafna Ron 13 Sep '18

13 Sep '18

Hi, I have been seeing random failures of tests in different projects caused by vdsm monitoring. I need someone from vdsm to please help debug this issue. From what I can see, the test suspend/resume vm failed because we could not query the status of the vm on the host. you can see full log from failed tests here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10208/artifa… Here are the errors that I can see in the vdsm which seem to suggest that there is an issue getting stats on the vm which was suspended and caused the failure of the test: 2018-09-11 00:40:05,896-0400 INFO (monitor/c1fe6e6) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:734) 2018-09-11 00:40:07,957-0400 DEBUG (qgapoller/1) [vds] Not sending QEMU-GA command 'guest-get-users' to vm_id='8214433a-f233-4aaa-aeda-2ce1d31c78dc', command is not supported (qemuguestagent:192) 2018-09-11 00:40:08,068-0400 DEBUG (periodic/3) [virt.sampling.VMBulkstatsMonitor] sampled timestamp 4296118.49 elapsed 0.010 acquired True domains all (sampling:443) 2018-09-11 00:40:08,271-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] Calling 'Image.prepare' in bridge with {u'allowIllegal': True, u'storagepoolID': u'e80a56d9-74da-498a-b010-4a9df287f11d', u'imageID': u'd4c831e6-02d2-4d89-b516-0ec4597 5e024', u'volumeID': u'15b07af1-625b-42e3-b62a-8e7c7a120a56', u'storagedomainID': u'f1744940-41b6-4d35-b7bf-870c4e07d995'} (__init__:329) 2018-09-11 00:40:10,846-0400 DEBUG (vmchannels) [virt.vm] (vmId='8214433a-f233-4aaa-aeda-2ce1d31c78dc') Guest connection timed out (guestagent:556) 2018-09-11 00:40:11,637-0400 DEBUG (jsonrpc/5) [jsonrpc.JsonRpcServer] Calling 'Host.getStats' in bridge with {} (__init__:329) 2018-09-11 00:40:11,637-0400 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.201.4,49184 (api:47) 2018-09-11 00:40:11,643-0400 DEBUG (jsonrpc/5) [root] cannot read eth0 speed (nic:42) 2018-09-11 00:40:11,645-0400 DEBUG (jsonrpc/5) [root] cannot read eth1 speed (nic:42) 2018-09-11 00:40:11,647-0400 DEBUG (jsonrpc/5) [root] cannot read eth2 speed (nic:42) 2018-09-11 00:40:11,649-0400 DEBUG (jsonrpc/5) [root] cannot read eth3 speed (nic:42) 2018-09-11 00:40:11,667-0400 INFO (jsonrpc/5) [api.host] FINISH getStats error=[Errno 22] Invalid argument from=::ffff:192.168.201.4,49184 (api:51) 2018-09-11 00:40:11,667-0400 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1407, in getStats multipath=True)} File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 49, in get_stats decStats = stats.produce(first_sample, last_sample) File "/usr/lib/python2.7/site-packages/vdsm/host/stats.py", line 71, in produce stats.update(get_interfaces_stats()) File "/usr/lib/python2.7/site-packages/vdsm/host/stats.py", line 153, in get_interfaces_stats return net_api.network_stats() File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 63, in network_stats return netstats.report() File "/usr/lib/python2.7/site-packages/vdsm/network/netstats.py", line 31, in report stats = link_stats.report() File "/usr/lib/python2.7/site-packages/vdsm/network/link/stats.py", line 41, in report speed = vlan.speed(i.device) File "/usr/lib/python2.7/site-packages/vdsm/network/link/vlan.py", line 36, in speed dev_speed = nic.read_speed_using_sysfs(dev_name) File "/usr/lib/python2.7/site-packages/vdsm/network/link/nic.py", line 48, in read_speed_using_sysfs s = int(f.read()) IOError: [Errno 22] Invalid argument 2018-09-11 00:40:11,669-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats failed (error -32603) in 0.03 seconds (__init__:312) 2018-09-11 00:40:14,656-0400 DEBUG (jsonrpc/2) [jsonrpc.JsonRpcServer] Calling 'Host.getAllVmStats' in bridge with {} (__init__:329) 2018-09-11 00:40:14,657-0400 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,46772 (api:47) 2018-09-11 00:40:14,659-0400 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,46772 (api:53) 2018-09-11 00:40:14,660-0400 DEBUG (jsonrpc/2) [jsonrpc.JsonRpcServer] Return 'Host.getAllVmStats' in bridge with (suppressed) (__init__:356) 2018-09-11 00:40:14,661-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2018-09-11 00:40:14,674-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] Calling 'Host.getAllVmIoTunePolicies' in bridge with {} (__init__:329) 2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [api.host] START getAllVmIoTunePolicies() from=::1,46772 (api:47) 2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'message': 'Done', 'code': 0}, 'io_tune_policies_dict': {'8214433a-f233-4aaa-aeda-2ce1d31c78dc': {'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/blockSD/f1744940-41b6-4d35-b7bf-870c4e07d995/images/9a3659dc-339e-4b3f-a1d1-6ee3debb6168/e2570635-a798-4e9e-a0ea-88436d5b296f', 'name': 'vda'}]}}} from=::1,46772 (api:53) 2018-09-11 00:40:14,675-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] Return 'Host.getAllVmIoTunePolicies' in bridge with {'8214433a-f233-4aaa-aeda-2ce1d31c78dc': {'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/blockSD/f1744940-41b6-4d35-b7bf-870c4e07d995/images/9a3659dc-339e-4b3f-a1d1-6ee3debb6168/e2570635-a798-4e9e-a0ea-88436d5b296f', 'name': 'vda'}]}} (__init__:356) 2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:312) 2018-09-11 00:40:16,118-0400 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] Calling 'Host.getAllVmStats' in bridge with {} (__init__:329) 2018-09-11 00:40:16,119-0400 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.201.4,49184 (api:47) 2018-09-11 00:40:16,121-0400 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.201.4,49184 (api:53) 2018-09-11 00:40:16,122-0400 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] Return 'Host.getAllVmStats' in bridge with (suppressed) (__init__:356) 2018-09-11 00:40:16,122-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2018-09-11 00:40:17,956-0400 DEBUG (qgapoller/2) [vds] Not sending QEMU-GA command 'guest-get-users' to vm_id='8214433a-f233-4aaa-aeda-2ce1d31c78dc', command is not supported (qemuguestagent:192) 2018-09-11 00:40:18,025-0400 DEBUG (periodic/3) [virt.periodic] Looking for stale paused VMs (periodic:388) I can also see this in Super vdsm which may be related but I an not 100% sure its related to current issue: MainProcess|jsonrpc/2::DEBUG::2018-09-11 00:33:20,012::cmdutils::159::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 ifup/on039d2d88d8ee4::DEBUG::2018-09-11 00:33:20,238::cmdutils::159::root::(exec_cmd) FAILED: <err> = 'Running scope as unit 2f1f417d-85b9-46c0-b7d7-6082836daaae.scope.\n/etc/sysconfig/network-scripts/ifup-eth: line 321: 20715 Terminated /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nDevice "on039d2d88d8ee4" does not exist.\nCannot find device "on039d2d88d8ee4"\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d 8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevic e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d 8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevic e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d 8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevic e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\n'; <rc> = 1 ifup/on039d2d88d8ee4::ERROR::2018-09-11 00:33:20,238::concurrent::202::root::(run) FINISH thread <Thread(ifup/on039d2d88d8ee4, stopped daemon 140281550731008)> failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 948, in _exec_ifup _exec_ifup_by_name(iface.name, cgroup) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 934, in _exec_ifup_by_name raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '') ConfigNetworkError: (29, '\n') thanks, Dafna

3 12

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 11-09-2018 ] [ 002_bootstrap.add_cluster ]
by Dafna Ron 13 Sep '18

13 Sep '18

Hi, we have failures on both upgrade and basic suites for master. The patch reported as cause is: https://gerrit.ovirt.org/#/c/93345/10 - engine : Add finer grained monitoring thresholds for memory consumption on Hypervisors Ravi, can you please check this issue? You can see the logs here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10187/artifa… Here is the error: 2018-09-11 02:08:44,511-04 WARN [org.ovirt.engine.core.bll.AddClusterCommand] (default task-2) [1362d97c-9d55-40d2-9b03-ebe93ae2fe67] Validation of action 'AddCluster' failed for user admin@internal-authz. Reasons: VAR__TYPE__CLUSTER,VAR __ACTION__CREATE,must be greater than or equal to 1,$groups [Ljava.lang.Class;@29a040fc,$message {javax.validation.constraints.Min.message},$payload [Ljava.lang.Class;@64094510,$value 1,ACTION_TYPE_FAILED_ATTRIBUTE_PATH,$path cluster.logM axMemoryUsedThreshold,$validatedValue 0 2018-09-11 02:08:44,511-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [1362d97c-9d55-40d2-9b03-ebe93ae2fe67] method: runAction, params: [AddCluster, ManagementNetworkOnClusterOperationPara meters:{commandId='e5eb6cf6-4e3d-4f50-be6a-9fe18bfc4d97', user='null', commandType='Unknown'}], timeElapsed: 56ms 2018-09-11 02:08:44,516-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [must be greater than or equal to 1, Attribute: cluster.logMaxMemoryUsedThreshold] 2018-09-11 02:08:44,533-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-2) [] Entered SsoRestApiAuthFilter Thanks, Dafna

3 12

Issue in host deploy
by Ravi Shankar Nori 12 Sep '18

12 Sep '18

Hi, Host deploy is failing on master with errors. The issue seems to be that the opcode that is returned from vdsm has Unicode null characters in it (obtained using a break point in VdsDeploy.userVisibleLog). "Failed to execute stage 'Initializing': Invalid response opcode '\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000CONFIRM'" Resulting in below errors when the message is inserted into database. Can someone take a look 2018-09-11 13:38:11,121-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [669a0081] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host 192.168.122.57. Stage: Initializing. 2018-09-11 13:38:39,231-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [669a0081] transaction rolled back 2018-09-11 13:38:39,231-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [669a0081] Error during deploy dialog 2018-09-11 13:38:39,246-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-11) [669a0081] Error during host 192.168.122.57 install 2018-09-11 13:38:41,791-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-11) [669a0081] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host 192.168.122.57: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00.

2 3

[ OST Failure Report ] [ oVirt Master (ovirt-ansible-manageiq) ] [ 11-09-2018 ] [ metrics_bootstrap.metrics_and_log_collector ]
by Dafna Ron 11 Sep '18

11 Sep '18

Hi, we have a failure in basic suite for test metrics_bootstrap.metrics_and_log_collector. We failed to configure the hosts for metrics collection which causes failure in the test. The change that is reported as cause of failure is: https://github.com/oVirt/ovirt-ansible-manageiq/commit/ 6697fc98f934beb9fece6d0224c0b013411b84e6 You can see the full logs here: https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3078/artifact/b… error: PLAY RECAP ********************************************************************* lago-basic-suite-4-2-host-0 : ok=1 changed=0 unreachable=0 failed=1 lago-basic-suite-4-2-host-1 : ok=1 changed=0 unreachable=0 failed=1 localhost : ok=53 changed=24 unreachable=0 failed=0 2018-09-11 09:06:18,713::ssh.py::ssh::96::lago.ssh::DEBUG::Command d69e71be on lago-basic-suite-4-2-engine errors: [WARNING]: Could not match supplied host pattern, ignoring: metrics_store 2018-09-11 09:06:18,713::utils.py::_ret_via_queue::60::lago.utils::ERROR::Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/003_00_metrics_bootstrap.py", line 54, in configure_metrics ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) AssertionError: Configuring ovirt machines for metrics failed. Exit code is 2 2018-09-11 09:06:38,897::ssh.py::ssh::81::lago.ssh::DEBUG::Command d648c70a on lago-basic-suite-4-2-engine returned with 0 2018-09-11 09:06:38,897::ssh.py::ssh::89::lago.ssh::DEBUG::Command d648c70a on lago-basic-suite-4-2-engine output: This command will collect system configuration and diagnostic information from this system.

2 1

failure in ost test - Invalid argument - help in debugging issue
by Dafna Ron 11 Sep '18

11 Sep '18

Hi, can someone please have a look at this ost failure? it is not related to the change that failed and I think its probably a race. you can find the logs here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10175/artifa… The error I can see is this: https://pastebin.com/pm6x0W62 Thanks, Dafna

3 2

vdsm has been tagged (v4.20.39.1)
by Francesco Romani 07 Sep '18

07 Sep '18

Vdsm 4.20.39.1 for oVirt 4.2.6 (async) sub-branch ovirt-4.2.6 created -- Francesco Romani (@fromanirh)

1 0