- Devel - oVirt List Archives

GlusterFS rebase for oVirt 4.3
by Sandro Bonazzola 28 Nov '18

28 Nov '18

According to Sahina we dropped dependency on gluster-gnfs and it's safe to move on from the unsupported 3.12 version to a newer one 4.1 / 5.0. This is for getting an agreement on what we should require in oVirt 4.3: CentOS Storage SIG provides bot 4.1 and 5 for x86_64[1] and ppc64le[2] If I understood correctly, 5 is shipping glusterd2 which requires a significant effort to get support for while 4.1 is still on glusterd which should work with current oVirt code. Sahina, can you give directions on what we should move to? Thanks. [1] http://mirror.centos.org/centos/7/storage/x86_64/ [2] http://mirror.centos.org/altarch/7/storage/ppc64le/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

4 11

error when running VDSM build
by Nir Levy 28 Nov '18

28 Nov '18

*I am installing over * *CentOS Linux release 7.5.1804* make pylint which runs: tox -e pylint static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc failed for: Problem importing module base.py: cannot import name get_node_last_lineno *./.tox/pylint/lib/python2.7/site-packages/pylint/checkers/base.py:48*: from pylint.checkers.utils import get_node_last_lineno which is located at: *./.tox/pylint-py3k/lib/python2.7/site-packages/pylint/checkers/utils.py:912:* *def get_node_last_lineno(node):* what can cause this error? Regards, *Nir.* *-----------------* *full output:* *-----------------* tox -e pylint \ static/usr/share/vdsm/sitecustomize.py \ lib/vdsm \ lib/vdsmclient \ lib/yajsonrpc \ pylint recreate: /home/root/vdsm/.tox/pylint pylint installdeps: pylint==1.8 pylint installed: astroid==1.6.5,backports.functools-lru-cache==1.5,backports.ssl-match-hostname==3.5.0.1,blivet==0.61.15.69,cffi==1.6.0,chardet==2.2.1,configobj==4.7.2,configparser==3.5.0,configshell-fb==1.1.23,coverage==4.4.1,cryptography==1.7.2,decorator==3.4.0,dnspython==1.12.0,enum34==1.1.6,ethtool==0.8,filelock==3.0.10,funcsigs==1.0.2,futures==3.2.0,gssapi==1.2.0,idna==2.4,iniparse==0.4,ioprocess==1.2.0,ipaclient==4.5.4,ipaddress==1.0.16,ipalib==4.5.4,ipaplatform==4.5.4,ipapython==4.5.4,IPy==0.75,isort==4.3.4,jwcrypto==0.4.2,kitchen==1.1.1,kmod==0.1,langtable==0.0.31,lazy-object-proxy==1.3.1,libvirt-python==3.9.0,Magic-file-extensions==0.2,mccabe==0.6.1,mock==2.0.0,netaddr==0.7.19,netifaces==0.10.4,ovirt-imageio-common==1.5.0,pbr==3.1.1,perf==0.1,pluggy==0.8.0,ply==3.4,policycoreutils-default-encoding==0.1,pthreading==0.1.3,py==1.7.0,pyasn1==0.1.9,pyasn1-modules==0.0.8,pycparser==2.14,pycurl==7.19.0,pygobject==3.22.0,pygpgme==0.3,pyinotify==0.9.4,pykickstart==1.99.66.18,pyliblzma==0.5.3,pylint==1.8.0,pyOpenSSL==0.13.1,pyparsing==1.5.6,pyparted==3.9,python-augeas==0.5.0,python-dateutil==2.4.2,python-ldap==2.4.15,python-linux-procfs==0.4.9,python-nss==0.16.0,python-yubico==1.2.3,pyudev==0.15,pyusb==1.0.0b1,pyxattr==0.5.1,PyYAML==3.10,qrcode==5.0.1,requests==2.6.0,rtslib-fb==2.1.63,sanlock-python==3.6.0,schedutils==0.4,seobject==0.1,sepolicy==1.1,singledispatch==3.4.0.3,six==1.10.0,slip==0.4.0,slip.dbus==0.4.0,SSSDConfig==1.16.0,subprocess32==3.2.6,targetcli-fb===2.1.fb46,toml==0.10.0,tox==2.9.1,urlgrabber==3.10,urllib3==1.10.2,urwid==1.1.1,virtualenv==16.1.0,WebOb==1.2.3,wrapt==1.10.11,yum-langpacks==0.4.2,yum-metadata-parser==1.1.4 pylint runtests: PYTHONHASHSEED='2893585131' pylint runtests: commands[0] | pylint -j0 --reports=no --score=no static/usr/share/vdsm/sitecustomize.py lib/vdsm lib/vdsmclient lib/yajsonrpc Problem importing module base.py: cannot import name get_node_last_lineno Problem importing module base.pyc: cannot import name get_node_last_lineno Problem importing module base.py: cannot import name get_node_last_lineno Problem importing module base.py: cannot import name get_node_last_lineno Problem importing module base.py: cannot import name get_node_last_lineno

3 3

[VDSM] Travis Fedora 29 first build results
by Nir Soffer 28 Nov '18

28 Nov '18

We have patches[1] for Fedora 29 build on Travis[2]. Here are initial results: - storage tests passed - virt tests passed - python 3.7 compile test failed (async is a keyword now) - python 3.6 tests skipped (need to add py37 env) - lib tests failed - network test failed - legacy nose tests failed [1] https://gerrit.ovirt.org/q/topic:dockerfile+is:open [2] https://travis-ci.org/nirs/vdsm/jobs/459969129 Nir

2 2

Hosts in Unassigned state after upgrading libvirt to 4.9
by Nir Soffer 28 Nov '18

28 Nov '18

I updated 2 Fedora 28 hosts today, getting new ovirt-master-release.rpm, which exposes new virt-preview repo providing libvirt 4.9 and qemu 3.1. After the update, connecting with engine master (built few week ago) fail with: 2018-11-26 22:07:51,702+02 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "[Errno 2] No such file or directory: '/usr/share/libvirt/cpu_map.xml'"}] Looks like contents of /usr/share/libvirt/ is different now: $ ls -1 /usr/share/libvirt/cpu_map/*.xml | head /usr/share/libvirt/cpu_map/index.xml /usr/share/libvirt/cpu_map/ppc64_POWER6.xml /usr/share/libvirt/cpu_map/ppc64_POWER7.xml /usr/share/libvirt/cpu_map/ppc64_POWER8.xml /usr/share/libvirt/cpu_map/ppc64_POWER9.xml /usr/share/libvirt/cpu_map/ppc64_POWERPC_e5500.xml /usr/share/libvirt/cpu_map/ppc64_POWERPC_e6500.xml /usr/share/libvirt/cpu_map/ppc64_vendors.xml /usr/share/libvirt/cpu_map/x86_486.xml /usr/share/libvirt/cpu_map/x86_athlon.xml Do we have a fix for this? Nir

3 7

[VDSM] New: warnings in build
by Nir Soffer 28 Nov '18

28 Nov '18

Since we moved to pytest 4.0, we have warnings summary at the end of the build. Here is example summary: *00:10:06.166* =============================== warnings summary ===============================*00:10:06.167* /home/jenkins/workspace/vdsm_master_check-patch-fc28-x86_64/vdsm/lib/vdsm/storage/mailbox.py:73*00:10:06.168* /home/jenkins/workspace/vdsm_master_check-patch-fc28-x86_64/vdsm/lib/vdsm/storage/mailbox.py:73: DeprecationWarning: fromstring() is deprecated. Use frombytes() instead.*00:10:06.171* _zeroCheck = checksum(EMPTYMAILBOX, CHECKSUM_BYTES) ... You can see complete warnings here: https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2413/cons… We have only few warnings, but they are repeated a lot. Nir

1 0

mismatch in disk size while uploading a disk in chunks using Image Transfer
by Ranjit DSouza 27 Nov '18

27 Nov '18

Hi I am trying to upload a snapshot disk in chunks. Everything seems to work fine, but observed that the actual_size after upload, is much lesser than the actual_size of the original disk. Here are the steps: 1. Take a snapshot of a vm disk and download it (using Image Transfer mechanism). Save it on the file system somewhere. This disk name is 3gbdisk. It is Raw + sparse. Resides on nfs storage. The size of this downloaded file is 3 GB. "actual_size" : "1389109248", //1 GB "alias" : "3gbdisk", "content_type" : "data", "format" : "raw", "image_id" : "8fbac55e-0c86-4c0b-911b-f5b0a6722834", "propagate_errors" : "false", "provisioned_size" : "3221225472", "shareable" : "false", "sparse" : "true", "status" : "ok", "storage_type" : "image", "total_size" : "0", "wipe_after_delete" : "false", 2. Now create a new floating disk, (raw + sparse), with provisioned_size = 3221225472, or 3 GB. This disk name is vmRestoreDisk 3. Upload to this disk using Image Transfer API, using libCurl in chunks of 128 MB. This is done in a while loop, sequentially reading portions of the file downloaded in step 1 and uploading these chunks via libcurl. I Use the Transfer URL, not proxy URL. Here is the trace of the first chunk. Note the Content-Range and Content-Length headers. Start offset = 0, end offset = 134217727 (or 128 MB) upload request for chunk, start offset: 0, end offset: 134217727 Upload Started Header:Content-Range: bytes 0-134217727/3221225472 Header:Content-Length: 3221225472 * Trying 10.210.46.215... * TCP_NODELAY set * Connected to pnm86hpch30bl15.pne.ven.veritas.com (10.210.46.215) port 54322 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: O=pne.ven.veritas.com; CN=pnm86hpch30bl15.pne.ven.veritas.com * start date: Oct 7 08:55:24 2018 GMT * expire date: Oct 7 08:55:24 2023 GMT * issuer: C=US; O=pne.ven.veritas.com; CN=pravauto20.pne.ven.veritas.com.59289 * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway. > PUT /images/8ebc9fa8-d322-423e-8a14-5e46ca10ed4e HTTP/1.1 Host: pnm86hpch30bl15.pne.ven.veritas.com:54322 Accept: */* Content-Length: 134217728 Expect: 100-continue * Done waiting for 100-continue * We are completely uploaded and fine * HTTP 1.0, assume close after body < HTTP/1.0 200 OK < Date: Fri, 23 Nov 2018 11:52:53 GMT < Server: WSGIServer/0.1 Python/2.7.5 < Content-Type: application/json; charset=UTF-8 < Content-Length: 0 < * Closing connection 0 http response code from curl 200 Upload Finished. Return Value: 0 4. Finalize the Image Transfer after all chunks are uploaded. Observed that the disk status goes from 'uploading via API' to finalizing to OK. 5. Do a GET call on the disk (vmRestoreDisk). "actual_size" : "134217728", //128MB "alias" : "vmRestoreDisk", "content_type" : "data", "format" : "raw", "image_id" : "3eda3df2-514a-4e78-b999-1729216b25db", "propagate_errors" : "false", "provisioned_size" : "3221225472", "shareable" : "false", "sparse" : "true", "status" : "ok", "storage_type" : "image", "total_size" : "0", "wipe_after_delete" : "false", As you can see, the actual size is just 128 MB, not 1 GB. I have attached the logs of the upload operation. I think I may be missing something, let me know in case you need further information. Thanks Ranjit

3 6

ovirt-engine has been tagged (ovirt-engine-4.2.8)
by Tal Nisan 27 Nov '18

27 Nov '18

1 0

Re: splitting tests for stdci v2
by Nir Soffer 27 Nov '18

27 Nov '18

On Mon, Nov 26, 2018 at 3:00 PM Marcin Sobczyk <msobczyk(a)redhat.com> wrote: > Hi, > > I'm currently working on paralleling our stdci v2. > > I've already extracted 'linters' stage, more patches (and more > substages) are on the way. > > This part i.e. : > > if git diff-tree --no-commit-id --name-only -r HEAD | egrep --quiet > 'vdsm.spec.in|Makefile.am|automation' ; then > ./automation/build-artifacts.sh" > ... > > seems to be an excellent candidate for extraction to a separate substage. > > The question is - how should we proceed with tests? I can create > substage for each of: > > tox -e "tests,{storage,lib,network,virt}" > > But the original 'check-patch' combined the coverage reports into one - > we would lose that. > My long term goal is to get rid of all the ugly bash code in the makefile, and run everything via tox, but as first step I think we can split the work by running: make tests In the tests substage, instead of "make check" today. Does it change anything about coverage? Theoretically we can split also to storage/network/virt/infra jobs but I think this will consume too many resources and harm other projects sharing the slaves. > There is a possibility that we could work on something that gathers > coverage data from multiple sources (tests, OST) as a completely > separate jenkins job or smth, but that will be a bigger effort. > What do you think about it? > > Marcin > > > > >

3 3

OST: Warning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
by Yedidyah Bar David 27 Nov '18

27 Nov '18

Hi all, I noticed this warning in OST's lago log, e.g. [1]: 2018-11-27 07:32:44,684::log_utils.py::start_log_task::655::nose::INFO:: # detach_vm_network_from_host_0: [0m [0m 2018-11-27 07:32:50,360::network_utils_v4.py::<genexpr>::119::py.warnings::WARNING:: * Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal 2018-11-27 07:32:50,360::log_utils.py::end_log_task::670::nose::INFO:: # detach_vm_network_from_host_0: [32mSuccess [0m (in 0:00:05) This seems to be a result of OST:basic-suite-master/test_utils/network_utils_v4.py:get_network_attachment , which does: network = next(net for net in dc.networks_service().list() if net.name == network_name) Apparently, dc.networks_service() returns a str, not unicode. Is this normal? Expected? Should the caller (OST) convert to unicode if needed, or the SDK? If so, [2] should fix (pushed just now, check-patch should finish in an hour or so, only then we can know for sure). Adding Ondra for considering the SDK side. For reference: to debug this on my own OST machine, I did this (after running the basic suite): python import ovirtsdk4 as sdk4 engine = sdk4.Connection(url='https://%s/ovirt-engine/api' % '192.168.201.4', username='admin@internal', password='123', insecure=True, debug=True).system_service() dc = engine.data_centers_service().data_center_service(engine.data_centers_service().list(search='name={}'.format('test-dc'))[0].id) [(net.name, type(net.name)) for net in dc.networks_service().list()] Which returned: [('ovirtmgmt', <type 'str'>), ('VM Network with a very long name and \xd7\xa2\xd7\x91\xd7\xa8\xd7\x99\xd7\xaa', <type 'str'>), ('Labeled_Network', <type 'str'>), ('Migration_Net', <type 'str'>)] Best regards, [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11697/artifa… [2] https://gerrit.ovirt.org/95768 -- Didi

1 0

oVirt impersonation and sessions
by Anastasiya Ruzhanskaya 26 Nov '18

26 Nov '18

Hello everyone! I wanted to find out how the impersonation technique used in oVirt works? I know from libvirt developers, that oVirt opens one connection only for multiple clients. How does this work? Also I found out in source code that in ActionParameterBase class the sessionId field is marked transient but, for example, for GWT rpc message, which goes to the server and says what action will be made (shut down, pause vm) this is the only field in all sent information which says what the session is. Where is the session sent instead? There was also a field with session id in https headers, but this was related to cookie so I am not completely sure if this can help to identify the current user.

2 2