October 2018 - Devel - oVirt List Archives

Fwd: VDSM API schema inconsistencies
by Nir Soffer 10 Oct '18

10 Oct '18

Adding devel. ---------- Forwarded message --------- From: Nir Soffer <nsoffer(a)redhat.com> Date: Wed, 10 Oct 2018, 11:07 Subject: Re: VDSM API schema inconsistencies To: Piotr Kliczewski <pkliczew(a)redhat.com> Cc: Dan Kenigsberg <danken(a)redhat.com>, Edward Haas <ehaas(a)redhat.com>, Milan Zamazal <mzamazal(a)redhat.com>, Tomasz Baranski <tbaransk(a)redhat.com>, Francesco Romani <fromani(a)redhat.com>, Irit Goihman <igoihman(a)redhat.com>, Marcin Sobczyk <msobczyk(a)redhat.com> On Wed, 10 Oct 2018, 10:21 Piotr Kliczewski, <pkliczew(a)redhat.com> wrote: > +Nir > > On Wed, Oct 10, 2018 at 8:39 AM Marcin Sobczyk <msobczyk(a)redhat.com> > wrote: > >> Hi, >> I would love to get some feedback from you about it on the VDSM call >> today so if you have a moment, please take a look. >> Marcin >> >> ---------- Forwarded message --------- >> From: Marcin Sobczyk <msobczyk(a)redhat.com> >> Date: Mon, Oct 8, 2018 at 2:40 PM >> Subject: VDSM API schema inconsistencies >> To: devel <devel(a)ovirt.org> >> >> >> Hi, >> >> when I was working on BZ#1624306 it turned out, that even if I'd fix the >> help-printing code in vdsm-client, there's still a lot of inconsistencies >> in API's schema files. >> >> After some investigation, I found that there's a code for reporting >> schema inconsistencies, but it's turned off because it bloats the logs. >> > Right, an it should remain like this. In runtime we never want these warnings. There's also a "strict" mode that raises an exception each time an >> inconsistency is found, but it's also off because there's so many of them. >> > Same. Both are debugging aids for developers, to make it easy to check the schema. > I think that no one wants to maintain both: the actual code and schema >> files. >> > This is not a big issue. We rarely add or change APIs. But it is nice if we don't need to manualy make the schema correct. > My idea would be to make the schema derived from the code at build time as >> a long-term effort. >> > This cannot work, since clients depend on the schema, it should not change if when someone makes uintended change in the code. It should work in the opposite way, you change the schema, and generate new code frome schema. Threre are existing tools like grpc and thrift that work in this way. For long term we should drop our jsonrpc/stomp stack and move to grpc. For short term we can just fix the schema to reflect actual engine interaction. when engine and schema do not match, engine is usualy right. For responses, if engine can handle the response the code is right. Please do not make RPC code more complex and inefficient for improving schema issues detection. I would rather remove this code instead. Before we can do that though, we need to address the inconsistencies first. >> >> >> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:v… >> >> This is a series of patches that helps fixing schema issues. What it >> does, is it adds a bunch of information from stack to vastly ease the >> process of fixing them. The logging level is also changed from 'WARNING' to >> 'DEBUG'. Here's an example of a logged entry: >> >> Host.setKsmTune >> With message: Required property pages_to_scan is not provided when >> calling Host.setKsmTune >> With backtrace: [ >> [ >> "_verify_object_type", >> { >> "call_arg_keys":[ >> "merge_across_nodes", >> "run" >> ], >> "schema_type_name":"KsmTuneParams" >> } >> ], >> [ >> "_verify_complex_type", >> { >> "schema_type_type":"object" >> } >> ] >> ] >> >> To make it work, a patch for OST's 'basic-master-suite' is on the way >> that switches 'schema.inconsistency' logger's logging level do 'DEBUG'. >> That way, we can find all reported errors in 'vdsm-schema-error.log' files. >> >> An initial report that groups reported errors by namespaces/methods has >> also been made. Please note, that an implementation that completely avoids >> error duplicates is really hard and IMHO not worth the effort. You can find >> the results here: >> >> - Host: https://pastebin.com/YM2XnvTV >> - Image: https://pastebin.com/GRc1yErL >> - LVMVolumeGroup: https://pastebin.com/R1276gTu >> - SDM: https://pastebin.com/P3tfYEDD >> - StorageDomain: https://pastebin.com/Q0DxKgF5 >> - StoragePool: https://pastebin.com/VXFWpfRC >> - VM: https://pastebin.com/tDC60c29 >> - Volume: https://pastebin.com/TYkr9Zsd >> - |jobs|: https://pastebin.com/jeXpYyz9 >> - |virt|: https://pastebin.com/nRREuEub >> >> File a bug per vertical (infra, storage, virt) for these issues. Since we don't know about real user issues with these apis this must be incorrect schema. Nir

2 2

[VDSM] Do not use commands.execCmd(cmd, sync=False) or AsyncProcessOperation
by Nir Soffer 10 Oct '18

10 Oct '18

I just found that this patch was merged last month: https://gerrit.ovirt.org/c/94269/ Adding another use of commands.execCmd(cmd, sync=False) and AsyncProcessOperation. These APIs are deprecated and will be removed soon, please do merge any code using them. Also running commands anachronistically with a timeout is a bad idea in general. If the command is stuck on storage, we will created huge amount of commands. We have the same bad pattern in storage (iscsi/fcp rescan), it should be replaced by running command synchronically (sync=False) in a dedicated thread, and never running more then one command if the command get stuck. Marcin is working on the new improved infrastructure for running commands, based on https://gerrit.ovirt.org/c/83923. Nir

1 0

[ OST Failure Report ] [ oVirt master (vdsm) ] [ 09-10-2018 ] [ 005_network_by_label ]
by Daniel Belenky 09 Oct '18

09 Oct '18

Hi, The following patch is suspected to be failing OST: https://gerrit.ovirt.org/#/c/94754/2 Error snippet from vdsm log: 2018-10-09 08:51:50,799-0400 ERROR (Thread-7) [root] Shutdown by QEMU Guest Agent failed (vm:5269) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5260, in qemuGuestAgentShutdown self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2482, in shutdownFlags if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self) libvirtError: Guest agent is not responding: QEMU guest agent is not connected -- DANIEL BELENKY

2 1

BUG - Cannot install Host with empty password.
by Shawn Weeks 04 Oct '18

04 Oct '18

Is there a current open bug for https://bugzilla.redhat.com/show_bug.cgi?id=1057575. It's still happening in the current release but it should have been fixed back in 2016. Guide me fails with this error but add host works fine. Thanks Shawn

1 0

Adding Fedora 28 host fail with "NotImplementedError: Packager install not implemented"
by Nir Soffer 04 Oct '18

04 Oct '18

Trying to add Fedora 28 host to engine master, host installation fails immediately with: 2018-10-02 20:34:45,440+0300 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventStart STAGE internal_packages METHOD otopi.plugins.ovirt_host_deploy.vd sm.vdsmid.Plugin._packages (None) 2018-10-02 20:34:45,443+0300 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-qh2D5sbRmG/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-qh2D5sbRmG/otopi-plugins/ovirt-host-deploy/vdsm/vdsmid.py", line 84, in _packages self.packager.install(('dmidecode',)) File "/tmp/ovirt-qh2D5sbRmG/pythonlib/otopi/packager.py", line 102, in install raise NotImplementedError(_('Packager install not implemented')) NotImplementedError: Packager install not implemented 2018-10-02 20:34:45,444+0300 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Environment packages setup': Packager install not implemented engine version: 4.3.0_master (efabde5a36cbf85b43a60ead37e42943c35c2741) Is this a known issue? any workaround? Nir

5 8

random failures on test 004_basic_sanity.check_snapshot_with_memory
by Dafna Ron 04 Oct '18

04 Oct '18

Hi, I have been noticing random failures for test 004_basic_sanity.check_snapshot_with_memory The failure is not connected directly to any of the failed changes and its either a problem with the test or a problem in vdsm which is causing the failure of this test randomly. the error we are getting is: Fault reason is "Operation Failed". Fault detail is "[Cannot run VM. Low disk space on Storage Domain iscsi.]". HTTP response code is 409. I think that its related the sparsify failure we have seen in the past [1] but it has been happening more often now. Tal, can someone in your team review this again? I think that if this is not resolved soon I will skip the test so that it does not fail changes for no reason. You can see failed build here: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3277/ You can see the logs for the failed test here: https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3277/artifact/b… [1] https://ovirt-jira.atlassian.net/browse/OVIRT-2024 Thanks, Dafna

1 0

vdsm has been tagged (v4.20.42)
by Milan Zamazal 03 Oct '18

03 Oct '18

Vdsm 4.20.42 for oVirt 4.2.7 rc2

1 0

Re: Issue with vdsm-hook-vmfex-dev = 4.20.40-11.git6d99cc8.el7 shipping on u/s 4.2
by Eyal Edri 03 Oct '18

03 Oct '18

Adding devel and infra as well. On Wed, Oct 3, 2018 at 3:10 PM Michael Burman <mburman(a)redhat.com> wrote: > Hi > > I think we have an issue with vdsm-hook-vmfex-dev on 4.2 u/s > > When i try to update i get this Dependency Resolution > > Error: Package: vdsm-4.20.40-11.git6d99cc8.el7.x86_64 (ovirt-4.2-snapshot) > Requires: vdsm-hook-vmfex-dev = 4.20.40-11.git6d99cc8.el7 > Installed: vdsm-hook-vmfex-dev-4.20.40-10.giteffb1ef.el7.noarch > (@ovirt-4.2-snapshot) > vdsm-hook-vmfex-dev = 4.20.40-10.giteffb1ef.el7 > Available: vdsm-hook-vmfex-dev-4.20.23-1.el7.noarch > (ovirt-4.2-centos-ovirt42) > vdsm-hook-vmfex-dev = 4.20.23-1.el7 > Available: vdsm-hook-vmfex-dev-4.20.39.1-1.el7.noarch > (ovirt-4.2-centos-ovirt42) > vdsm-hook-vmfex-dev = 4.20.39.1-1.el7 > Error: Package: vdsm-hook-vmfex-dev-4.20.40-10.giteffb1ef.el7.noarch > (@ovirt-4.2-snapshot) > Requires: vdsm = 4.20.40-10.giteffb1ef.el7 > Removing: vdsm-4.20.40-10.giteffb1ef.el7.x86_64 > (@ovirt-4.2-snapshot) > vdsm = 4.20.40-10.giteffb1ef.el7 > Updated By: vdsm-4.20.40-11.git6d99cc8.el7.x86_64 > (ovirt-4.2-snapshot) > vdsm = 4.20.40-11.git6d99cc8.el7 > Available: vdsm-4.20.23-1.el7.x86_64 (ovirt-4.2-centos-ovirt42) > vdsm = 4.20.23-1.el7 > Available: vdsm-4.20.39.1-1.el7.x86_64 > (ovirt-4.2-centos-ovirt42) > vdsm = 4.20.39.1-1.el7 > > > https://resources.ovirt.org/pub/ovirt-4.2-snapshot/rpm/el7Server/noarch/ > > Is missing the correct version and this is the problem > > Thanks, > -- > > Michael Burman > > Senior Quality engineer - rhv network - redhat israel > > Red Hat > > <https://www.redhat.com> > > mburman(a)redhat.com M: 0545355725 IM: mburman > <https://red.ht/sig> > -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

3 3

[ENGINE] make install-dev fails, trying to install outside of ENGINE_PREFIX
by Nir Soffer 02 Oct '18

02 Oct '18

Trying to make install-dev with engine master: $ git clean -dxf $ git log | head -1 commit efabde5a36cbf85b43a60ead37e42943c35c2741 The build failes, trying to install stuff out of ENGINE_PREFIX: $ make install-dev ENGINE_PREFIX=/home/nsoffer/ovirt-engine ... make copy-recursive SOURCEDIR=packaging/sys-etc TARGETDIR="/usr/local/etc"... make[2]: Entering directory `/home/nsoffer/src/ovirt-engine' ( cd "packaging/sys-etc" && find . -type d -printf '%P\n' ) | while read d; do \ install -d -m 755 "/usr/local/etc/${d}"; \ done install: cannot change permissions of ‘/usr/local/etc/logrotate.d’: No such file or directory Is this know issue? any workaround? Thanks, Nir

1 1

increasingly common OST failure: Cannot setup Networks. Another Setup Networks or Host Refresh process in progress on the host. Please try later.
by Dan Kenigsberg 02 Oct '18

02 Oct '18

FYI, Ales is working on retrying setupNetwork in case there is an async getCapabilities going on the host. https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-q… However, the real bug is partially hidden by Lago complaining about ExtractPathNoPathError: scp: /root/fluentd: No such file or directory What is it? Is it already tracked by someone?

4 4