- Devel - oVirt List Archives

ovirt-system-tests_hc-basic-suite-4.2 is failing (on glusterfs storage?)
by Sandro Bonazzola 06 Jun '18

06 Jun '18

https://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/235/art… In supervdsm: MainProcess|jsonrpc/3::ERROR::2018-06-05 21:12:42,094::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in vdoVolumeList Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/gluster/thinstorage.py", line 109, in vdoVolumeList for vdo, data in iteritems(vdoData["VDOs"]): File "/usr/lib/python2.7/site-packages/six.py", line 599, in iteritems return d.iteritems(**kw) AttributeError: 'str' object has no attribute 'iteritems' Following this issue, HE VM goes up and down without logging anything meaningful on libvirt side: Jun 5 21:17:04 lago-hc-basic-suite-4-2-host0 vdsm[19399]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Jun 5 21:17:19 lago-hc-basic-suite-4-2-host0 vdsm[19399]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Jun 5 21:17:20 lago-hc-basic-suite-4-2-host0 kvm: 1 guest now active Jun 5 21:17:20 lago-hc-basic-suite-4-2-host0 kvm: 0 guests now active Jun 5 21:17:20 lago-hc-basic-suite-4-2-host0 kvm: 1 guest now active Jun 5 21:17:31 lago-hc-basic-suite-4-2-host0 kvm: 0 guests now active Jun 5 21:17:33 lago-hc-basic-suite-4-2-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=virt-copy-out -a /var/tmp/localvm363phZ/images/f57df841-51ed-45e3-b888-fa43d1c185e4/23e54499-3a21-453b-a816-c51ca11b9c7f /etc/hosts "/var/tmp/localvm363phZ" removes=None creates=None chdir=None stdin=None Jun 5 21:17:33 lago-hc-basic-suite-4-2-host0 kvm: 1 guest now active Jun 5 21:17:34 lago-hc-basic-suite-4-2-host0 vdsm[19399]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Jun 5 21:17:43 lago-hc-basic-suite-4-2-host0 kvm: 0 guests now active Jun 5 21:17:45 lago-hc-basic-suite-4-2-host0 python: ansible-lineinfile Invoked with directory_mode=None force=None remote_src=None backrefs=False insertafter=None path=/var/tmp/localvm363phZ/hosts owner=None follow=False line=None group=None insertbefore=None unsafe_writes=None create=False state=absent content=NOT_LOGGING_PARAMETER serole=None setype=None dest=/var/tmp/localvm363phZ/hosts selevel=None regexp=# hosted-engine-setup-/var/tmp/localvm363phZ$ validate=None src=None seuser=None delimiter=None mode=None firstmatch=False attributes=None backup=False Jun 5 21:17:48 lago-hc-basic-suite-4-2-host0 python: ansible-command Invoked with warn=True executable=None _uses_shell=False _raw_params=virt-copy-in -a /var/tmp/localvm363phZ/images/f57df841-51ed-45e3-b888-fa43d1c185e4/23e54499-3a21-453b-a816-c51ca11b9c7f "/var/tmp/localvm363phZ"/hosts /etc removes=None creates=None chdir=None stdin=None Jun 5 21:17:49 lago-hc-basic-suite-4-2-host0 kvm: 1 guest now active Jun 5 21:17:49 lago-hc-basic-suite-4-2-host0 vdsm[19399]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Jun 5 21:17:59 lago-hc-basic-suite-4-2-host0 kvm: 0 guests now active -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

Propose Milan Zamazal as virt maintainer
by Francesco Romani 04 Jun '18

04 Jun '18

Hi all, Milan Zamazal has been working on the oVirt project for more than 2.5 years. Let me highlight some of his many contributions to the project: - Since January this year - 2018, he's been a maintainer for the stable branches. - He developed important features like memory hotunplug, which required tight cooperation and communication with the other layers of the stack (libvirt, qemu). - He is a mentor in the Outreachy program, which lead to creation of the oVirt Log Analyzer: https://github.com/mz-pdm/ovirt-log-analyzer - He contributed more than 290 patches to the Vdsm project in master branch alone, excluding backports and contributions to Engine - He contributed and is contributing testcases and fixes to the oVirt System Test suite, a tool which was already pivotal in ensuring the quality of the oVirt project. As reviewer, Milan is responsive and his comments are always comprehensive and well focused, with strong attitude towards getting things done and done right. Milan also demonstrated his ability to adapt to the needs of the project: - he demonstrated careful and thoughtful patch management while maintaining the stable branches - he also demonstrated he's not shy to tackle large and needed changes during the 4.1 and 4.2 cycles, when we deeply reorganized the XML processing in the virt code. For those reasons, and many more, I think he will be a good addition to the maintainers team, and I propose him as virt co-maintainer. Please share your thoughts Bests, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

6 5

[ OST Failure Report ] [ oVirt master (ovirt-engine) ] [ 03.06.18 ] [098_ovirt_provider_ovn.use_ovn_provider ]
by Gal Ben Haim 03 Jun '18

03 Jun '18

Specifically, it failed to unplug a nic from a VM. Suspected patches: https://gerrit.ovirt.org/#/c/91195/ Link to Job: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8019/ Link to all logs: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8019/artifac… (Relevant) error snippet from the log: <error> 2018-06-03 06:35:30,764-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnplugNicVDSCommand] (default task-1) [32c965fd] FINISH, HotUnplugNicVDSCommand, return: , log id: 18d9b4e9 2018-06-03 06:35:30,764-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [32c965fd] method: runVdsCommand, params: [HotUnplugNic, VmNicDeviceVDSParameters:{hostId='d0f5b521-ed04-4b9e-9524-7f8d4be842d0', vm.vm_name='vm0', nic='VmNic:{id='9419aca6-2fae-4851-a902-a6b5a88ac691', vnicProfileId='8d12fcbb-23e8-432d-845a-ab2b97039955', speed='10000', type='3', macAddress='00:1a:4a:16:01:0e', linked='true', vmId='ec4303e9-2b7a-451d-9f43-836c29767652', vmTemplateId='null'}', vmDevice='VmDevice:{id='VmDeviceId:{deviceId='9419aca6-2fae-4851-a902-a6b5a88ac691', vmId='ec4303e9-2b7a-451d-9f43-836c29767652'}', device='bridge', type='INTERFACE', specParams='[]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}'}], timeElapsed: 55ms 2018-06-03 06:35:30,765-04 ERROR [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (default task-1) [32c965fd] Command 'org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotUnplugNicVDS, error = General Exception: ('macAddr',), code = 100 (Failed with error GeneralException and code 100) 2018-06-03 06:35:30,774-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [32c965fd] method: get, params: [d0f5b521-ed04-4b9e-9524-7f8d4be842d0], timeElapsed: 3ms 2018-06-03 06:35:30,781-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler5) [] Rescheduling DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData#-9223372036854775795 as there is no unfired trigger. 2018-06-03 06:35:30,781-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler6) [] Rescheduling DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices#-9223372036854775801 as there is no unfired trigger. 2018-06-03 06:35:30,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32c965fd] EVENT_ID: NETWORK_DEACTIVATE_VM_INTERFACE_FAILURE(1,015), Failed to unplug Network Interface eth2 (VirtIO) from VM vm0. (User: admin@internal-authz) 2018-06-03 06:35:30,791-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-1) [32c965fd] Compiled stored procedure. Call string i </error> -- *GAL bEN HAIM* RHV DEVOPS

2 3

Recent upgrade issues caused by mess in database scripts
by Martin Perina 03 Jun '18

03 Jun '18

Hi, recently [1], [2] were raised, but the main reason for those bugs was the we are no paying enough attention to consequences when doing backports. Below are two examples which we need to pay more attention to: 1. Database upgrade scripts naming and numbering - there's no problem if the same script has different name or number between y-streams (for example 4.3 and 4.2), but names and number need to match between z-streams - if we are going to do async release (for example 4.2.3.z) and we need to backport to async branch patch which includes db upgrade script, we need to make sure to align db scripts in branch from which next z-stream will be built (for example 4.2, patch [3] can be used as an example how to fix this issue) 2. Use the same name of database upgrade script when doing backports - when backporting some patches to z-stream branch please use the same db script name and change only db script number as needed, otherwise backports are quite confusing (for example [4] and [5]) Patch [3] fixes the situation, so when included into 4.2.4 build it should allow smooth upgrade from 4.2.3.z to 4.2.4, but users which already performed upgrade are in trouble. @Eli is there any way how we can force execution of db scripts, which were skipped due to the mess in names between 4.2.3.z and 4.2.4 to help users which have already broken database? @Tal/@Piotr, could you please pay even more attention when merging patches containing db scripts to z-stream async branches (normal branch, for example ovirt-engine-4.2, should be fine, but ovirt-engine-4.2.3.z requires more attention)? I agree that developer who backports the patch should handle the situation (feel free to ask infra team if unsure about consequences), but you are the last safety check we have. Thanks a lot Martin [1] https://bugzilla.redhat.com/1583562 [2] https://bugzilla.redhat.com/1583664 [3] https://gerrit.ovirt.org/91874 [4] https://gerrit.ovirt.org/90695 [5] https://gerrit.ovirt.org/90698 -- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

3 4

[ OST Failure Report ] [ hc-basic-suite-4.2 ] [30-05-2018 ] [ 002_bootstrap.add_hosts ]
by Gal Ben Haim 01 Jun '18

01 Jun '18

Hi, 4.2 Hyper-converged suite is failing on "add_hosts" test (host 1 deployment failed). Link to Job: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… Link to all logs: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… (Relevant) error snippet from the host deploy log: <error> 2018-05-30 05:30:09,706-0400 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventStart STAGE misc METHOD otopi.plugins.ovirt_host_deploy.hosted-engine.configureha.Plugin._set_ha_conf (None) 2018-05-30 05:30:09,713-0400 ERROR otopi.plugins.ovirt_host_deploy.hosted-engine.configureha configureha._set_ha_conf:114 HA client was not imported 2018-05-30 05:30:09,713-0400 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-lQCcc6VFAV/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-lQCcc6VFAV/otopi-plugins/ovirt-host-deploy/hosted-engine/configureha.py", line 116, in _set_ha_conf _('Cannot resolve ovirt_hosted_engine_ha module') RuntimeError: Cannot resolve ovirt_hosted_engine_ha module <error> -- *GAL bEN HAIM* RHV DEVOPS

3 3

Re: I'm working on the cockpit-gluster plugin and I'd like to be on the wihtelist
by Eyal Edri 01 Jun '18

01 Jun '18

Hi Rohan! The process is quite simple, you just need to ask one of the existing maintainers to send a patch to the 'jenkins-whitelist' repo on gerrit.ovirt.org and one of the infra members can merge it once it has +1 from an existing maintainer. I'm also adding 'infra-support' to open a ticket. On Fri, Jun 1, 2018 at 9:20 AM, Rohan Joseph <rojoseph(a)redhat.com> wrote: > Hi! > > I'm working on the cockpit-gluster plugin and I'd like to be on the > wihtelist to submit patches to gerrit.ovirt.org. > > Thanks! > > - Rohan CJ > > > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ > message/QTQLCWVB5SMNU2MYOBUYUFW5M3HEPWVN/ > > -- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

I want to Translate Ovirt To Chinese
by 唐朝 01 Jun '18

01 Jun '18

Hi: I am a software engineer in China. Chinese is my native language. I prefer to Ovirt than Openstack.Your jobs is very prefessional. Best wishes!

2 1

ovirt-engine-cli | SDK 3 | oVirt 4.3
by Sandro Bonazzola 31 May '18

31 May '18

Hi, Bug 1399709 <https://bugzilla.redhat.com/show_bug.cgi?id=1399709> - [RFE] Deprecate and remove the ovirt-shell tool Stated: "The ovirt-shell is deprecated since version 4.0 of the oVirt, and it will be removed in version 4.2 of oVirt, together with version 3 of the API." and is currently targeted to 4.4. Integration team is in the process to move Jenkins jobs to fedora 28 and I'd like to understand if I need to include ovirt-engine-sdk 3 and ovirt-engine-cli into fedora 28 or not. Right now, ovirt-engine-cli is required by ovirt-engine so if we want to drop it, we need to remove the dependency in ovirt-engine. Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

1 0

post-merge CI triggering from GitHub
by Barak Korren 31 May '18

31 May '18

Its broken right now, we're investigating the issue. Tracker ticket: https://ovirt-jira.atlassian.net/browse/OVIRT-2071 Thanks Tomas Golembiovsky for reporting! -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 1

Re: [ovirt-devel] dynamic ownership changes
by Michal Skrivanek 31 May '18

31 May '18

--Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Elad, why did you install vdsm-hook-allocate_net? adding Dan as I think the hook is not supposed to fail this badly in any = case Thanks, michal > On 5 May 2018, at 19:22, Elad Ben Aharon <ebenahar(a)redhat.com> wrote: >=20 > Start VM fails on: >=20 > 2018-05-05 17:53:27,399+0300 INFO (vm/e6ce66ce) [virt.vm] = (vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path: = 'dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/= images/6cdabfe5-=20 > d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a01928211' -> = u'*dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c6= 2/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-=20= > f35a01928211' (storagexml:334)=20 > 2018-05-05 17:53:27,888+0300 INFO (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3D'940fe6f3-b0c6-4d0c-a921-198e7819c1cc', = options=3DNone) from=3D::ffff:10.35.161.127,53512, = task_id=3Dc70ace39-dbfe-4f5c-ae49-a1e3a82c=20 > 2758 (api:46)=20 > 2018-05-05 17:53:27,909+0300 INFO (vm/e6ce66ce) [root] = /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: rc=3D2 = err=3Dvm net allocation hook: [unexpected error]: Traceback (most recent = call last):=20 > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 105, in <module>=20 > main()=20 > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 93, in main=20 > allocate_random_network(device_xml)=20 > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 62, in allocate_random_network=20 > net =3D _get_random_network()=20 > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 50, in _get_random_network=20 > available_nets =3D _parse_nets()=20 > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 46, in _parse_nets=20 > return [net for net in os.environ[AVAIL_NETS_KEY].split()]=20 > File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__=20 > raise KeyError(key)=20 > KeyError: 'equivnets'=20 >=20 >=20 > (hooks:110)=20 > 2018-05-05 17:53:27,915+0300 ERROR (vm/e6ce66ce) [virt.vm] = (vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process = failed (vm:943)=20 > Traceback (most recent call last):=20 > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in = _startUnderlyingVm=20 > self._run()=20 > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, = in _run=20 > domxml =3D hooks.before_vm_start(self._buildDomainXML(),=20 > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, = in _buildDomainXML=20 > dom, self.id <http://self.id/>, self._custom['custom'])=20 > File = "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py", line = 240, in replace_device_xml_with_hooks_xml=20 > dev_custom)=20 > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line = 134, in before_device_create=20 > params=3DcustomProperties)=20 > File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line = 120, in _runHooksDir=20 > raise exception.HookError(err)=20 > HookError: Hook Error: ('vm net allocation hook: [unexpected error]: = Traceback (most recent call last):\n File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line = 105, in <module>\n main()\n > File "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 93, in main\n allocate_random_network(device_xml)\n File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, = i > n allocate_random_network\n net =3D _get_random_network()\n File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50, = in _get_random_network\n available_nets =3D _parse_nets()\n File = "/us > r/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46, = in _parse_nets\n return [net for net in = os.environ[AVAIL_NETS_KEY].split()]\n File = "/usr/lib64/python2.7/UserDict.py", line 23, in __getit > em__\n raise KeyError(key)\nKeyError: \'equivnets\'\n\n\n',) >=20 >=20 >=20 > Hence, the success rate was 28% against 100% running with d/s (d/s). = If needed, I'll compare against the latest master, but I think you get = the picture with d/s. >=20 > vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64=20 > libvirt-3.9.0-14.el7_5.3.x86_64=20 > qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64=20 > kernel 3.10.0-862.el7.x86_64 > rhel7.5 >=20 >=20 > Logs attached >=20 > On Sat, May 5, 2018 at 1:26 PM, Elad Ben Aharon <ebenahar(a)redhat.com = <mailto:ebenahar@redhat.com>> wrote: > nvm, found gluster 3.12 repo, managed to install vdsm >=20 > On Sat, May 5, 2018 at 1:12 PM, Elad Ben Aharon <ebenahar(a)redhat.com = <mailto:ebenahar@redhat.com>> wrote: > No, vdsm requires it: >=20 > Error: Package: vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64 = (/vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64)=20 > Requires: glusterfs-fuse >=3D 3.12=20 > Installed: glusterfs-fuse-3.8.4-54.8.el7.x86_64 (@rhv-4.2.3) >=20 > Therefore, vdsm package installation is skipped upon force install. >=20 > On Sat, May 5, 2018 at 11:42 AM, Michal Skrivanek = <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: >=20 >=20 >> On 5 May 2018, at 00:38, Elad Ben Aharon <ebenahar(a)redhat.com = <mailto:ebenahar@redhat.com>> wrote: >>=20 >> Hi guys,=20 >>=20 >> The vdsm build from the patch requires glusterfs-fuse > 3.12. This is = while the latest 4.2.3-5 d/s build requires 3.8.4 (3.4.0.59rhs-1.el7) >=20 > because it is still oVirt, not a downstream build. We can=E2=80=99t = really do downstream builds with unmerged changes:/ >=20 >> Trying to get this gluster-fuse build, so far no luck. >> Is this requirement intentional?=20 >=20 > it should work regardless, I guess you can force install it without = the dependency >=20 >>=20 >> On Fri, May 4, 2018 at 2:38 PM, Michal Skrivanek = <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: >> Hi Elad, >> to make it easier to compare, Martin backported the change to 4.2 so = it is actually comparable with a run without that patch. Would you = please try that out?=20 >> It would be best to have 4.2 upstream and this[1] run to really = minimize the noise. >>=20 >> Thanks, >> michal >>=20 >> [1] = http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_64= /28/ = <http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el7-x86_6= 4/28/> >>=20 >>> On 27 Apr 2018, at 09:23, Martin Polednik <mpolednik(a)redhat.com = <mailto:mpolednik@redhat.com>> wrote: >>>=20 >>> On 24/04/18 00:37 +0300, Elad Ben Aharon wrote: >>>> I will update with the results of the next tier1 execution on = latest 4.2.3 >>>=20 >>> That isn't master but old branch though. Could you run it against >>> *current* VDSM master? >>>=20 >>>> On Mon, Apr 23, 2018 at 3:56 PM, Martin Polednik = <mpolednik(a)redhat.com <mailto:mpolednik@redhat.com>> >>>> wrote: >>>>=20 >>>>> On 23/04/18 01:23 +0300, Elad Ben Aharon wrote: >>>>>=20 >>>>>> Hi, I've triggered another execution [1] due to some issues I saw = in the >>>>>> first which are not related to the patch. >>>>>>=20 >>>>>> The success rate is 78% which is low comparing to tier1 = executions with >>>>>> code from downstream builds (95-100% success rates) [2]. >>>>>>=20 >>>>>=20 >>>>> Could you run the current master (without the dynamic_ownership = patch) >>>>> so that we have viable comparision? >>>>>=20 >>>>> =46rom what I could see so far, there is an issue with move and = copy >>>>>> operations to and from Gluster domains. For example [3]. >>>>>>=20 >>>>>> The logs are attached. >>>>>>=20 >>>>>>=20 >>>>>> [1] >>>>>> *https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv = <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv> >>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/ >>>>>> <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv = <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv> >>>>>> -4.2-ge-runner-tier1-after-upgrade/7/testReport/>* >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> [2] >>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ = <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/> >>>>>>=20 >>>>>> rhv-4.2-ge-runner-tier1-after-upgrade/7/ >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> [3] >>>>>> 2018-04-22 13:06:28,316+0300 INFO (jsonrpc/7) [vdsm.api] FINISH >>>>>> deleteImage error=3DImage does not exist in domain: >>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33, >>>>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4' >>>>>> from=3D: >>>>>> :ffff:10.35.161.182,40936, = flow_id=3Ddisks_syncAction_ba6b2630-5976-4935, >>>>>> task_id=3D3d5f2a8a-881c-409e-93e9-aaa643c10e42 (api:51) >>>>>> 2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) = [storage.TaskManager.Task] >>>>>> (Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') Unexpected error = (task:875) >>>>>> Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", = line 882, >>>>>> in >>>>>> _run >>>>>> return fn(*args, **kargs) >>>>>> File "<string>", line 2, in deleteImage >>>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line = 49, in >>>>>> method >>>>>> ret =3D func(*args, **kwargs) >>>>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line = 1503, >>>>>> in >>>>>> deleteImage >>>>>> raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) >>>>>> ImageDoesNotExistInSD: Image does not exist in domain: >>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33, >>>>>> domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4' >>>>>>=20 >>>>>> 2018-04-22 13:06:28,317+0300 INFO (jsonrpc/7) = [storage.TaskManager.Task] >>>>>> (Task=3D'3d5f2a8a-881c-409e-93e9-aaa643c10e42') aborting: Task is = aborted: >>>>>> "Image does not exist in domain: 'image=3Dcabb8846-7a4b-4244-9835- >>>>>> 5f603e682f33, domain=3De5fd29c8-52ba-467e-be09-ca40ff054dd4'" - = code 268 >>>>>> (task:1181) >>>>>> 2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) = [storage.Dispatcher] FINISH >>>>>> deleteImage error=3DImage does not exist in domain: >>>>>> 'image=3Dcabb8846-7a4b-4244-9835-5f603e682f33, >>>>>> domain=3De5fd29c8-52ba-467e-be09 >>>>>> -ca40ff054d >>>>>> d4' (dispatcher:82) >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On Thu, Apr 19, 2018 at 5:34 PM, Elad Ben Aharon = <ebenahar(a)redhat.com <mailto:ebenahar@redhat.com>> >>>>>> wrote: >>>>>>=20 >>>>>> Triggered a sanity tier1 execution [1] using [2], which covers = all the >>>>>>> requested areas, on iSCSI, NFS and Gluster. >>>>>>> I'll update with the results. >>>>>>>=20 >>>>>>> [1] >>>>>>> https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2 = <https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2> >>>>>>> _dev/job/rhv-4.2-ge-flow-storage/1161/ >>>>>>>=20 >>>>>>> [2] >>>>>>> https://gerrit.ovirt.org/#/c/89830/ = <https://gerrit.ovirt.org/#/c/89830/> >>>>>>> vdsm-4.30.0-291.git77aef9a.el7.x86_64 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> On Thu, Apr 19, 2018 at 3:07 PM, Martin Polednik = <mpolednik(a)redhat.com <mailto:mpolednik@redhat.com>> >>>>>>> wrote: >>>>>>>=20 >>>>>>> On 19/04/18 14:54 +0300, Elad Ben Aharon wrote: >>>>>>>>=20 >>>>>>>> Hi Martin, >>>>>>>>>=20 >>>>>>>>> I see [1] requires a rebase, can you please take care? >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>> Should be rebased. >>>>>>>>=20 >>>>>>>> At the moment, our automation is stable only on iSCSI, NFS, = Gluster and >>>>>>>>=20 >>>>>>>>> FC. >>>>>>>>> Ceph is not supported and Cinder will be stabilized soon, = AFAIR, it's >>>>>>>>> not >>>>>>>>> stable enough at the moment. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>> That is still pretty good. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> [1] https://gerrit.ovirt.org/#/c/89830/ = <https://gerrit.ovirt.org/#/c/89830/> >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Thanks >>>>>>>>>=20 >>>>>>>>> On Wed, Apr 18, 2018 at 2:17 PM, Martin Polednik = <mpolednik(a)redhat.com <mailto:mpolednik@redhat.com> >>>>>>>>> > >>>>>>>>> wrote: >>>>>>>>>=20 >>>>>>>>> On 18/04/18 11:37 +0300, Elad Ben Aharon wrote: >>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Hi, sorry if I misunderstood, I waited for more input = regarding what >>>>>>>>>>=20 >>>>>>>>>>> areas >>>>>>>>>>> have to be tested here. >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> I'd say that you have quite a bit of freedom in this regard. >>>>>>>>>> GlusterFS >>>>>>>>>> should be covered by Dennis, so iSCSI/NFS/ceph/cinder with = some suite >>>>>>>>>> that covers basic operations (start & stop VM, migrate it), = snapshots >>>>>>>>>> and merging them, and whatever else would be important for = storage >>>>>>>>>> sanity. >>>>>>>>>>=20 >>>>>>>>>> mpolednik >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On Wed, Apr 18, 2018 at 11:16 AM, Martin Polednik < >>>>>>>>>> mpolednik(a)redhat.com <mailto:mpolednik@redhat.com> >>>>>>>>>> > >>>>>>>>>>=20 >>>>>>>>>> wrote: >>>>>>>>>>>=20 >>>>>>>>>>> On 11/04/18 16:52 +0300, Elad Ben Aharon wrote: >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>> We can test this on iSCSI, NFS and GlusterFS. As for ceph = and >>>>>>>>>>>> cinder, >>>>>>>>>>>>=20 >>>>>>>>>>>> will >>>>>>>>>>>>> have to check, since usually, we don't execute our = automation on >>>>>>>>>>>>> them. >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Any update on this? I believe the gluster tests were = successful, >>>>>>>>>>>>> OST >>>>>>>>>>>>>=20 >>>>>>>>>>>> passes fine and unit tests pass fine, that makes the = storage >>>>>>>>>>>> backends >>>>>>>>>>>> test the last required piece. >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> On Wed, Apr 11, 2018 at 4:38 PM, Raz Tamir = <ratamir(a)redhat.com <mailto:ratamir@redhat.com>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> +Elad >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg = <danken(a)redhat.com <mailto:danken@redhat.com> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer = <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On Wed, Apr 11, 2018 at 12:31 PM Eyal Edri = <eedri(a)redhat.com <mailto:eedri@redhat.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Please make sure to run as much OST suites on this patch = as >>>>>>>>>>>>>>>> possible >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> before merging ( using 'ci please build' ) >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> But note that OST is not a way to verify the patch. >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Such changes require testing with all storage types we = support. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Nir >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On Tue, Apr 10, 2018 at 4:09 PM, Martin Polednik < >>>>>>>>>>>>>>>> mpolednik(a)redhat.com <mailto:mpolednik@redhat.com> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> Hey, >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> I've created a patch[0] that is finally able to = activate >>>>>>>>>>>>>>>>>> libvirt's >>>>>>>>>>>>>>>>>> dynamic_ownership for VDSM while not negatively = affecting >>>>>>>>>>>>>>>>>> functionality of our storage code. >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> That of course comes with quite a bit of code = removal, mostly >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> area of host devices, hwrng and anything that touches = devices; >>>>>>>>>>>>>>>>>> bunch >>>>>>>>>>>>>>>>>> of test changes and one XML generation caveat = (storage is >>>>>>>>>>>>>>>>>> handled >>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>> VDSM, therefore disk relabelling needs to be disabled = on the >>>>>>>>>>>>>>>>>> VDSM >>>>>>>>>>>>>>>>>> level). >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Because of the scope of the patch, I welcome >>>>>>>>>>>>>>>>>> storage/virt/network >>>>>>>>>>>>>>>>>> people to review the code and consider the = implication this >>>>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>> on current/future features. >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> [0] https://gerrit.ovirt.org/#/c/89830/ = <https://gerrit.ovirt.org/#/c/89830/> >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> In particular: dynamic_ownership was set to 0 = prehistorically >>>>>>>>>>>>>>>>>> (as >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> part >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> of https://bugzilla.redhat.com/show_bug.cgi?id=3D554961 = <https://bugzilla.redhat.com/show_bug.cgi?id=3D554961> ) because >>>>>>>>>>>>>>> libvirt, >>>>>>>>>>>>>>> running as root, was not able to play properly with = root-squash >>>>>>>>>>>>>>> nfs >>>>>>>>>>>>>>> mounts. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Have you attempted this use case? >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I join to Nir's request to run this with storage QE. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Raz Tamir >>>>>>>>>>>>>> Manager, RHV QE >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >>>>>=20 >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org <mailto:Devel@ovirt.org> >>> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> >>>=20 >>>=20 >>=20 >>=20 >=20 >=20 >=20 >=20 > <logs.tar.gz> --Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hi = Elad,<div class=3D"">why did you install = vdsm-hook-allocate_net?</div><div class=3D""><br class=3D""></div><div = class=3D"">adding Dan as I think the hook is not supposed to fail this = badly in any case</div><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">michal<br class=3D""><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 5 May = 2018, at 19:22, Elad Ben Aharon <<a href=3D"mailto:ebenahar@redhat.com"= class=3D"">ebenahar(a)redhat.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">Start VM fails on:<div class=3D""><br class=3D""></div><div = class=3D""><span style=3D"font-family:monospace" class=3D""><span = style=3D"background-color: rgb(255, 255, 255);" class=3D"">2018-05-05 = 17:53:27,399+0300 INFO  (vm/e6ce66ce) [virt.vm] = (vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') drive 'vda' path: = 'dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c62/= images/6cdabfe5- </span><br = class=3D"">d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425-f35a0192821= 1' -> = u'*dev=3D/rhev/data-center/mnt/blockSD/db5a6696-d907-4938-9a78-bdd13a843c6= 2/images/6cdabfe5-d1ca-40af-ae63-9834f235d1c8/7ef97445-30e6-4435-8425- = <br class=3D"">f35a01928211' (storagexml:334) <br class=3D"">2018-05-05 = 17:53:27,888+0300 INFO  (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3D'940fe6f3-b0c6-4d0c-a921-198e7819c1cc', = options=3DNone) from=3D::ffff:10.35.161.127,53512, = task_id=3Dc70ace39-dbfe-4f5c-ae49-a1e3a82c <br class=3D"">2758 (api:46) = <br class=3D"">2018-05-05 17:53:27,909+0300 INFO  (vm/e6ce66ce) = [root] /usr/libexec/vdsm/hooks/before_device_create/10_allocate_net: = rc=3D2 err=3Dvm net allocation hook: [unexpected error]: Traceback (most = recent call last): <br class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line = 105, in <module> <br class=3D"">   main() <br = class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 93, = in main <br = class=3D"">   allocate_random_network(device_xml) <br = class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, = in allocate_random_network <br class=3D"">   net =3D = _get_random_network() <br class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50, = in _get_random_network <br class=3D"">   available_nets =3D= _parse_nets() <br class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 46, = in _parse_nets <br class=3D"">   return [net for net in = os.environ[AVAIL_NETS_KEY].split()] <br class=3D""> File = "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__ <br = class=3D"">   raise KeyError(key) <br class=3D"">KeyError: = 'equivnets' <br class=3D""> <br class=3D""> <br class=3D"">(hooks:110) <br class=3D"">2018-05-05 17:53:27,915+0300 = <span style=3D"color:rgb(255,255,255);background-color:rgb(0,0,0)" = class=3D"">ERROR</span><span style=3D"background-color: rgb(255, 255, = 255);" class=3D""> (vm/e6ce66ce) [virt.vm] = (vmId=3D'e6ce66ce-852f-48c5-9997-5d2959432a27') The vm start process = failed (vm:943) </span><br class=3D"">Traceback (most recent call last): = <br class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in = _startUnderlyingVm <br class=3D"">   self._run() <br = class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2861, in _run = <br class=3D"">   domxml =3D = hooks.before_vm_start(self._buildDomainXML(), <br class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2254, in = _buildDomainXML <br class=3D"">   dom, <a = href=3D"http://self.id/" class=3D"">self.id</a>, self._custom['custom']) = <br class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py", line = 240, in replace_device_xml_with_hooks_xml <br = class=3D"">   dev_custom) <br class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 134, in = before_device_create <br = class=3D"">   params=3DcustomProperties) <br = class=3D""> File = "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 120, in = _runHooksDir <br class=3D"">   raise = exception.HookError(err) <br class=3D"">HookError: Hook Error: ('vm net = allocation hook: [unexpected error]: Traceback (most recent call = last):\n  File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line = 105, in <module>\n    main()\n<br = class=3D""> File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 93, = in main\n    allocate_random_network(device_xml)\n =  File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 62, = i<br class=3D"">n allocate_random_network\n    net =3D = _get_random_network()\n  File = "/usr/libexec/vdsm/hooks/before_device_create/10_allocate_net", line 50, = in _get_random_network\n    available_nets =3D = _parse_nets()\n  File "/us<br = class=3D"">r/libexec/vdsm/hooks/before_device_create/10_allocate_net", = line 46, in _parse_nets\n    return [net for net in = os.environ[AVAIL_NETS_KEY].split()]\n  File = "/usr/lib64/python2.7/UserDict.py", line 23, in __getit<br = class=3D"">em__\n    raise KeyError(key)\nKeyError: = \'equivnets\'\n\n\n',)<br class=3D""></span><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">Hence, the success rate was 28% against 100% running with d/s = (d/s). If needed, I'll compare against the latest master, but I think = you get the picture with d/s.</div><div class=3D""><br = class=3D""></div><div class=3D""><span style=3D"font-family:monospace" = class=3D""><span style=3D"background-color: rgb(255, 255, 255);" = class=3D"">vdsm-4.20.27-3.gitfee7810.el7.centos.x86_64 </span><br = class=3D"">libvirt-3.9.0-14.el7_5.3.x86_64 <br = class=3D"">qemu-kvm-rhev-2.10.0-21.el7_5.2.x86_64 <br class=3D"">kernel = 3.10.0-862.el7.x86_64</span></div><div class=3D""><span = style=3D"font-family:monospace" class=3D"">rhel7.5<br = class=3D""></span><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">Logs attached</div></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Sat, = May 5, 2018 at 1:26 PM, Elad Ben Aharon <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:ebenahar@redhat.com" target=3D"_blank" = class=3D"">ebenahar(a)redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">nvm, found gluster 3.12 repo, managed to install = vdsm</div><div class=3D"HOEnZb"><div class=3D"h5"><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Sat, = May 5, 2018 at 1:12 PM, Elad Ben Aharon <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:ebenahar@redhat.com" target=3D"_blank" = class=3D"">ebenahar(a)redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">No, vdsm requires it:<div class=3D""><br = class=3D""></div><div class=3D""><span style=3D"font-family:monospace" = class=3D""><span style=3D"background-color: rgb(255, 255, 255);" = class=3D"">Error: Package: vdsm-4.20.27-3.gitfee7810.el7.<wbr = class=3D"">centos.x86_64 (/vdsm-4.20.27-3.gitfee7810.el<wbr = class=3D"">7.centos.x86_64) </span><br = class=3D"">          Req= uires: glusterfs-fuse >=3D 3.12 <br = class=3D"">          Ins= talled: glusterfs-fuse-3.8.4-54.8.el7.<wbr class=3D"">x86_64 = (@rhv-4.2.3)<br class=3D""></span><br class=3D""></div><div = class=3D"">Therefore, vdsm package installation is skipped upon force = install.</div></div><div class=3D"m_8270803836802176999HOEnZb"><div = class=3D"m_8270803836802176999h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Sat, May 5, 2018 at 11:42 AM, = Michal Skrivanek <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek(a)redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""><br= class=3D""><div class=3D""><span class=3D""><br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D"">On 5 May 2018, at 00:38, Elad = Ben Aharon <<a href=3D"mailto:ebenahar@redhat.com" target=3D"_blank" = class=3D"">ebenahar(a)redhat.com</a>> wrote:</div><br = class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788A= pple-interchange-newline"><div class=3D""><div dir=3D"ltr" class=3D"">Hi = guys, <div class=3D""><br class=3D""></div><div class=3D"">The = vdsm build from the patch requires glusterfs-fuse > 3.12. This = is while the latest 4.2.3-5 d/s build requires 3.8.4 (<span = style=3D"font-family:monospace" class=3D""><span = style=3D"background-color:rgb(255,255,255)" = class=3D"">3.4.0.59rhs-1.el7)</span><br = class=3D""></span></div></div></div></blockquote><div class=3D""><br = class=3D""></div></span>because it is still oVirt, not a downstream = build. We can=E2=80=99t really do downstream builds with unmerged = changes:/</div><div class=3D""><span class=3D""><br class=3D""><blockquote= type=3D"cite" class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div= class=3D""><font face=3D"monospace" class=3D"">Trying to get this = gluster-fuse build, so far no luck.</font></div><div class=3D""><font = face=3D"monospace" class=3D"">Is this requirement = intentional? </font></div></div></div></blockquote><div = class=3D""><br class=3D""></div></span>it should work regardless, I = guess you can force install it without the dependency</div><div = class=3D""><div = class=3D"m_8270803836802176999m_4224343900157515506h5"><div class=3D""><br= class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Fri, = May 4, 2018 at 2:38 PM, Michal Skrivanek <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:michal.skrivanek@redhat.com" = target=3D"_blank" class=3D"">michal.skrivanek(a)redhat.com</a>></span> = wrote:<br class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word;line-break:after-white-space" class=3D"">Hi = Elad,<div class=3D"">to make it easier to compare, Martin backported the = change to 4.2 so it is actually comparable with a run without that = patch. Would you please try that out? </div><div class=3D"">It = would be best to have 4.2 upstream and this[1] run to really minimize = the noise.</div><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">michal</div><div class=3D""><br = class=3D""></div><div class=3D"">[1] <a = href=3D"http://jenkins.ovirt.org/job/vdsm_4.2_build-artifacts-on-demand-el= 7-x86_64/28/" target=3D"_blank" class=3D"">http://jenkins.ovirt.org/j<wbr = class=3D"">ob/vdsm_4.2_build-artifacts-on<wbr = class=3D"">-demand-el7-x86_64/28/</a></div><div class=3D""><br = class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><div = class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788h= 5"><div class=3D"">On 27 Apr 2018, at 09:23, Martin Polednik <<a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a>> wrote:</div><br = class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788m= _-2464431127513935993Apple-interchange-newline"></div></div><div = class=3D""><div class=3D""><div class=3D""><div = class=3D"m_8270803836802176999m_4224343900157515506m_-5974818518343566788h= 5">On 24/04/18 00:37 +0300, Elad Ben Aharon wrote:<br = class=3D""><blockquote type=3D"cite" class=3D"">I will update with the = results of the next tier1 execution on latest 4.2.3<br = class=3D""></blockquote><br class=3D"">That isn't master but old branch = though. Could you run it against<br class=3D"">*current* VDSM master?<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">On Mon, = Apr 23, 2018 at 3:56 PM, Martin Polednik <<a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a>><br class=3D"">wrote:<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">On = 23/04/18 01:23 +0300, Elad Ben Aharon wrote:<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">Hi, I've triggered = another execution [1] due to some issues I saw in the<br class=3D"">first = which are not related to the patch.<br class=3D""><br class=3D"">The = success rate is 78% which is low comparing to tier1 executions with<br = class=3D"">code from downstream builds (95-100% success rates) [2].<br = class=3D""><br class=3D""></blockquote><br class=3D"">Could you run the = current master (without the dynamic_ownership patch)<br class=3D"">so = that we have viable comparision?<br class=3D""><br class=3D"">=46rom = what I could see so far, there is an issue with move and copy<br = class=3D""><blockquote type=3D"cite" class=3D"">operations to and from = Gluster domains. For example [3].<br class=3D""><br class=3D"">The logs = are attached.<br class=3D""><br class=3D""><br class=3D"">[1]<br = class=3D"">*<a = href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv" = target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-v<wbr = class=3D"">ms.eng.rdu2.redhat.com/job/rhv</a><br = class=3D"">-4.2-ge-runner-tier1-after-upg<wbr = class=3D"">rade/7/testReport/<br class=3D""><<a = href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv" = target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-v<wbr = class=3D"">ms.eng.rdu2.redhat.com/job/rhv</a><br = class=3D"">-4.2-ge-runner-tier1-after-upg<wbr = class=3D"">rade/7/testReport/>*<br class=3D""><br class=3D""><br = class=3D""><br class=3D"">[2]<br class=3D""><a = href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/" = target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-vm<wbr = class=3D"">s.eng.rdu2.redhat.com/job/</a><br class=3D""><br = class=3D"">rhv-4.2-ge-runner-tier1-after-<wbr class=3D"">upgrade/7/<br = class=3D""><br class=3D""><br class=3D""><br class=3D"">[3]<br = class=3D"">2018-04-22 13:06:28,316+0300 INFO  (jsonrpc/7) = [vdsm.api] FINISH<br class=3D"">deleteImage error=3DImage does not exist = in domain:<br class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr = class=3D"">-5f603e682f33,<br class=3D"">domain=3De5fd29c8-52ba-467e-be09<w= br class=3D"">-ca40ff054dd4'<br class=3D"">from=3D:<br = class=3D"">:ffff:10.35.161.182,40936, flow_id=3Ddisks_syncAction_ba6b2<wbr= class=3D"">630-5976-4935,<br = class=3D"">task_id=3D3d5f2a8a-881c-409e-93e<wbr class=3D"">9-aaa643c10e42 = (api:51)<br class=3D"">2018-04-22 13:06:28,317+0300 ERROR (jsonrpc/7) = [storage.TaskManager.Task]<br = class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9<wbr class=3D"">-aaa643c10e42')= Unexpected error (task:875)<br class=3D"">Traceback (most recent call = last):<br class=3D"">File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/vdsm/storage/task.py", line 882,<br class=3D"">in<br = class=3D"">_run<br class=3D"">  return fn(*args, **kargs)<br = class=3D"">File "<string>", line 2, in deleteImage<br = class=3D"">File "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/vdsm/common/api.py", line 49, in<br class=3D"">method<br = class=3D"">  ret =3D func(*args, **kwargs)<br class=3D"">File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/vdsm/storage/hsm.py", = line 1503,<br class=3D"">in<br class=3D"">deleteImage<br class=3D""> =  raise se.ImageDoesNotExistInSD(imgUU<wbr class=3D"">ID, sdUUID)<br = class=3D"">ImageDoesNotExistInSD: Image does not exist in domain:<br = class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr = class=3D"">-5f603e682f33,<br class=3D"">domain=3De5fd29c8-52ba-467e-be09<w= br class=3D"">-ca40ff054dd4'<br class=3D""><br class=3D"">2018-04-22 = 13:06:28,317+0300 INFO  (jsonrpc/7) [storage.TaskManager.Task]<br = class=3D"">(Task=3D'3d5f2a8a-881c-409e-93e9<wbr class=3D"">-aaa643c10e42')= aborting: Task is aborted:<br class=3D"">"Image does not exist in = domain: 'image=3Dcabb8846-7a4b-4244-9835<wbr class=3D"">-<br = class=3D"">5f603e682f33, domain=3De5fd29c8-52ba-467e-be09<wbr = class=3D"">-ca40ff054dd4'" - code 268<br class=3D"">(task:1181)<br = class=3D"">2018-04-22 13:06:28,318+0300 ERROR (jsonrpc/7) = [storage.Dispatcher] FINISH<br class=3D"">deleteImage error=3DImage does = not exist in domain:<br class=3D"">'image=3Dcabb8846-7a4b-4244-9835<wbr = class=3D"">-5f603e682f33,<br class=3D"">domain=3De5fd29c8-52ba-467e-be09<b= r class=3D"">-ca40ff054d<br class=3D"">d4' (dispatcher:82)<br = class=3D""><br class=3D""><br class=3D""><br class=3D"">On Thu, Apr 19, = 2018 at 5:34 PM, Elad Ben Aharon <<a = href=3D"mailto:ebenahar@redhat.com" target=3D"_blank" = class=3D"">ebenahar(a)redhat.com</a>><br class=3D"">wrote:<br = class=3D""><br class=3D"">Triggered a sanity tier1 execution [1] using = [2], which covers all the<br class=3D""><blockquote type=3D"cite" = class=3D"">requested areas, on iSCSI, NFS and Gluster.<br class=3D"">I'll = update with the results.<br class=3D""><br class=3D"">[1]<br class=3D""><a= href=3D"https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.2" = target=3D"_blank" class=3D"">https://rhv-jenkins.rhev-ci-vm<wbr = class=3D"">s.eng.rdu2.redhat.com/view/4.2</a><br = class=3D"">_dev/job/rhv-4.2-ge-flow-stora<wbr class=3D"">ge/1161/<br = class=3D""><br class=3D"">[2]<br class=3D""><a = href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/8<wbr class=3D"">9830/</a><br = class=3D"">vdsm-4.30.0-291.git77aef9a.el7<wbr class=3D"">.x86_64<br = class=3D""><br class=3D""><br class=3D""><br class=3D"">On Thu, Apr 19, = 2018 at 3:07 PM, Martin Polednik <<a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a>><br class=3D"">wrote:<br = class=3D""><br class=3D"">On 19/04/18 14:54 +0300, Elad Ben Aharon = wrote:<br class=3D""><blockquote type=3D"cite" class=3D""><br = class=3D"">Hi Martin,<br class=3D""><blockquote type=3D"cite" = class=3D""><br class=3D"">I see [1] requires a rebase, can you please = take care?<br class=3D""><br class=3D""><br class=3D""></blockquote>Should= be rebased.<br class=3D""><br class=3D"">At the moment, our automation = is stable only on iSCSI, NFS, Gluster and<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">FC.<br class=3D"">Ceph = is not supported and Cinder will be stabilized soon, AFAIR, it's<br = class=3D"">not<br class=3D"">stable enough at the moment.<br = class=3D""><br class=3D""><br class=3D""></blockquote>That is still = pretty good.<br class=3D""><br class=3D""><br class=3D"">[1] <a = href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/8<wbr class=3D"">9830/</a><br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><br = class=3D""><br class=3D"">Thanks<br class=3D""><br class=3D"">On Wed, = Apr 18, 2018 at 2:17 PM, Martin Polednik <<a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a><br class=3D"">><br = class=3D"">wrote:<br class=3D""><br class=3D"">On 18/04/18 11:37 +0300, = Elad Ben Aharon wrote:<br class=3D""><br class=3D""><blockquote = type=3D"cite" class=3D""><br class=3D"">Hi, sorry if I misunderstood, I = waited for more input regarding what<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">areas<br class=3D"">have = to be tested here.<br class=3D""><br class=3D""><br class=3D"">I'd say = that you have quite a bit of freedom in this regard.<br = class=3D""></blockquote>GlusterFS<br class=3D"">should be covered by = Dennis, so iSCSI/NFS/ceph/cinder with some suite<br class=3D"">that = covers basic operations (start & stop VM, migrate it), snapshots<br = class=3D"">and merging them, and whatever else would be important for = storage<br class=3D"">sanity.<br class=3D""><br class=3D"">mpolednik<br = class=3D""><br class=3D""><br class=3D"">On Wed, Apr 18, 2018 at 11:16 = AM, Martin Polednik <<br class=3D""><a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a><br class=3D"">><br class=3D""><br = class=3D"">wrote:<br class=3D""><blockquote type=3D"cite" class=3D""><br = class=3D"">On 11/04/18 16:52 +0300, Elad Ben Aharon wrote:<br = class=3D""><br class=3D""><br class=3D""><blockquote type=3D"cite" = class=3D"">We can test this on iSCSI, NFS and GlusterFS. As for ceph = and<br class=3D"">cinder,<br class=3D""><br class=3D"">will<br = class=3D""><blockquote type=3D"cite" class=3D"">have to check, since = usually, we don't execute our automation on<br class=3D"">them.<br = class=3D""><br class=3D""><br class=3D"">Any update on this? I believe = the gluster tests were successful,<br class=3D"">OST<br class=3D""><br = class=3D""></blockquote>passes fine and unit tests pass fine, that makes = the storage<br class=3D"">backends<br class=3D"">test the last required = piece.<br class=3D""><br class=3D""><br class=3D"">On Wed, Apr 11, 2018 = at 4:38 PM, Raz Tamir <<a href=3D"mailto:ratamir@redhat.com" = target=3D"_blank" class=3D"">ratamir(a)redhat.com</a>><br = class=3D"">wrote:<br class=3D""><br class=3D""><br class=3D"">+Elad<br = class=3D""><blockquote type=3D"cite" class=3D""><br class=3D""><br = class=3D"">On Wed, Apr 11, 2018 at 4:28 PM, Dan Kenigsberg <<a = href=3D"mailto:danken@redhat.com" target=3D"_blank" = class=3D"">danken(a)redhat.com</a><br class=3D""><blockquote type=3D"cite" = class=3D"">><br class=3D"">wrote:<br class=3D""><br class=3D"">On = Wed, Apr 11, 2018 at 12:34 PM, Nir Soffer <<a = href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" = class=3D"">nsoffer(a)redhat.com</a>><br class=3D"">wrote:<br = class=3D""><br class=3D""><br class=3D"">On Wed, Apr 11, 2018 at 12:31 = PM Eyal Edri <<a href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D"">eedri(a)redhat.com</a>><br class=3D""><blockquote = type=3D"cite" class=3D"">wrote:<br class=3D""><br class=3D""><br = class=3D"">Please make sure to run as much OST suites on this patch = as<br class=3D""><blockquote type=3D"cite" class=3D"">possible<br = class=3D""><br class=3D"">before merging ( using 'ci please build' )<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><br = class=3D""><br class=3D"">But note that OST is not a way to verify the = patch.<br class=3D""><br class=3D""><br class=3D""></blockquote>Such = changes require testing with all storage types we support.<br = class=3D""><br class=3D"">Nir<br class=3D""><br class=3D"">On Tue, Apr = 10, 2018 at 4:09 PM, Martin Polednik <<br class=3D""><a = href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" = class=3D"">mpolednik(a)redhat.com</a><br class=3D"">><br class=3D""><br = class=3D"">wrote:<br class=3D""><br class=3D""><br class=3D""><blockquote = type=3D"cite" class=3D"">Hey,<br class=3D""><br class=3D""><br = class=3D"">I've created a patch[0] that is finally able to activate<br = class=3D""><blockquote type=3D"cite" class=3D"">libvirt's<br = class=3D"">dynamic_ownership for VDSM while not negatively affecting<br = class=3D"">functionality of our storage code.<br class=3D""><br = class=3D"">That of course comes with quite a bit of code removal, = mostly<br class=3D"">in<br class=3D"">the<br class=3D"">area of host = devices, hwrng and anything that touches devices;<br class=3D"">bunch<br = class=3D"">of test changes and one XML generation caveat (storage is<br = class=3D"">handled<br class=3D"">by<br class=3D"">VDSM, therefore disk = relabelling needs to be disabled on the<br class=3D"">VDSM<br = class=3D"">level).<br class=3D""><br class=3D"">Because of the scope of = the patch, I welcome<br class=3D"">storage/virt/network<br = class=3D"">people to review the code and consider the implication = this<br class=3D"">change<br class=3D"">has<br class=3D"">on = current/future features.<br class=3D""><br class=3D"">[0] <a = href=3D"https://gerrit.ovirt.org/#/c/89830/" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/8<wbr class=3D"">9830/</a><br = class=3D""><br class=3D""><br class=3D"">In particular: =  dynamic_ownership was set to 0 prehistorically<br class=3D"">(as<br = class=3D""><br class=3D""><br class=3D""></blockquote>part<br = class=3D""><br class=3D""></blockquote><br class=3D"">of <a = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D554961" = target=3D"_blank" class=3D"">https://bugzilla.redhat.com/sh<wbr = class=3D"">ow_bug.cgi?id=3D554961</a> ) because<br = class=3D""></blockquote>libvirt,<br class=3D"">running as root, was not = able to play properly with root-squash<br class=3D"">nfs<br = class=3D"">mounts.<br class=3D""><br class=3D"">Have you attempted this = use case?<br class=3D""><br class=3D"">I join to Nir's request to run = this with storage QE.<br class=3D""><br class=3D""><br class=3D""><br = class=3D""><br class=3D"">--<br class=3D""></blockquote><br class=3D""><br= class=3D"">Raz Tamir<br class=3D"">Manager, RHV QE<br class=3D""><br = class=3D""><br class=3D""><br class=3D""><br class=3D""><br = class=3D""></blockquote></blockquote></blockquote></blockquote></blockquot= e></blockquote></blockquote><br class=3D""></blockquote></blockquote><br = class=3D""><br class=3D""></blockquote></blockquote></div></div><span = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel(a)ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""><br class=3D""><br = class=3D""></span></div></div></blockquote></div><br = class=3D""></div></div></blockquote></div><br class=3D""></div> </div></blockquote></div><br = class=3D""></div></div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> <span = id=3D"cid:77382D15-7BFB-4164-A6D0-F8FA5BE5E692@mrkev"><logs.tar.gz><= /span></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_82B55436-9E61-4F82-AF39-6B64C656C187--

5 11