April 2018 - Devel - oVirt List Archives

OST Failure - Weekly update [14/04/2018-20/04/2018]
by Dafna Ron 20 Apr '18

20 Apr '18

Hi, I wanted to give a short status on this week's failures and OST current status. We had one regression this week which was reported by Gal to the list and was fix quickly. The regression was caused by change: https://gerrit.ovirt.org/#/c/90376/ - core: @Inject DAOs to QoS CRUD commands Tal debugged the issue and found that WELD doesn't allow injections of from generic declared types and revered the change until the issue can be fixed ( https://gerrit.ovirt.org/#/c/90376/ - Revert "core: @Inject DAOs to QoS CRUD commands"). We have an issue with CQ alerts ( https://ovirt-jira.atlassian.net/browse/OVIRT-1974) and we have been monitoring CQ manually until the issue is resolved. This is why I am unable to issue this week's statistics as they are based on CQ reports on failures. I am sure this would be fixed soon and I will be able to issue reports as usual next week. for now, please note that CQ looks good with no failures in the past few hours. Thank you, Dafna

1 0

Wrong pep8 library on centos
by Petr Kotas 20 Apr '18

20 Apr '18

Hi, Sending this to spread information. I have encountered and issue with Centos 7.4.1708 (Core) when compiling the oVirt engine. The pep8 validation was not passing correctly. It turned out, that the Centos has the old and bugged version present by default. The solution is to use the same version as used in oVirt CI. Just add http://resources.ovirt.org/repos/ci-tools/el7/ to yum.repos.d and it will start passing again. Best, Petr

2 1

Broken repoclosure in today 4.2.3 RC2 compose
by Sandro Bonazzola 18 Apr '18

18 Apr '18

Hi, today oVirt 4.2.3 RC2 release is blocked on repository closure error detected by ovirt-engine-appliance build: 13:49:14 11:49:13,944 INFO program:Error: Package: ovirt-engine-backend-4.2.3.2-1.el7.centos.noarch (ovirt-4.2-pre)13:49:14 11:49:13,946 INFO program:Requires: vdsm-jsonrpc-java >= 1.4.1213:49:14 11:49:13,947 INFO program:Available: vdsm-jsonrpc-java-1.4.11-1.el7.noarch (ovirt-4.2-centos-ovirt42)13:49:14 11:49:13,949 INFO program:vdsm-jsonrpc-java = 1.4.11-1.el713:49:14 11:49:13,950 INFO program:Available: vdsm-jsonrpc-java-1.4.11-1.el7.centos.noarch (ovirt-4.2)13:49:14 11:49:13,951 INFO program:vdsm-jsonrpc-java = 1.4.11-1.el7.centos Looks like vdsm-jsonrpc-java >= 1.4.12 has not been released yet. Piotr please check its status and add it to release configuration file for 4.2.3 RC2. You can see an example in https://gerrit.ovirt.org/#/c/90436/ Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

2 1

Fwd: [CentOS-devel] [SIG Process] Point-Release Freeze of SIG Content while CR is Active
by Sandro Bonazzola 18 Apr '18

18 Apr '18

FYI ---------- Forwarded message ---------- From: Brian Stinson <brian(a)bstinson.com> Date: 2018-04-17 15:30 GMT+02:00 Subject: [CentOS-devel] [SIG Process] Point-Release Freeze of SIG Content while CR is Active To: centos-devel(a)centos.org Hi Folks, At the CBS meeting yesterday we discussed a proposal to freeze the Sign and Push-to-mirror processes during the period between when CR is active and a point release. https://lists.centos.org/pipermail/centos-devel/2018-April/016610.html You may recall that we did something similar for 7.1708 We need to Freeze because we make CR content available in CBS for builds, and in CI for testing purposes. But, releasing content built against CR to mirror.centos.org would have had some unintended consequences (e.g. we don't want to require that end-users enable CR to get content from the SIGs). The Point-release Freeze does not affect content moving from the testing tags in CBS to buildlogs.c.o, that will continue to show up as it is built. Do note: since we do want to give testers and developers an opportunity to build and test against an upcoming point-release before it goes GA, content coming from buildlogs.centos.org may require enabling the CR repo. Any new requests for sign+push (adding new tags to mirror.c.o) during Point-release Freeze will be held until the point release goes GA. We approved this as part of the CR checklist during the meeting yesterday, and will include another note about this in the CR release announcement. https://www.centos.org/minutes/2018/April/centos-devel.2018-04-16-14.00.html If there are any questions, please let us know and discuss here on the list. Cheers! -- Brian Stinson _______________________________________________ CentOS-devel mailing list CentOS-devel(a)centos.org https://lists.centos.org/mailman/listinfo/centos-devel -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

1 0

[ OST Failure Report ] [ oVirt master ] [ 17.04.18 ] [ 002_bootstrap.add_qos ]
by Gal Ben Haim 17 Apr '18

17 Apr '18

Hi, >From last night, the Change Queue fails on every run because of "add_qos" test. >From the engine log: 2018-04-16 10:31:51,813-04 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (default task-30) [] An exception has occurred while trying to create a command object for command 'AddCpuQos' with parameters 'QosParametersBase:{commandId='ccb4d6f4-ca86-4706-921c-f87439e0550f', user='null', commandType='Unknown'}': WELD-001407: Cannot declare an injection point with a type variable: [BackedAnnotatedField] @Inject private org.ovirt.engine.core.bll.qos.QosCommandBase.qosDao at org.ovirt.engine.core.bll.qos.QosCommandBase.qosDao(QosCommandBase.java:0) 2018-04-16 10:31:51,819-04 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (default task-30) [] Exception: org.jboss.weld.exceptions.DefinitionException: WELD-001407: Cannot declare an injection point with a type variable: [BackedAnnotatedField] @Inject private org.ovirt.engine.core.bll.qos.QosCommandBase.qosDao at org.ovirt.engine.core.bll.qos.QosCommandBase.qosDao(QosCommandBase.java:0) StackTrace at org.jboss.weld.bootstrap.Validator.validateInjectionPointForDefinitionErrors(Validator.java:295) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bootstrap.Validator.validateInjectionPoint(Validator.java:281) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bootstrap.Validator.validateProducer(Validator.java:400) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.injection.producer.InjectionTargetService.validateProducer(InjectionTargetService.java:36) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.manager.InjectionTargetFactoryImpl.validate(InjectionTargetFactoryImpl.java:135) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:69) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.manager.BeanManagerImpl.createInjectionTarget(BeanManagerImpl.java:1140) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.util.ForwardingBeanManager.createInjectionTarget(ForwardingBeanManager.java:201) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.di.Injector.injectMembers(Injector.java:29) [vdsbroker.jar:] at org.ovirt.engine.core.bll.CommandsFactory.createCommand(CommandsFactory.java:100) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161] Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6839/artifact… -- *GAL bEN HAIM* RHV DEVOPS

2 2

[OST][HC] HE fails to deploy
by Sahina Bose 16 Apr '18

16 Apr '18

HE fails to deploy at waiting for host to be up in the local HE VM. The setup logs does not indicate why it failed - atleast I couldn't find anything ---------- Forwarded message ---------- From: <jenkins(a)jenkins.phx.ovirt.org> Date: Mon, Apr 2, 2018 at 7:50 PM Subject: [oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 276 - Still Failing! To: infra(a)ovirt.org, sabose(a)redhat.com Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic- suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic- suite-master/276/ Build Number: 276 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #265 [Gal Ben Haim] Check if the prefix exists before printing its size [Sandro Bonazzola] ovirt-engine: add jobs for 4.1.10 async Changes for Build #266 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #267 [Gal Ben Haim] Check if the prefix exists before printing its size [Daniel Belenky] ppc repos: Use qemu EV release instead of test [Daniel Belenky] global_setup: Add generic package remove function [Daniel Belenky] Fix package verification in verify_packages Changes for Build #268 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #269 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #270 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #271 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #272 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #273 [Eitan Raviv] network: macpool: test disallowing dups while dups exist [Daniel Belenky] docker cleanup:Fix edge case for unamed containers [Daniel Belenky] nested_config: Count nesting level of options [Daniel Belenky] Introduce conditional execution in STDCI DSL [Daniel Belenky] Add OST STDCI V2 jobs Changes for Build #274 [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch Changes for Build #275 [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch Changes for Build #276 [Barak Korren] Force STDCI V2 job to use physical host [Daniel Belenky] Build container on changes to docker_cleanup ----------------- Failed Tests: ----------------- No tests ran.

3 7

OST Failure - Weekly update [07/04/2018-13/04/2018]
by Dafna Ron 13 Apr '18

13 Apr '18

On Fri, Apr 13, 2018 at 3:49 PM, Dafna Ron <dron(a)redhat.com> wrote: > Hello, > > I would like to update on this week's failures and OST current status. > > We had some issues from previous week. once is still sporadically failing > but since there were a few patches merged and another one in the processes > I decided we can close this sprint with no backlog. > > this week's failures were one regression, sporadic failures and one > packaging issue. > > We are still seeing sporadic failures on migration tests but Gal has a fix > which would probably be merged soon. https://gerrit.ovirt.org/#/c/90166/ > - master,4.2: Wait for migration to end > > we have found and reported a regression change: > https://gerrit.ovirt.org/#/c/89581/ - Subnet name should be an optional > field > This was fixed quickly by: https://gerrit.ovirt.org/#/c/89980/ - Fix > mistaken mapping for retrieving subnet name from row > > We moved to Ansible 2.5 but the packages were not updated in the repos. > this was fixed by this change: https://gerrit.ovirt.org/#/c/90015/ - 4.2: > Take ansible from epel > > > > *Below you can see the chart for this week's resolved issues but cause of > failure:**Code* = regression of working components/functionalities > *Infra* = infrastructure/OST Infrastructure/Lago related issues/Power > outages > *OST* *Tests *- package related issues, failed build artifacts > > > > > > > > > > > *Below is a chart of resolved failures based on ovirt version:* > > > > *Below is a chart showing failures by suite type: * > > > *Below you can see the issues opened this week vs the backlog issues from > previous week: * > > > Thanks, > Dafna > > > > > > > >

1 0

OST: Enabling DEBUG logging level in Vdsm?
by Milan Zamazal 13 Apr '18

13 Apr '18

Hi, it's quite inconvenient that DEBUG messages are missing in OST Vdsm logs. Would it be possible to enable them some way? Thanks, Milan

2 2

make check on master fails due to UnicodeDecodeError
by Shani Leviim 11 Apr '18

11 Apr '18

Hi there, I'm trying to run make check, and I have ~13 tests on vdsm/tests which failes due to the following: File "/home/sleviim/git/vdsm/lib/vdsm/api/vdsmapi.py", line 212, in __init__ loaded_schema = pickle.load(f) File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128) (Those lines are common to all failures) Here is an example: ====================================================================== ERROR: test_ok_response (vdsmapi_test.DataVerificationTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/sleviim/git/vdsm/tests/vdsmapi_test.py", line 96, in test_ok_response _schema.schema().verify_retval( File "/home/sleviim/git/vdsm/tests/vdsmapi_test.py", line 67, in schema self._schema = vdsmapi.Schema(paths, True) File "/home/sleviim/git/vdsm/lib/vdsm/api/vdsmapi.py", line 212, in __init__ loaded_schema = pickle.load(f) File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128) I've also tried to git clean -dxf && ./autogen.sh --system but it didn't help. Can you please assist? Thanks! *Regards,* *Shani Leviim*

2 5

planned Jenkins restart
by Evgheni Dereveanchin 11 Apr '18

11 Apr '18

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. Regards, Evgheni Dereveanchin

1 1

[ OST Failure Report ] [ oVirt master ] [ 2018-04-08 ] [098_ovirt_provider_ovn.use_ovn_provider]
by Barak Korren 08 Apr '18

08 Apr '18

Test failed: 098_ovirt_provider_ovn.use_ovn_provider Link to suspected patches: https://gerrit.ovirt.org/#/c/89581/3 Link to Job: https://gerrit.ovirt.org/#/c/89581/3 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6714/artifact… Error snippet from log: <error> 'name' -------------------- >> begin captured logging << -------------------- lago.providers.libvirt.cpu: DEBUG: numa : cpus_per_cell: 1, total_cells: 2 lago.providers.libvirt.cpu: DEBUG: numa: <numa> <cell cpus="0" id="0" memory="1023" unit="MiB"/> <cell cpus="1" id="1" memory="1023" unit="MiB"/> </numa> lago.providers.libvirt.cpu: DEBUG: numa : cpus_per_cell: 1, total_cells: 2 lago.providers.libvirt.cpu: DEBUG: numa: <numa> <cell cpus="0" id="0" memory="1023" unit="MiB"/> <cell cpus="1" id="1" memory="1023" unit="MiB"/> </numa> lago.providers.libvirt.cpu: DEBUG: numa : cpus_per_cell: 1, total_cells: 2 lago.providers.libvirt.cpu: DEBUG: numa: <numa> <cell cpus="0" id="0" memory="2048" unit="MiB"/> <cell cpus="1" id="1" memory="2048" unit="MiB"/> </numa> requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 py.warnings: WARNING: * Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/tokens/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/networks/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/ports/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/subnets/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/networks/ HTTP/1.1" 201 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/subnets/ HTTP/1.1" 201 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/ports/ HTTP/1.1" 201 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/networks/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/ports/ HTTP/1.1" 200 None requests.packages.urllib3.connectionpool: INFO: * Starting new HTTPS connection (1): 192.168.201.4 requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/subnets/ HTTP/1.1" 200 None --------------------- >> end captured logging << --------------------- </error> Note: we're seeing similar issues on the same patches in both the 'master' and the 4.2 change queues. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

3 3

Re: [ovirt-devel] [virt-tools-list] Project for profiles and defaults for libvirt domains
by Martin Kletzander 08 Apr '18

08 Apr '18

--ieNMXl1Fr3cevapt Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable [ I fixed up ovirt-devel(a)redhat.com to be devel(a)ovirt.org since the former is deprecated. I'm also not trimming down much of the reply so that they can get the whole picture. Sorry for the confusion ] On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrang=E9 wrote: >On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote: >> 1) Default devices/values >> >> Libvirt itself must default to whatever values there were before any >> particular element was introduced due to the fact that it strives to >> keep the guest ABI stable. That means, for example, that it can't just >> add -vmcoreinfo option (for KASLR support) or magically add the pvpanic >> device to all QEMU machines, even though it would be useful, as that >> would change the guest ABI. >> >> For default values this is even more obvious. Let's say someone figures >> out some "pretty good" default values for various HyperV enlightenment >> feature tunables. Libvirt can't magically change them, but each one of >> the projects building on top of it doesn't want to keep that list >> updated and take care of setting them in every new XML. Some projects >> don't even expose those to the end user as a knob, while others might. > >This gets very tricky, very fast. > >Lets say that you have an initial good set of hyperv config >tunables. Now sometime passes and it is decided that there is a >different, better set of config tunables. If the module that is >providing this policy to apps like OpenStack just updates itself >to provide this new policy, this can cause problems with the >existing deployed applications in a number of ways. > >First the new config probably depends on specific versions of >libvirt and QEMU, and you can't mandate to consuming apps which >versions they must be using. So you need a matrix of libvirt + >QEMU + config option settings. > >Even if you have the matching libvirt & QEMU versions, it is not >safe to assume the application will want to use the new policy. >An application may need live migration compatibility with older >versions. Or it may need to retain guaranteed ABI compatibility >with the way the VM was previously launched and be using transient >guests, generating the XML fresh each time. > >The application will have knowledge about when it wants to use new >vs old hyperv tunable policy, but exposing that to your policy module >is very tricky because it is inherantly application specific logic >largely determined by the way the application code is written. > The idea was for updating XML based on policy, which is something you want for new machines. You should then keep the XML per domain and only do changes to if requested by the user or when libvirt fills in new values in a guest ABI compatible fashion. > >> One more thing could be automatically figuring out best values based on >> libosinfo-provided data. >> >> 2) Policies >> >> Lot of the time there are parts of the domain definition that need to be >> added, but nobody really cares about them. Sometimes it's enough to >> have few templates, another time you might want to have a policy >> per-scenario and want to combine them in various ways. For example with >> the data provided by point 1). >> >> For example if you want PCI-Express, you need the q35 machine type, but >> you don't really want to care about the machine type. Or you want to >> use SPICE, but you don't want to care about adding QXL. >> >> What if some of these policies could be specified once (using some DSL >> for example), and used by virtuned to merge them in a unified and >> predictable way? >> >> 3) Abstracting the XML >> >> This is probably just usable for stateless apps, but it might happen >> that some apps don't really want to care about the XML at all. They >> just want an abstract view of the domain, possibly add/remove a device >> and that's it. We could do that as well. I can't really tell how much >> of a demand there is for it, though. > >It is safe to say that applications do not want to touch XML at all. >Any non-trivial application has created an abstraction around XML, >so that they have an API to express what they want, rather than >manipulating of strings to format/parse XML. > Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps). However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea. Since you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below). [...] >If there was something higher level that gets more interesting, >but the hard bit is that you still need a way to get at all the >low level bits becuase a higher level abstracted API will never >cover every niche use case. > Oh, definitely not every, but I see two groups of projects that have a lot in common between themselves and between the groups as well. On the other hand just templating and defaults is something that's easy enough to do that it's not worth outsourcing that into another one's codebase. >> 4) Identifying devices properly >> >> In contrast to the previous point, stateful apps might have a problem >> identifying devices after hotplug. For example, let's say you don't >> care about the addresses and leave that up to libvirt. You hotplug a >> device into the domain and dump the new XML of it. Depending on what >> type of device it was, you might need to identify it based on different >> values. It could be <target dev=3D''/> for disks, <mac address=3D''/> f= or >> interfaces etc. For some devices it might not even be possible and you >> need to remember the addresses of all the previous devices and then >> parse them just to identify that one device and then throw them away. >> >> With new enough libvirt you could use the user aliases for that, but >> turns out it's not that easy to use them properly anyway. Also the >> aliases won't help users identify that device inside the guest. > >NB, relating between host device config and guest visible device >config is a massive problem space in its own right, and not very >easy to address. In OpenStack we ended up defining a concept of >"device tagging" via cloud-init metadata, where openstack allows >users to set opaque string tags against devices their VM has. >OpenStack that generates a metadata file that records various >pieces of identifying hardware attributes (PCI address, MAC >addr, disk serial, etc) alongside the user tag. This metadata >file is exposed to the guest with the hope that there's enough >info to allow the user to decide which device is to be used for >which purpose > This is good point, but I was mostly thinking about identifying devices =66rom the host POV between two different XMLs (pre- and post- some XML-modifying action, like hotplug). >https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/vir= t-device-role-tagging.html >https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/1= 0/html/networking_guide/use-tagging > >> <rant> >> We really should've gone with new attribute for the user alias instead >> of using an existing one, given how many problems that is causing. >> </rant> >> >> 5) Generating the right XML snippet for device hot-(un)plug >> >> This is kind of related to some previous points. >> >> When hot-plugging a device and creating an XML snippet for it, you want >> to keep the defaults from point 1) and policies from 2) in mind. Or >> something related to the already existing domain which you can describe >> systematically. And adding something for identification (see previous >> point). >> >> Doing the hot-unplug is easy depending on how much information about >> that device is saved by your application. The less you save about the >> device (or show to the user in a GUI, if applicable) the harder it might >> be to generate an XML that libvirt will accept. Again, some problems >> with this should be fixed in libvirt, some of them are easy to >> workaround. But having a common ground that takes care of this should >> help some projects. >> >> Hot-unplug could be implemented just based on the alias. This is >> something that would fit into libvirt as well. >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> >> To mention some pre-existing solutions: >> >> - I understand OpenStack has some really sensible and wisely chosen >> and/or tested default values. > >In terms of default devices and OS specific choices, OpenStack's >decisions have been largely inspired by previous work in oVirt >and / or virt-manager. So there's obviously overlap in the >conceptual area, but there's also plenty that is very specific >to OpenStack - untangling the two extract the common bits from >the app specific bits is hard. > It definitely is, but do you think it's so difficult it's worthless to pursuit? I did a tiny PoC based on the code from virt-manager, which was trivial mainly thanks to the XMLBuilder for the domain objects. Maybe exposing an easy way to work with the XML would be enough for some projects. Little birdie from oVirt told me that they would like some of sort of thing that does what you can achieve with virt-xml if we, for example, made it work on pure XML definitions without connecting to libvirt. >> - I know KubeVirt has VirtualMachinePresets. That is something closely >> related to points 1) and 2). Also their abstraction of the XML might >> be usable for point 3). >> >> - There was an effort on creating policy based configuration of libvirt >> objects called libvirt-designer. This is closely related to points 2) >> and 3). Unfortunately there was no much going on lately and part of >> virt-manager repository has currently more features implemented with >> the same ideas in mind, just not exported for public use. > >This is the same kind of problem we faced wrt libvirt-gconfig and >libvirt-gobject usage from virt-manager - it has an extensive code >base that already works, and rewriting it to use something new >is alot of work for no short-term benefit. libvirt-gconfig/gobject >were supposed to be the "easy" bits for virt-manager to adopt, as >they don't really include much logic that would step on virt-manager's >toes. libvirt-designer was going to be a very opinionated library >and in retrospective that makes it even harder to consider adopting >it for usage in virt-manager, as it'll have signficant liklihood >of making functionally significant changes in behaviour. > The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest. >There's also the problem with use of native libraries that would >impact many apps. We only got OpenStack to grudgingly allow the By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API. >use of libosinfo native library via GObject Introspection, by >promising to do work to turn the osinfo database into an approved >stable format which OpenStack could then consume directly, dropping >the native API usage :-( Incidentally, the former was done (formal >spec for the DB format), but the latter was not yet (direct DB usage >by OpenStack) > > >BTW, I don't like that I'm being so negative to your proposal :-( >I used to hope that we would be able to build higher level APIs on >top of libvirt to reduce the overlap between different applications >reinventing the wheel. Even the simplest bits we tried like the >gconfig/gobject API are barely used. libvirt-designer is basically >a failure. Though admittedly it didn't have enough development resource >applied to make it compelling, in retrospect adoption was always going >to be a hard sell except in greenfield developments. > I'm glad for the knowledge you provided. So maybe instead of focusing on de-duplication of existing codebases we could _at least_ aim at future mgmt apps. OTOH improving documentation on how to properly build higher level concepts on top of libvirt would benefit them as well. >Libosinfo is probably the bit we've had most success with, and has >most promise for the future, particularly now that we formally allow >apps to read the osinfo database directly and bypass the API. It is >quite easy to fit into existing application codebases which helps alot. >Even there I'm still disappointed that we only have GNOME Boxes using >the kickstart generator part of osinfo - oVirt and Oz both still have >their own kickstart generator code for automating OS installs. > >In general though, I fear anything API based is going to be a really >hard sell to get wide adoption for based on what we've seen before. > >I think the biggest bang-for-buck is identifying more areas where we >can turn code into data. There's definitely scope for recording more >types of information in the osinfo database. There might also be >scope for defining entirely new databases to complement the osinfo >data, if something looks out of scope for libosinfo. > >Regards, >Daniel >--=20 >|: https://berrange.com -o- https://www.flickr.com/photos/dberrang= e :| >|: https://libvirt.org -o- https://fstop138.berrange.co= m :| >|: https://entangle-photo.org -o- https://www.instagram.com/dberrang= e :| --ieNMXl1Fr3cevapt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEiXAnXDYdKAaCyvS1CB/CnyQXht0FAlqzw4kACgkQCB/CnyQX ht0G+xAAnvv+RKtfL7aDX7nVWclilDw7XYt+0a01KuLoF7a9FZ0emZgRr2tiX8NO iTU1t68Ch62vni+FY77cuPM7eixUdYEwUaVybptJvWKzAgladXpA93ZfsT0wGxpi qIrkpgbJmiAULogBrc8ia2bqy2MpFDIFAPzsMDTsznJH/cJMErA+Hg5A8VVOt0Cl j6uwGCT7MYRBK2/th3Q0POVLtnj6MnXbuTda3sl45aNMnOEDmxKNW+zNdgUfbAxO 0GdjfypwJyFIz4FZRrVvlnMEHlqkZyJcDF6BQN+lVKsTizcDAYV6QM3snNLTFgsF huIcrMsWNnDG/GsbkFyc6XkvPeZ6xonMW0LEYHP6pLTyg1CGIJ7E6VaPNlEVRU+J aJjqHEW14oTc6ttuf8m7El972ATXgY6l1W4FiQisIy7PpwpGb896zoaEyYryGJT4 WWXFW60QoDPNmjHgA4S0PdefEuKrD1u5BUYVExpVW7ud8uyggssKz6id7mv+4J6Y Jv0WaaIA/l53mInyEzOpTHT6tMExH64tHF2RdghQ+r7tzNxtCmBDmj3QcwHzaa97 +R0txkevMqJ57l2QvWKc2z7CJHsuZqolDrGpMUJ35PFOqAeq2elj9F4fmUlV+nt5 Ju4qL3mrJ4AdhtXBACcQLNqUsbgGs1E99gY2v9mpJOPvSyVD5O0= =L9yQ -----END PGP SIGNATURE----- --ieNMXl1Fr3cevapt--

3 4

[ OST Failure Report ] [ oVirt 4.2 (vdsm) ] [ 05-04-2018 ] [ 004_basic_sanity.hotplug_cpu ]
by Dafna Ron 08 Apr '18

08 Apr '18

Hi, We had a failure on 004_basic_sanity.hotplug_cpu. looking at the logs, I do not see an issue on the cpu hotplug but it seems that the host had an issue with network config. can you please have a look? *Link and headline of suspected patches: Failed patch: * *virt: storage: minimal changes to the drive XML - https://gerrit.ovirt.org/#/c/89508/ <https://gerrit.ovirt.org/#/c/89508/> * *Reported as failure cause: * *virt: extract local adjustments to XML - https://gerrit.ovirt.org/#/c/89506/ <https://gerrit.ovirt.org/#/c/89506/> * *Link to Job:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1565/ <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1565/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1565/artifa… <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1565/artifact/ex…>(Relevant) error snippet from the log: <error>MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:17,451::cmdutils::150::root::(exec_cmd) /sbin/ifdown eth2 (cwd None)MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:17,719::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:17,720::cmdutils::150::root::(exec_cmd) /sbin/ifdown eth3 (cwd None)MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:17,982::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0ifup/onc5bbb5eed6a84::DEBUG::2018-04-05 18:58:18,165::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'Running scope as unit b32c8477-857f-476c-96c8-580daf0d4217.scope.\n/etc/sysconfig/network-scripts/ifup-eth: line 304: 12145 Terminated /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nDevice "onc5bbb5eed6a84" does not exist.\nCannot find device "onc5bbb5eed6a84"\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\nDevice "onc5bbb5eed6a84" does not exist.\n'; <rc> = 1ifup/onc5bbb5eed6a84::ERROR::2018-04-05 18:58:18,165::concurrent::201::root::(run) FINISH thread <Thread(ifup/onc5bbb5eed6a84, stopped daemon 139876090296064)> failedTraceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 942, in _exec_ifup _exec_ifup_by_name(iface.name <http://iface.name>, cgroup) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 928, in _exec_ifup_by_name raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')ConfigNetworkError: (29, '\n')MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:20,075::cmdutils::150::root::(exec_cmd) /sbin/ip addr flush dev eth2 scope global (cwd None)MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:20,081::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:20,082::ifcfg::479::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth2MainProcess|jsonrpc/2::DEBUG::2018-04-05 18:58:20,084::ifcfg::512::root::(_persistentBackup) backing up ifcfg-eth2: HWADDR="54:52:c0:a8:c9:02" BOOTPROTO="dhcp" ONBOOT="yes" TYPE="Ethernet" NAME="eth2" </error>*

2 1

OST Failure - Weekly update [30/03/2018-06/04/2018]
by Dafna Ron 06 Apr '18

06 Apr '18

Hello, I would like to update on this week's failures and OST current status. We had a few failures this week and some are still on-going issues. We had two unrelated issues failing the test: 006_migrations.prepare_migration_attachments_ipv6 This failure was periodical and is causing failure for several changes. This issue is still on-going and seems to be pointing to this change as the root cause: https://gerrit.ovirt.org/#/c/89812/1 - examples: export template to an export domain Issue is still under investigation. On Thursday evening and Friday we seem to have had failures which indicate issues with migration and ipv6 configuration, The tests that failed were 006_migrations.migrate_vm and 004_basic_sanity.hotplug_cpu cpu hotplug failure was reporting these changes: Failed patch: virt: storage: minimal changes to the drive XML - https://gerrit.ovirt.org/#/c/89508/ Reported as failure cause: virt: extract local adjustments to XML - https://gerrit.ovirt.org/#/c/89506/ and migration failure was reporting this patch: *aaa: can't switch user when accessing the engine with an active kerberos ticket - https://gerrit.ovirt.org/#/c/89872/ <https://gerrit.ovirt.org/#/c/89872/>* *This three issues are still on-going and you can see the cases resolved vs the backlog which is going forward to next week: * *Infra/OST tests related issues: * We also had a failure which seems to be related to the ost test and was fixed in this patch: https://gerrit.ovirt.org/#/c/89781/ - Prevent network to be imported by auto_sync Further discussion on this can be found in the google doc opened by Dominik: https://docs.google.com/document/d/1VeKfI7luw-HHTCCmfogLNiYLTtIG_3maENVQlDw… *Below you can see the chart for this week's resolved issues but cause of failure:* *Code* = regression of working components/functionalities *Infra* = infrastructure/OST Infrastructure/Lago related issues/Power outages *OST Tests* - package related issues, failed build artifacts *Below is a chart of resolved failures based on ovirt version* *Below is a chart showing failures by suite type: * Thanks, Dafna

2 1

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 06-04-2018 ] [ 006_migrations.migrate_vm ]
by Dafna Ron 06 Apr '18

06 Apr '18

hi, 006_migrations.migrate_vm failure points to the below patch. There seems to be an issue to migrate vms but I am not sure its related to this patch. can you please check? *Link and headline of suspected patches: aaa: can't switch user when accessing the engine with an active kerberos ticket - https://gerrit.ovirt.org/#/c/89872/ <https://gerrit.ovirt.org/#/c/89872/>Link to Job:* * http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6704/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6704/>Link to all logs:* http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6704/artifact… <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6704/> *(Relevant) error snippet from the log: <error>2018-04-06 04:07:56,517-0400 ERROR (qgapoller/2) [Executor] Unhandled exception in <Task discardable <NetworkInterfacesCheck vm=aad106a1-9043-4661-bb9b-9894f66b506b at 0x7fc4681a2590> timeout=10, duration=0.00 at 0x7fc4541193d0> (executor:317)Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 314, in __call__ self._execute() File "/usr/lib/python2.7/site-packages/vdsm/virt/qemuguestagent.py", line 388, in _execute for ifname, ifparams in six.iteritems(interfaces): File "/usr/lib/python2.7/site-packages/six.py", line 599, in iteritems return d.iteritems(**kw)AttributeError: 'list' object has no attribute 'iteritems'*2018-04-06 04:23:10,165-0400 ERROR (migsrc/aad106a1) [virt.vm] (vmId='aad106a1-9043-4661-bb9b-9894f66b506b') migration destination error: Virtual machine already exists (migration:290) *</error>*

2 2

How To Construct An Local Developing Enviroment?
by sundw 06 Apr '18

06 Apr '18

This is a multi-part message in MIME format. ------=_001_NextPart537252258082_=---- Content-Type: text/plain; charset="GB2312" Content-Transfer-Encoding: base64 SGkgR3V5cyENCkkgcGxhbiB0byBjb25zdHJ1Y3QgYW4gbG9jYWwgZGV2ZWxvcGluZyBlbnZpcm9u bWVudCBmb3Igb1ZpcnQuDQpCdXQgSSBjYW4gbm90IGZpbmQgYW55IHJlbGF0aXZlIHBhZ2VzIGZy b20gdGhlIHdlYiBzaXRlKHd3dy5vdmlydC5vcmcpLg0KTWF5YmUgSSBhbSBub3QgcGF0aWVudCA6 KS4NCg0KQ291bGQgeW91IHBsZWFzZSBnaXZlIG1lIHNvbWUgYWR2aWNlIG9yIHNvbWUgbGlua3Mg YWJvdXQgSG93IFRvIENvbnN0cnVjdCBBbiBMb2NhbCBEZXZlbG9waW5nIEVudmlyb21lbnQ/DQoN ClRoYW5rcyEgDQoNCg0KDQrL77TzzqENCrGxvqm/xtL4vqmzyby8yvXT0M/euavLvi/QwrL6xrfR 0Leisr8NCjEzMzc4MTA1NjI1DQo= ------=_001_NextPart537252258082_=---- Content-Type: text/html; charset="GB2312" Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse= t=3DGB2312"><style>body { line-height: 1.5; }body { font-size: 10.5pt; fon= t-family: =CE=A2=C8=ED=D1=C5=BA=DA; color: rgb(0, 0, 0); line-height: 1.5;= }</style></head><body>=0A<div><span></span><span style=3D"color: rgb(0, 0= , 0); background-color: rgba(0, 0, 0, 0);">Hi Guys!<br>I plan&nb= sp;to construct an local developing environment for&nb= sp;oVirt.<br>But I can not find any relative=  pages from the web site(www.ovirt.org).<br>Maybe&nbsp= ;I am not patient :).<br><br>Could you pleas= e give me some advice or some links abo= ut </span><span style=3D"background-color: rgba(0, 0, 0, 0); font-siz= e: 10.5pt; line-height: 1.5;">How To Construct An Loca= l Developing Enviroment</span><span style=3D"background-color: r= gba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">?</span></div><div>= <span style=3D"color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"><= br>Thanks! </span></div>=0A<div><br></div><hr style=3D"width: 210px; = height: 1px;" color=3D"#b5c4df" size=3D"1" align=3D"left">=0A<div><span><d= iv style=3D"margin: 10px; font-size: 21px;"><b><font face=3D"=BB=AA=CE=C4= =D0=C2=CE=BA">=CB=EF=B4=F3=CE=A1</font></b></div><div style=3D"MARGIN: 10p= x; FONT-FAMILY: verdana; FONT-SIZE: 10pt">=B1=B1=BE=A9=BF=C6=D2=F8=BE=A9= =B3=C9=BC=BC=CA=F5=D3=D0=CF=DE=B9=AB=CB=BE/=D0=C2=B2=FA=C6=B7=D1=D0=B7=A2= =B2=BF</div><div style=3D"MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 1= 0pt">13378105625</div></span></div>=0A</body></html> ------=_001_NextPart537252258082_=------

6 6

Re: [ovirt-devel] [oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 143 - Still Failing!
by Sandro Bonazzola 05 Apr '18

05 Apr '18

Adding devel mailing list and some relevant people. 2018-04-05 15:41 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com>: > The first error is here: > > > *13:25:37* [ INFO ] TASK [Wait for the engine to come up on the target VM]*13:37:01* [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.316574", "end": "2018-04-05 09:37:00.826888", "rc": 0, "start": "2018-04-05 09:37:00.510314", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1838 (Thu Apr 5 09:36:51 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=1838 (Thu Apr 5 09:36:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Wed Dec 31 19:31:35 1969\\n\", \"hostname\": \"lago-he-basic-ansible-suite-master-host0\", \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"72e3e0e9\", \"local_conf_timestamp\": 1838, \"host-ts\": 1838}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=1838 (Thu Apr 5 09:36:51 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=1838 (Thu Apr 5 09:36:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Wed Dec 31 19:31:35 1969\\n\", \"hostname\": \"lago-he-basic-ansible-suite-master-host0\", \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"72e3e0e9\", \"local_conf_timestamp\": 1838, \"host-ts\": 1838}, \"global_maintenance\": false}"]}*13:37:01* [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook > > > Anso so we have to check VDSM logs where we can find: > > 2018-04-05 09:27:01,083-0400 ERROR (vm/9bb2deab) [virt.vm] (vmId='9bb2deab-f686-4629-acee-90e2774a66b6') The vm start process failed (vm:945) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 874, in _startUnderlyingVm > self._run() > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2832, in _run > domxml = hooks.before_vm_start(self._buildDomainXML(), > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2258, in _buildDomainXML > return vmxml.format_xml(dom, pretty=True) > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmxml.py", line 71, in format_xml > stream, encoding='utf-8', xml_declaration=True) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 820, in write > serialize(write, self._root, encoding, qnames, namespaces) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml > _serialize_xml(write, e, encoding, qnames, None) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml > _serialize_xml(write, e, encoding, qnames, None) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml > _serialize_xml(write, e, encoding, qnames, None) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 932, in _serialize_xml > v = _escape_attrib(v, encoding) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1092, in _escape_attrib > _raise_serialization_error(text) > File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1052, in _raise_serialization_error > "cannot serialize %r (type %s)" % (text, type(text).__name__) > TypeError: cannot serialize 0 (type int) > 2018-04-05 09:27:01,084-0400 INFO (vm/9bb2deab) [virt.vm] (vmId='9bb2deab-f686-4629-acee-90e2774a66b6') Changed state to Down: cannot serialize 0 (type int) (code=1) (vm:1685) > > > It seams another regression with the libvirt XML generated by the engine. > > > > On Thu, Apr 5, 2018 at 3:37 PM, <jenkins(a)jenkins.phx.ovirt.org> wrote: > >> Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans >> ible-suite-master/ >> Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ans >> ible-suite-master/143/ >> Build Number: 143 >> Build Status: Still Failing >> Triggered By: Started by user Simone Tiraboschi >> >> ------------------------------------- >> Changes Since Last Success: >> ------------------------------------- >> Changes for Build #139 >> [Gal Ben Haim] he: Share control.sh between all the non-ansible suites >> >> [Daniel Belenky] Workaround for a race in gerrit-trigger plugin >> >> >> Changes for Build #140 >> [Gal Ben Haim] he: Share control.sh between all the non-ansible suites >> >> >> Changes for Build #141 >> [Gal Ben Haim] he: Share control.sh between all the non-ansible suites >> >> >> Changes for Build #142 >> [Yaniv Kaul] Disable 006_migrations.prepare_migration_attachments_ipv6 >> >> >> Changes for Build #143 >> [Yaniv Kaul] Disable 006_migrations.prepare_migration_attachments_ipv6 >> >> >> >> >> ----------------- >> Failed Tests: >> ----------------- >> No tests ran. > > > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

2 1

[vdsm][ci] EL7 failing for timeout lately
by Francesco Romani 04 Apr '18

04 Apr '18

Hi all, in the last few days quite a lot of CI tests on LE7 workers failed for timeout. Random example: http://jenkins.ovirt.org/job/vdsm_4.2_check-patch-el7-x86_64/246/consoleFull This happened both on 4.2 and on master test suite, quite often but not always. I wonder if we should just increase the timeout or if something else is going on. Thoughts? -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

2 1

oVirt log analyzer
by Milan Zamazal 04 Apr '18

04 Apr '18

Hi, during last year Outreachy internship a tool for analyzing oVirt logs was created. When it is provided with oVirt logs (such as SOS reports, logs gathered by Lago, single or multiple log files) it tries to identify and classify important lines from the logs and present them in a structured form. Its primary purpose is to get a quick and easy overview of actions and errors. The tool analyses given logs and produces text files with the extracted information. There is an Emacs user interface that presents the output in a nice way with added functionality such as filtering. Emacs haters can use the plain text files or write another user interface. :-) You can get ovirt-log-analyzer from https://github.com/mz-pdm/ovirt-log-analyzer README.md explains how to use it. Note that ovirt-log-analyzer has been created within the limited resources of an Outreachy internship with some additional work and not everything is perfect. Feel free to make improvements. Regards, Milan

6 10

[ OST Failure Report ] [ oVirt 4.2 ] [ 2018-04-03 ] [006_migrations.prepare_migration_attachments_ipv6]
by Barak Korren 04 Apr '18

04 Apr '18

Test failed: [ 006_migrations.prepare_migration_attachments_ipv6 ] Link to suspected patches: (Patch seems unrelated - do we have sporadic communication issues arising in PST?) https://gerrit.ovirt.org/c/89737/1 - vdsm - automation: check-patch: attempt to install vdsm-gluster Link to Job: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1521/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1521/artifact/ex… Error snippet from log: <error> Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/006_migrations.py", line 139, in prepare_migration_attachments_ipv6 engine, host_service, MIGRATION_NETWORK, ip_configuration) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test_utils/network_utils_v4.py", line 71, in modify_ip_config check_connectivity=True) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36729, in setup_networks return self._internal_action(action, 'setupnetworks', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

4 7

Lago v0.43 Release Announcement
by Gal Ben Haim 03 Apr '18

03 Apr '18

Hi All, On behalf of the Lago team, I'm pleased to announce Lago v0.43 is available! For a detailed release announcement please refer to our Github page at: https://github.com/lago-project/lago/releases/tag/0.43 Resources ========== Lago Docs: http://lago.readthedocs.io/en/latest/ GitHub: https://github.com/lago-project/lago/ YUM Repository: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ Pypi: https://pypi.python.org/pypi/lago Changelog: http://lago.readthedocs.io/en/0.43/_static/ChangeLog.txt As always, if you find any problems, or willing to contribute, visit our GitHub page. Enjoy! -- *GAL bEN HAIM* RHV DEVOPS

1 0

Failed To Build ovirt-node following the guide
by sundw 02 Apr '18

02 Apr '18

This is a multi-part message in MIME format. ------=_001_NextPart058565458615_=---- Content-Type: text/plain; charset="GB2312" Content-Transfer-Encoding: base64 SGVsbG8sZ3V5c6OhDQoNCkkgYnVpbHQgb3ZpcnQtbm9kZSBmb2xsb3dpbmcgdGhpcyBndWlkZSho dHRwczovL3d3dy5vdmlydC5vcmcvZGV2ZWxvcC9wcm9qZWN0cy9ub2RlL2J1aWxkaW5nLykuDQpC dXQgSSBmYWlsZWQuDQpJIGdvdCB0aGUgZm9sbG93aW5nIEVycm9yIG1lc3NhZ2UgYWZ0ZXIgcnVu bmluZyAibWFrZSBpc28gcHVibGlzaCI6DQoNCkVycm9yIGNyZWF0aW5nIExpdmUgQ0QgOiBGYWls ZWQgdG8gZmluZCBwYWNrYWdlICdvdmlydC1ub2RlLXBsdWdpbi12ZHNtJyA6IE5vIHBhY2thZ2Uo cykgYXZhaWxhYmxlIHRvIGluc3RhbGwNCg0KQ291bGQgeW91IHBsZWFzZSBnaXZlIHNvbWUgYWR2 aWNlPw0KDQpCVFc6IElzIHRoZSBjb250ZW50IG9mIHRoaXMgdXJsICJodHRwczovL3d3dy5vdmly dC5vcmcvZGV2ZWxvcC9wcm9qZWN0cy9ub2RlL2J1aWxkaW5nLyIgT1VUIE9GIERBVEU/DQoNCg0K DQrL77TzzqENCrGxvqm/xtL4vqmzyby8yvXT0M/euavLvi/QwrL6xrfR0Leisr8NCjEzMzc4MTA1 NjI1DQo= ------=_001_NextPart058565458615_=---- Content-Type: text/html; charset="GB2312" Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse= t=3DGB2312"><style>body { line-height: 1.5; }body { font-size: 10.5pt; fon= t-family: =CE=A2=C8=ED=D1=C5=BA=DA; color: rgb(0, 0, 0); line-height: 1.5;= }</style></head><body>=0A<div><span></span><span style=3D"color: rgb(0, 0= , 0); background-color: rgba(0, 0, 0, 0);">Hello,guys=A3=A1</span></div><d= iv><span style=3D"color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);= "><br>I built ovirt-node following this guide(htt= ps://www.ovirt.org/develop/projects/node/building/)<br>But I fa= iled.<br>I got the following Error message after = running "make iso publish":<br><br>Error creating&nbsp= ;Live CD : Failed to find package 'ovir= t-node-plugin-vdsm' : No package(s) available to&= nbsp;install<br><br>Could you please give some ad= vice?</span></div><div><span style=3D"color: rgb(0, 0, 0); background-colo= r: rgba(0, 0, 0, 0);"><br></span></div><div><b><span style=3D"color: rgb(0= , 0, 0); background-color: rgba(0, 0, 0, 0);">BTW: Is the content of this = url "</span><span style=3D"font-size: 10.5pt; line-height: 1.5; background= -color: window;">https://www.ovirt.org/develop/projects/node/building/</sp= an><span style=3D"background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; l= ine-height: 1.5;">" OUT OF DATE?</span></b></div><div><br></div><hr style= =3D"width: 210px; height: 1px;" color=3D"#b5c4df" size=3D"1" align=3D"left= ">=0A<div><span><div style=3D"margin: 10px; font-size: 21px;"><b><font fac= e=3D"=BB=AA=CE=C4=D0=C2=CE=BA">=CB=EF=B4=F3=CE=A1</font></b></div><div sty= le=3D"MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">=B1=B1=BE=A9=BF= =C6=D2=F8=BE=A9=B3=C9=BC=BC=CA=F5=D3=D0=CF=DE=B9=AB=CB=BE/=D0=C2=B2=FA=C6= =B7=D1=D0=B7=A2=B2=BF</div><div style=3D"MARGIN: 10px; FONT-FAMILY: verdan= a; FONT-SIZE: 10pt">13378105625</div></span></div>=0A</body></html> ------=_001_NextPart058565458615_=------

3 2

Build virt-node on fedora26 failed
by sundw 02 Apr '18

02 Apr '18

This is a multi-part message in MIME format. ------=_001_NextPart018013562620_=---- Content-Type: text/plain; charset="GB2312" Content-Transfer-Encoding: base64 SGkhDQpJIGJ1aWxkIHZpcnQtbm9kZSBiYXNlIG9uIHRoaXMgZ3VpZGUoaHR0cHM6Ly93d3cub3Zp cnQub3JnL2RldmVsb3AvcHJvamVjdHMvbm9kZS9idWlsZGluZy8pLg0KSSBnb3QgdGhlIGZvbGxv d2luZyBlcnJvciBtZXNzYWdlcyBhZnRlciBJIHJ1biAibWFrZSBwdWJsaXNoIjoNCg0KSW5zdGFs bGluZyB0bXAucmVwb3MvU1JQTVMvb3ZpcnQtbm9kZS0zLjcuMC0wLjAubWFzdGVyLmZjMjcuc3Jj LnJwbQ0KZXJyb3I6IEZhaWxlZCBidWlsZCBkZXBlbmRlbmNpZXM6DQovdXNyL3NoYXJlL3NlbGlu dXgvZGV2ZWwvcG9saWN5aGVscCBpcyBuZWVkZWQgYnkgb3ZpcnQtbm9kZS0zLjcuMC0wLjAubWFz dGVyLmZjMjcubm9hcmNoDQptYWtlWzFdOiAqKiogW01ha2VmaWxlOjgxMzogcnBtXSBFcnJvciAx DQptYWtlWzFdOiBMZWF2aW5nIGRpcmVjdG9yeSAnL2hvbWUvY29yZXRlay9naXQvb3ZpcnQtbm9k ZScNCm1ha2U6ICoqKiBbTWFrZWZpbGU6ODI3OiBwdWJsaXNoXSBFcnJvciAyDQoNCkNhbiBhbnlv bmUgZ2l2ZSBtZSBzb21lIHN1Z2dlc3Rpb25zPyANClRoYW5rcyENCg0KDQoNCg0Ky++0886hDQqx sb6pv8bS+L6ps8m8vMr109DP3rmry74v0MKy+sa30dC3orK/DQoxMzM3ODEwNTYyNQ0K ------=_001_NextPart018013562620_=---- Content-Type: text/html; charset="GB2312" Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse= t=3DGB2312"><style>body { line-height: 1.5; }body { font-size: 10.5pt; fon= t-family: =CE=A2=C8=ED=D1=C5=BA=DA; color: rgb(0, 0, 0); line-height: 1.5;= }</style></head><body>=0A<div><span></span>Hi!</div><div>I build virt-nod= e base on this guide(<span style=3D"background-color: rgba(0, 0, 0, 0); fo= nt-size: 10.5pt; line-height: 1.5;">https://www.ovirt.org/develop/projects= /node/building/</span><span style=3D"font-size: 10.5pt; line-height: 1.5; = background-color: window;">).</span></div><div>I got the following error m= essages after I run "make publish":</div><div><br></div><div><span style= =3D"color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">Installing&n= bsp;tmp.repos/SRPMS/ovirt-node-3.7.0-0.0.master.fc27.src.rpm<br>error:&nbs= p;Failed build dependencies:<br><span style=3D"white-space: pre;= "> </span>/usr/share/selinux/devel/policyhelp is needed by&= nbsp;ovirt-node-3.7.0-0.0.master.fc27.noarch<br>make[1]: *** [Ma= kefile:813: rpm] Error 1<br>make[1]: Leaving dire= ctory '/home/coretek/git/ovirt-node'<br>make: *** [Makefile= :827: publish] Error 2</span></div><div><span style=3D"colo= r: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"><br></span></div><di= v><span style=3D"color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"= >Can anyone give me some suggestions? </span></div><div><span style= =3D"color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">Thanks!</spa= n></div><div><br></div>=0A<div><br></div><hr style=3D"width: 210px; height= : 1px;" color=3D"#b5c4df" size=3D"1" align=3D"left">=0A<div><span><div sty= le=3D"margin: 10px; font-size: 21px;"><b><font face=3D"=BB=AA=CE=C4=D0=C2= =CE=BA">=CB=EF=B4=F3=CE=A1</font></b></div><div style=3D"MARGIN: 10px; FON= T-FAMILY: verdana; FONT-SIZE: 10pt">=B1=B1=BE=A9=BF=C6=D2=F8=BE=A9=B3=C9= =BC=BC=CA=F5=D3=D0=CF=DE=B9=AB=CB=BE/=D0=C2=B2=FA=C6=B7=D1=D0=B7=A2=B2=BF<= /div><div style=3D"MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">13= 378105625</div></span></div>=0A</body></html> ------=_001_NextPart018013562620_=------

2 1

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 27-03-2018 ] [ 004_basic_sanity.disk_operations ]
by Dafna Ron 02 Apr '18

02 Apr '18

Hi, We have a failure on 004_basic_sanity.disk_operations. the reported change seems to me to be connected to the failure. *Link and headline of suspected patches: core: introduce CreateSnapshotForVm - https://gerrit.ovirt.org/#/c/87671/ <https://gerrit.ovirt.org/#/c/87671/>Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6564/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6564/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6564/art… <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6564/artifact…>(Relevant) error snippet from the log: <error>2018-03-27 11:30:16,167-04 WARN [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (default task-11) [9f713a3d-82a9-4db8-8207-bb69a0a2c550] Validation of action 'CreateSnapshotForVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__CREATE,VAR__TYPE__SNAPSHOT,ACTION_TYPE_FAILED_SNAPSHOT_IS_BEING_TAKEN_FOR_VM,$VmName vm12018-03-27 11:30:16,176-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-11) [9f713a3d-82a9-4db8-8207-bb69a0a2c550] method: runAction, params: [CreateSnapshotForVm, CreateSnapshotForVmParameters:{commandId='d8bfdb0e-5ae4-4fed-8852-a0b29e377708', user='null', commandType='Unknown', vmId='65e7fea5-8b7d-4281-bd0d-6a84de207a00'}], timeElapsed: 57ms2018-03-27 11:30:16,186-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-11) [] Operation Failed: [Cannot create Snapshot. Snapshot is currently being created for VM vm1.]2018-03-27 11:30:16,820-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [16c08ce6-9a73-489f-a1f2-56e21699f14a] Command 'CopyImageGroupVolumesData' (id: '4c4023b4-21e8-4a98-8ac3-aba51967d160') waiting on child command id: '3d96b933-6e3b-4838-94bc-db3ff0cadcdc' type:'CopyData' to complete2018-03-27 11:30:16,833-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [16c08ce6-9a73-489f-a1f2-56e21699f14a] method: get, params: [a85992e9-40b8-4c80-9c93-3bf767f6efa4], timeElapsed: 8ms</error>*

4 9

[ OST Failure Report ] [ oVirt 4.2 ] [ 2018-04-01 ] [007_sd_reattach.reattach_storage_domain]
by Barak Korren 01 Apr '18

01 Apr '18

Test failed: [ 007_sd_reattach.reattach_storage_domain ] Link to suspected patches: https://gerrit.ovirt.org/c/89634/2 (ovirt-imageio): daemon: Fix OPTIONS to reflect actual capabilities Link to Job: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1502/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1502/artifact/ex… Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0