- Devel - oVirt List Archives

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 28-02-2019 ] [ 002_bootstrap.resize_and_refresh_storage_domain ]
by Dafna Ron 28 Feb '19

28 Feb '19

Hi, The test failed due to not finding master storage domain I am seeing a build of engine that passed so its possible that this is related to the random failures we have been having on other tests but since the change reported is storage related, can you please review the logs to see if anything happened to the master domain while trying to resize? CQ reported this change as root cause: https://gerrit.ovirt.org/#/c/98015/6 - blockSD: Move legacy block size constants to sd build failure: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13175/ logs: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13175/artifa… errors in log: 2019-02-27 17:35:02,222-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM la go-basic-suite-master-host-1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' 2019-02-27 17:35:02,222-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker. ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5 a9']]' 2019-02-27 17:35:02,222-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] HostName = lago-basic-suite-master-host-1 2019-02-27 17:35:02,223-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command 'ConnectStoragePoolVDSCommand(HostName = lago-basic-suite-master-host-1, ConnectStoragePoolVDSCommandParameters:{hostId='1ac0eda2-0980-40df-b029-b262680696a4', vdsId='1ac0eda2-0980-40df-b029-b262680696a4', storagePoolId='1c79b83e-32d6-43e0-bc61-2955e42c3a58', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' 2019-02-27 17:35:02,223-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:50) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedConnectProxyReturnValue(ConnectStoragePoolVDSCommand.java:48) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedProxyReturnValue(ConnectStoragePoolVDSCommand.java:36) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.executeVdsBrokerCommand(ConnectStoragePoolVDSCommand.java:32) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source) [:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation.execute(RefreshPoolSingleAsyncOperation.java:59) [bll.jar:] at org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:43) [utils.jar:] at org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:32) [utils.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:117) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:]

2 2

[ANN] oVirt 4.3.1 is now generally available
by Sandro Bonazzola 28 Feb '19

28 Feb '19

The oVirt Project is pleased to announce the general availability of oVirt 4.3.1, as of February 26th, 2019. This update is the first in a series of stabilization updates to the 4.3 series. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is available for EL7 only - oVirt Node is available for EL7 only [2] - Fedora 28 based appliance and node couldn't be built due to a bug in Lorax (the tool used to build the images) affecting Fedora 28 and still pending a build from Fedora project. oVirt Node has been updated including: - oVirt 4.3.1: http://www.ovirt.org/release/4.3.1/ - ansible-2.7.8: https://github.com/ansible/ansible/blob/stable-2.7/changelogs/CHANGELOG-v2.… Relevant EPEL Errata included: - FEDORA-EPEL-2019-5f2f93029f security update in Fedora EPEL 7 for ansible ( https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-5f2f93029f ) Relevant CentOS Errata included: - CESA-2019:0231 Important CentOS 7 spice Security Update ( https://lists.centos.org/pipermail/centos-announce/2019-February/023192.html ) - CESA-2019:0368 Important CentOS 7 systemd Security Update ( https://lists.centos.org/pipermail/centos-announce/2019-February/023202.html ) Additional Resources: * Read more about the oVirt 4.3.1 release highlights: http://www.ovirt.org/release/4.3.1/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.1/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

OST manual: enable engine 4.3
by Nir Soffer 28 Feb '19

28 Feb '19

When starting manual OST job, we can select engine version. The available options are: - master - 42 Since we have now 4.3 branch, we should be able to select 4.3 engine. See attached screenshot.

3 2

Vdsm branch ovirt-4.3 created
by Milan Zamazal 27 Feb '19

27 Feb '19

Hi, I created ovirt-4.3 branch. That means patches for 4.3 must be backported from master since now. Not everything regarding CI is ready yet, but until we make the branch, we can't make some things working nor prove they are working. We need the branch to move forward. *Please don't merge non-4.3 patches to master yet.* Let's make 4.3 backports completely safe and trivial for a while, until we have working CI on 4.3. I'm not particularly happy with the branching process, let's use this experience to improve it next time. At minimum some guidelines should exist rather than just "common knowledge" with missing bits and pieces. Thanks, Milan

5 11

broken OST network suite (?)
by Miguel Duarte de Mora Barroso 26 Feb '19

26 Feb '19

Hi, I'm seeing the following error on the OST network suite: 17:05:56 Collecting pylint==1.6.4 17:05:57 Downloading https://files.pythonhosted.org/packages/92/f3/41deb50322d579517f779c3421b92… (569kB) 17:05:57 Collecting isort>=4.2.5 (from pylint==1.6.4) 17:05:57 Downloading https://files.pythonhosted.org/packages/35/c5/025aaa76f0f337e3d91197973c42b… (62kB) 17:05:58 Complete output from command python setup.py egg_info: 17:05:58 /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' 17:05:58 warnings.warn(msg) 17:05:58 error in isort setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in futures; python_version < "3.2" at ; python_version < "3.2" This happens on - at least - [0]. From my understanding, isort released a couple of new versions - according to [1] - and looks like that pylint version is too old for whatever new stuff isort now has. On other project I'm running pylint 1.8.4, which runs fine against that isort version: tox -e pylint pylint recreate: /home/mduarted/ovirt/ovirt-provider-ovn/.tox/pylint pylint installdeps: pytest==3.5.1, pylint==1.8.4, -r/home/mduarted/ovirt/ovirt-provider-ovn/requirements.txt pylint installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,astroid==1.6.5,attrs==18.2.0,backports.functools-lru-cache==1.5,certifi==2018.11.29,chardet==3.0.4,configparser==3.7.3,enum34==1.1.6,extras==1.0.0,fixtures==3.0.0,funcsigs==1.0.2,futures==3.2.0,idna==2.8,isort==4.3.9,lazy-object-proxy==1.3.1,linecache2==1.0.0,mccabe==0.6.1,more-itertools==5.0.0,netaddr==0.7.19,ovs==2.10.0,ovsdbapp==0.15.0,pbr==5.1.2,pluggy==0.6.0,py==1.8.0,pylint==1.8.4,pytest==3.5.1,python-mimeparse==1.6.0,requests==2.21.0,singledispatch==3.4.0.3,six==1.12.0,sortedcontainers==2.1.0,testtools==2.3.0,traceback2==1.4.0,unittest2==1.1.0,urllib3==1.24.1,wrapt==1.11.1 pylint runtests: PYTHONHASHSEED='2585806524' pylint runtests: commands[0] | pylint --errors-only provider driver Using config file /home/mduarted/ovirt/ovirt-provider-ovn/.pylintrc make lint3____________________________________ summary _____________________________________ pylint: commands succeeded congratulations :) Pyling 2.2.2 also runs fine. Is there a reason for us to be using such an outdated version ? [0] - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4155/console [1] - https://pypi.org/project/isort/#history

1 1

[VDSM] Simplifying project ACL
by Nir Soffer 26 Feb '19

26 Feb '19

Hi all, I learned recently that we have these groups in the project: vdsm-maintainers: https://gerrit.ovirt.org/#/admin/groups/15,members vdsm-master-maintainers: https://gerrit.ovirt.org/#/admin/groups/106,members And we have this complicated access control list: https://gerrit.ovirt.org/#/admin/projects/vdsm,access Developers in vdsm-master-maintainers can approve patches, but cannot submit. I find this strange and more complicated than needed. I think we can trust people in the vdsm-master-maintainers group to do the right thing. I sent this patch to simplify the ACL (with help of Gerrit UI): https://gerrit.ovirt.org/c/97738/ Thoughts? Nir

2 2

Misc configuration': Failed to start service 'rh-postgresql95-postgresql
by Galit Rosenthal 24 Feb '19

24 Feb '19

Hi I receive this error message both in CQ and check_patch: 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout: 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details. 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc self._provisioning.provision() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision self.restartPG() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG state=state, File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe What can cause it? Thanks Galit https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_sta… Regards, Galit -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)gmail.com T: 972-9-7692230 <https://red.ht/sig>

12 33

OST failes since build 4134
by Nir Soffer 23 Feb '19

23 Feb '19

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/buildTimeTrend Here are the failures since build 4134: - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4135/testReport/jun… Not related to the patch tested: https://gerrit.ovirt.org/c/97996/ I started a new build: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4140/ - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4136/ (no test results) - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4137/ (no tests results) - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4138/ (no test results) - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4139/testReport/jun… UI test failure/error? Nir

3 3

Fwd: Planned Outage - Fedora Core Services 2019-02-27 20:00 UTC
by Sandro Bonazzola 22 Feb '19

22 Feb '19

FYI ---------- Forwarded message --------- From: Stephen John Smoogen <smooge(a)gmail.com> Date: ven 22 feb 2019 alle ore 20:43 Subject: Planned Outage - Fedora Core Services 2019-02-27 20:00 UTC To: <announce(a)lists.fedoraproject.org>, Development discussions related to Fedora <devel(a)lists.fedoraproject.org>, Fedora Infrastructure < infrastructure(a)lists.fedoraproject.org> There will be an outage starting at 2019-02-27 20:00 UTC, which will last approximately 4-6 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2019-02-27 20:00UTC' Reason for outage: Fedora Infrastructure would like to update and reboot all core systems and services. This will update kernels, glibc, and systemd plus many other services on the affected systems. Affected Services: All systems under fedoraproject.org will be affected. Important databases and authentication will be updated and rebooted first. Then other non-build systems will follow. Then build systems will finalize the update/reboot cycle Ticket Link: https://pagure.io/fedora-infrastructure/issue/7591 Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above. -- Stephen J Smoogen. _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

[Ovirt] [CQ weekly status] [22-02-2019]
by Dafna Ron 22 Feb '19

22 Feb '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) 1. rh-postgress was failing to start due to missing selinux policy causing engine to fail installation. Bug: https://bugzilla.redhat.com/show_bug.cgi Issue resolved by didi and Galit by: 1. adding a workaround to add the selinuc poslicy and also a proposed fix (as described in comment #6) 2. created new lago images with new selinux package *CQ-Master:* GREEN (#1) 1. The same issue in 4.2 has effected master (for upgrade suite) Current running jobs for 4.2 [1] and master [2] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0