exactly - just posted a patch again that sings all fails as broken . 
we'll get the report soon and I'll publish it as well. hope the run will take less time now

On Wed, Dec 16, 2015 at 6:19 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Nice, but we cannot enable this until all the tests pass or disabled.

There is no point in broken or flaky functional tests.

On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei@redhat.com> wrote:
> So its not stable. It won't block merges and at least give us report after
> each merge. It takes really long time to run it (because of the tests
> themselves. Lago things takes maximum 15minutes, but the run last for more
> than 2hrs right now and I suspect functional/storageTests.py gets stuck)
>
> Bellow you can see where we stand (before I added python-rtslib package).
>
> Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ -
> which enables this run after merges, and I still want you to consider the
> addition of Automation CI flag to our gerrit so that developer will be able
> to use it as a trigger for the check-merged.sh script run, just to see if
> their patch fixes\brakes something realted to the functional tests
>
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is
> an example of how the run looks like. I still work to improve the output
>
>
> Please reply and let me know if the idea around the automation flag is
> acceptable by you.. and please review the patch for comments and acks.
> We can ask dcaro to add the flag until Friday, otherwise we'll need to delay
> this effort after the holiday..
>
>
> functional.sosPluginTests.SosPluginTest
>     testSosPlugin                                               OK
> functional.vmRecoveryTests.RecoveryTests
>     test_vm_recovery                                            FAIL
> functional.vmQoSTests.VMQosTests
>     testSmallVMBallooning                                       FAIL
> functional.virtTests.VirtTest
>     testComplexVm                                               FAIL
>     testHeadlessVm                                              OK
>     testSimpleVm                                                FAIL
>     testVmDefinitionGraphics('spice')                           FAIL
>     testVmDefinitionGraphics('vnc')                             OK
>     testVmDefinitionLegacyGraphics('qxl')                       FAIL
>     testVmDefinitionLegacyGraphics('vnc')                       OK
>     testVmDefinitionMultipleGraphics('spice', 'vnc')            FAIL
>     testVmDefinitionMultipleGraphics('vnc', 'spice')            FAIL
>     testVmWithCdrom('self')                                     FAIL
>     testVmWithCdrom('specParams')                               FAIL
>     testVmWithCdrom('vmPayload')                                FAIL
>     testVmWithDevice('hotplugDisk')                             FAIL
>     testVmWithDevice('hotplugNic')                              FAIL
>     testVmWithDevice('smartcard')                               FAIL
>     testVmWithDevice('virtioNic')                               FAIL
>     testVmWithDevice('virtioRng')                               FAIL
>     testVmWithSla                                               FAIL
>     testVmWithStorage('iscsi')                                  SKIP:
> python-rtslib is not installed.
>     testVmWithStorage('localfs')                                FAIL
>     testVmWithStorage('nfs')                                    FAIL
> functional.storageTests.StorageTest
>     testCreatePoolErrors                                        OK
>     testStorage('glusterfs', 0)                                 ERROR
>     testStorage('glusterfs', 3)                                 ERROR
>     testStorage('iscsi', 0)                                     SKIP:
> python-rtslib is not installed.
>     testStorage('iscsi', 3)                                     SKIP:
> python-rtslib is not installed.
>     testStorage('localfs', 0)                                   FAIL
>     testStorage('localfs', 3)                                   FAIL
>     testStorage('nfs', 0)                                       FAIL
>     testStorage('nfs', 3)                                       FAIL
> functional.networkTests.NetworkTest
>     testAddVlanedBridgeless                                     ERROR
>     testAddVlanedBridgeless_oneCommand                          ERROR
>     testAfterNetworkSetupHook                                   ERROR
>     testBeforeNetworkSetupHook                                  ERROR
>     testBondHwAddress(False)                                    ERROR
>     testBondHwAddress(True)                                     ERROR
>     testBrokenNetworkReplacement(False)                         ERROR
>     testBrokenNetworkReplacement(True)                          ERROR
>     testDelNetworkBondAccumulation                              ERROR
>     testDelNetworkWithMTU(False)                                ERROR
>     testDelNetworkWithMTU(True)                                 ERROR
>     testDelWithoutAdd                                           ERROR
>     testDhclientLeases(4, 'default')                            ERROR
>     testDhclientLeases(4, 'local')                              ERROR
>     testDhclientLeases(6, None)                                 ERROR
>     testDhcpReplaceNicWithBridge                                ERROR
>     testFailWithInvalidBondingName(False)                       ERROR
>     testFailWithInvalidBondingName(True)                        ERROR
>     testFailWithInvalidBridgeName                               ERROR
>     testFailWithInvalidIpConfig                                 ERROR
>     testFailWithInvalidNic(False)                               ERROR
>     testFailWithInvalidNic(True)                                ERROR
>     testFailWithInvalidParams(False)                            ERROR
>     testFailWithInvalidParams(True)                             ERROR
>     testGetRouteDeviceTo                                        ERROR
>     testHonorBlockingDhcp                                       SKIP: Slow
> tests are disabled
>     testIpLinkWrapper                                           ERROR
>     testLowerMtuDoesNotOverride                                 FAIL
>     testNoBridgeLeftovers                                       FAIL
>     testReconfigureBrNetWithVanishedPort                        ERROR
>     testRedefineBondedNetworkIPs                                ERROR
>     testRemovingBridgeDoesNotLeaveBridge                        ERROR
>     testReorderBondingOptions(False)                            ERROR
>     testReorderBondingOptions(True)                             ERROR
>     testRestoreNetworksOnlyRestoreUnchangedDevices              ERROR
>     testRestoreToBlockingDHCP                                   ERROR
>     testSafeNetworkConfig(False)                                ERROR
>     testSafeNetworkConfig(True)                                 ERROR
>     testSelectiveRestoreDuringUpgrade                           ERROR
>     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
>     testSetupNetworkOutboundQos(False)                          ERROR
>     testSetupNetworkOutboundQos(True)                           FAIL
>     testSetupNetworksActiveSlave                                ERROR
>     testSetupNetworksAddBadParams(False)                        ERROR
>     testSetupNetworksAddBadParams(True)                         FAIL
>     testSetupNetworksAddBondWithManyVlans(False)                ERROR
>     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
>     testSetupNetworksAddDelBondedNetwork(False)                 ERROR
>     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
>     testSetupNetworksAddDelDhcp(False, (4, 6))                  ERROR
>     testSetupNetworksAddDelDhcp(False, (4,))                    ERROR
>     testSetupNetworksAddDelDhcp(False, (6,))                    ERROR
>     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
>     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
>     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
>     testSetupNetworksAddManyVlans(False)                        ERROR
>     testSetupNetworksAddManyVlans(True)                         FAIL
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(False)    ERROR
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
>     testSetupNetworksAddNetworkToNicAfterBondResizing(False)    ERROR
>     testSetupNetworksAddNetworkToNicAfterBondResizing(True)     ERROR
>     testSetupNetworksAddOverExistingBond(False)                 ERROR
>     testSetupNetworksAddOverExistingBond(True)                  ERROR
>     testSetupNetworksAddVlan(False)                             ERROR
>     testSetupNetworksAddVlan(True)                              FAIL
>     testSetupNetworksConnectivityCheck                          FAIL
>     testSetupNetworksConvertVlanNetBridgeness                   ERROR
>     testSetupNetworksDelOneOfBondNets                           ERROR
>     testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured   ERROR
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(False)  ERROR
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
>     testSetupNetworksMtus(False)                                ERROR
>     testSetupNetworksMtus(True)                                 SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
>     testSetupNetworksMultiMTUsOverBond(False)                   ERROR
>     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
>     testSetupNetworksMultiMTUsOverNic(False)                    ERROR
>     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
>     testSetupNetworksNicless                                    FAIL
>     testSetupNetworksNiclessBridgeless                          ERROR
>     testSetupNetworksOverDhcpIface                              ERROR
>     testSetupNetworksOverExistingBridge                         FAIL
>     testSetupNetworksReconfigureBridge                          FAIL
>     testSetupNetworksRemoveBondWithKilledEnslavedNics           ERROR
>     testSetupNetworksRemoveSlavelessBond                        ERROR
>     testSetupNetworksResizeBond(False)                          ERROR
>     testSetupNetworksResizeBond(True)                           ERROR
>     testSetupNetworksStableBond(False)                          ERROR
>     testSetupNetworksStableBond(True)                           ERROR
>     testStaticNetworkConfig((4, 6))                             FAIL
>     testStaticNetworkConfig((4,))                               FAIL
>     testStaticNetworkConfig((6,))                               FAIL
>     testStaticSourceRouting(False)                              ERROR
>     testStaticSourceRouting(True)                               FAIL
>     testTwiceAdd(False)                                         ERROR
>     testTwiceAdd(True)                                          ERROR
>     testVolatileConfig(False)                                   ERROR
>     testVolatileConfig(True)                                    ERROR
>     test_drop_initial_bond_slaves_ip_config                     FAIL
>     test_getVdsStats                                            ERROR
>     test_remove_bond_under_network                              FAIL
>     test_remove_initial_network_nic_ip_config                   FAIL
>     test_rollback                                               FAIL
>     test_setupNetworks_bond_with_custom_option                  ERROR
>     test_setupNetworks_on_external_bond                         ERROR
>     test_setupNetworks_on_external_vlaned_bond                  ERROR
> functional.networkTestsOVS.OVSNetworkTest
>     testHonorBlockingDhcp                                       SKIP: Slow
> tests are disabled
>     testLowerMtuDoesNotOverride                                 OK
>     testNoBridgeLeftovers                                       FAIL
>     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
>     testSetupNetworkOutboundQos(True)                           OK
>     testSetupNetworksAddBadParams(True)                         OK
>     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
>     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
>     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
>     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
>     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
>     testSetupNetworksAddManyVlans(True)                         OK
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
>     testSetupNetworksAddVlan(True)                              FAIL
>     testSetupNetworksConnectivityCheck                          OK
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
>     testSetupNetworksMtus(True)                                 SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
>     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
>     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
>     testSetupNetworksNicless                                    OK
>     testSetupNetworksOverExistingBridge                         OK
>     testSetupNetworksReconfigureBridge                          FAIL
>     testStaticNetworkConfig((4, 6))                             FAIL
>     testStaticNetworkConfig((4,))                               FAIL
>     testStaticNetworkConfig((6,))                               FAIL
>     testStaticSourceRouting(True)                               FAIL
>     test_drop_initial_bond_slaves_ip_config                     FAIL
>     test_ovirtmgmtm_to_ovs                                      FAIL
>     test_remove_bond_under_network                              SKIP: OVS
> does not support bridgeless networks
>     test_remove_initial_network_nic_ip_config                   OK
>     test_rollback                                               FAIL
>
>
> On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew@redhat.com>
> wrote:
>>
>> I like the idea but I have the same feelings as Francesco. I think that we
>> need to make sure that functional tests for each vertical are stable before
>> enabling this process.
>>
>> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri@redhat.com> wrote:
>>>
>>> adding also infra team for visibility on the change in CI.
>>> also inline.
>>>
>>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani@redhat.com>
>>> wrote:
>>>>
>>>> ----- Original Message -----
>>>> > From: "Yaniv Bronheim" <ybronhei@redhat.com>
>>>> > To: devel@ovirt.org, "Francesco Romani" <fromani@redhat.com>, "Nir
>>>> > Soffer" <nsoffer@redhat.com>, "Piotr Kliczewski"
>>>> > <pkliczew@redhat.com>
>>>> > Cc: "danken" <danken@redhat.com>, "David Caro" <dcaroest@redhat.com>,
>>>> > "Eyal Edri" <eedri@redhat.com>
>>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
>>>> > Subject: Automation CI for vdsm
>>>>
>>>> [...]
>>>> > We want to allow developers to trigger the script once reviews and
>>>> > verification are ready (last step before merge). To do so we agreed to
>>>> > add
>>>> > Continues Integration flag for each vdsm patch.
>>>
>>>
>>> This flag will be called 'Workflow' or we can name it otherwise, we just
>>> need to choose what makes sense.
>>> David/Yaniv - Please correct me if I'm wrong.
>>>
>>>>
>>>> Once this flag will be
>>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
>>>> > script
>>>> > (adding new button to gerrit is not an option - you can image that
>>>> > flag as
>>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail
>>>> > we'll get -1 and once new patchset is ready the developer will remove
>>>> > the
>>>> > +1 and add it back to the Continues Integration flag to re-trigger the
>>>> > job.
>>>> >
>>>> > Please ack the process before we move on with that
>>>>
>>>> Sounds good, even though I'm a little scared (just gut feeling, no
>>>> evidence
>>>> whatsoever) that this could add even more complexity and fragility to
>>>> the jenkins
>>>> fleet.
>>>>
>>>> In the long run, when this is reliable, it will help greatly.
>>>> In the short term, I'm scared because this can lead to false positives
>>>> and bogus
>>>> failures.
>>>>
>>>> Let me stress I don't have concrete item to share or specific flaws.
>>>>
>>>> As action item on me, I will find some time next week to check virt
>>>> functional tests,
>>>> to see if they need some fixes, work reliably and so forth
>>>>
>>>> > The patch for those scripts still under review and testing -
>>>> > https://gerrit.ovirt.org/#/c/48268
>>>>
>>>> Will review asap.
>>>>
>>>> --
>>>> Francesco Romani
>>>> RedHat Engineering Virtualization R & D
>>>> Phone: 8261328
>>>> IRC: fromani
>>>
>>>
>>>
>>>
>>> --
>>> Eyal Edri
>>> Supervisor, RHEV CI
>>> EMEA ENG Virtualization R&D
>>> Red Hat Israel
>>>
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>>
>
>
>
> --
> Yaniv Bronhaim.



--
Yaniv Bronhaim.