So its not stable. It won't block merges and at least give us report after each merge. It takes really long time to run it (because of the tests themselves. Lago things takes maximum 15minutes, but the run last for more than 2hrs right now and I suspect functional/storageTests.py gets stuck)

Bellow you can see where we stand (before I added python-rtslib package). 

Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ - which enables this run after merges, and I still want you to consider the addition of Automation CI flag to our gerrit so that developer will be able to use it as a trigger for the check-merged.sh script run, just to see if their patch fixes\brakes something realted to the functional tests 

http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is an example of how the run looks like. I still work to improve the output


Please reply and let me know if the idea around the automation flag is acceptable by you.. and please review the patch for comments and acks.
We can ask dcaro to add the flag until Friday, otherwise we'll need to delay this effort after the holiday..


functional.sosPluginTests.SosPluginTest
    testSosPlugin                                               OK
functional.vmRecoveryTests.RecoveryTests
    test_vm_recovery                                            FAIL
functional.vmQoSTests.VMQosTests
    testSmallVMBallooning                                       FAIL
functional.virtTests.VirtTest
    testComplexVm                                               FAIL
    testHeadlessVm                                              OK
    testSimpleVm                                                FAIL
    testVmDefinitionGraphics('spice')                           FAIL
    testVmDefinitionGraphics('vnc')                             OK
    testVmDefinitionLegacyGraphics('qxl')                       FAIL
    testVmDefinitionLegacyGraphics('vnc')                       OK
    testVmDefinitionMultipleGraphics('spice', 'vnc')            FAIL
    testVmDefinitionMultipleGraphics('vnc', 'spice')            FAIL
    testVmWithCdrom('self')                                     FAIL
    testVmWithCdrom('specParams')                               FAIL
    testVmWithCdrom('vmPayload')                                FAIL
    testVmWithDevice('hotplugDisk')                             FAIL
    testVmWithDevice('hotplugNic')                              FAIL
    testVmWithDevice('smartcard')                               FAIL
    testVmWithDevice('virtioNic')                               FAIL
    testVmWithDevice('virtioRng')                               FAIL
    testVmWithSla                                               FAIL
    testVmWithStorage('iscsi')                                  SKIP: python-rtslib is not installed.
    testVmWithStorage('localfs')                                FAIL
    testVmWithStorage('nfs')                                    FAIL
functional.storageTests.StorageTest
    testCreatePoolErrors                                        OK
    testStorage('glusterfs', 0)                                 ERROR
    testStorage('glusterfs', 3)                                 ERROR
    testStorage('iscsi', 0)                                     SKIP: python-rtslib is not installed.
    testStorage('iscsi', 3)                                     SKIP: python-rtslib is not installed.
    testStorage('localfs', 0)                                   FAIL
    testStorage('localfs', 3)                                   FAIL
    testStorage('nfs', 0)                                       FAIL
    testStorage('nfs', 3)                                       FAIL
functional.networkTests.NetworkTest
    testAddVlanedBridgeless                                     ERROR
    testAddVlanedBridgeless_oneCommand                          ERROR
    testAfterNetworkSetupHook                                   ERROR
    testBeforeNetworkSetupHook                                  ERROR
    testBondHwAddress(False)                                    ERROR
    testBondHwAddress(True)                                     ERROR
    testBrokenNetworkReplacement(False)                         ERROR
    testBrokenNetworkReplacement(True)                          ERROR
    testDelNetworkBondAccumulation                              ERROR
    testDelNetworkWithMTU(False)                                ERROR
    testDelNetworkWithMTU(True)                                 ERROR
    testDelWithoutAdd                                           ERROR
    testDhclientLeases(4, 'default')                            ERROR
    testDhclientLeases(4, 'local')                              ERROR
    testDhclientLeases(6, None)                                 ERROR
    testDhcpReplaceNicWithBridge                                ERROR
    testFailWithInvalidBondingName(False)                       ERROR
    testFailWithInvalidBondingName(True)                        ERROR
    testFailWithInvalidBridgeName                               ERROR
    testFailWithInvalidIpConfig                                 ERROR
    testFailWithInvalidNic(False)                               ERROR
    testFailWithInvalidNic(True)                                ERROR
    testFailWithInvalidParams(False)                            ERROR
    testFailWithInvalidParams(True)                             ERROR
    testGetRouteDeviceTo                                        ERROR
    testHonorBlockingDhcp                                       SKIP: Slow tests are disabled
    testIpLinkWrapper                                           ERROR
    testLowerMtuDoesNotOverride                                 FAIL
    testNoBridgeLeftovers                                       FAIL
    testReconfigureBrNetWithVanishedPort                        ERROR
    testRedefineBondedNetworkIPs                                ERROR
    testRemovingBridgeDoesNotLeaveBridge                        ERROR
    testReorderBondingOptions(False)                            ERROR
    testReorderBondingOptions(True)                             ERROR
    testRestoreNetworksOnlyRestoreUnchangedDevices              ERROR
    testRestoreToBlockingDHCP                                   ERROR
    testSafeNetworkConfig(False)                                ERROR
    testSafeNetworkConfig(True)                                 ERROR
    testSelectiveRestoreDuringUpgrade                           ERROR
    testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
    testSetupNetworkOutboundQos(False)                          ERROR
    testSetupNetworkOutboundQos(True)                           FAIL
    testSetupNetworksActiveSlave                                ERROR
    testSetupNetworksAddBadParams(False)                        ERROR
    testSetupNetworksAddBadParams(True)                         FAIL
    testSetupNetworksAddBondWithManyVlans(False)                ERROR
    testSetupNetworksAddBondWithManyVlans(True)                 FAIL
    testSetupNetworksAddDelBondedNetwork(False)                 ERROR
    testSetupNetworksAddDelBondedNetwork(True)                  FAIL
    testSetupNetworksAddDelDhcp(False, (4, 6))                  ERROR
    testSetupNetworksAddDelDhcp(False, (4,))                    ERROR
    testSetupNetworksAddDelDhcp(False, (6,))                    ERROR
    testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
    testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
    testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
    testSetupNetworksAddManyVlans(False)                        ERROR
    testSetupNetworksAddManyVlans(True)                         FAIL
    testSetupNetworksAddNetworkToNicAfterBondBreaking(False)    ERROR
    testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
    testSetupNetworksAddNetworkToNicAfterBondResizing(False)    ERROR
    testSetupNetworksAddNetworkToNicAfterBondResizing(True)     ERROR
    testSetupNetworksAddOverExistingBond(False)                 ERROR
    testSetupNetworksAddOverExistingBond(True)                  ERROR
    testSetupNetworksAddVlan(False)                             ERROR
    testSetupNetworksAddVlan(True)                              FAIL
    testSetupNetworksConnectivityCheck                          FAIL
    testSetupNetworksConvertVlanNetBridgeness                   ERROR
    testSetupNetworksDelOneOfBondNets                           ERROR
    testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured   ERROR
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow tests are disabled
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow tests are disabled
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(False)  ERROR
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
    testSetupNetworksMtus(False)                                ERROR
    testSetupNetworksMtus(True)                                 SKIP: This test fails because the 2 different networks are getting configured with the same MTU. The test should assert that the reported MTUs are equal to the requested ones.
    testSetupNetworksMultiMTUsOverBond(False)                   ERROR
    testSetupNetworksMultiMTUsOverBond(True)                    FAIL
    testSetupNetworksMultiMTUsOverNic(False)                    ERROR
    testSetupNetworksMultiMTUsOverNic(True)                     FAIL
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
    testSetupNetworksNicless                                    FAIL
    testSetupNetworksNiclessBridgeless                          ERROR
    testSetupNetworksOverDhcpIface                              ERROR
    testSetupNetworksOverExistingBridge                         FAIL
    testSetupNetworksReconfigureBridge                          FAIL
    testSetupNetworksRemoveBondWithKilledEnslavedNics           ERROR
    testSetupNetworksRemoveSlavelessBond                        ERROR
    testSetupNetworksResizeBond(False)                          ERROR
    testSetupNetworksResizeBond(True)                           ERROR
    testSetupNetworksStableBond(False)                          ERROR
    testSetupNetworksStableBond(True)                           ERROR
    testStaticNetworkConfig((4, 6))                             FAIL
    testStaticNetworkConfig((4,))                               FAIL
    testStaticNetworkConfig((6,))                               FAIL
    testStaticSourceRouting(False)                              ERROR
    testStaticSourceRouting(True)                               FAIL
    testTwiceAdd(False)                                         ERROR
    testTwiceAdd(True)                                          ERROR
    testVolatileConfig(False)                                   ERROR
    testVolatileConfig(True)                                    ERROR
    test_drop_initial_bond_slaves_ip_config                     FAIL
    test_getVdsStats                                            ERROR
    test_remove_bond_under_network                              FAIL
    test_remove_initial_network_nic_ip_config                   FAIL
    test_rollback                                               FAIL
    test_setupNetworks_bond_with_custom_option                  ERROR
    test_setupNetworks_on_external_bond                         ERROR
    test_setupNetworks_on_external_vlaned_bond                  ERROR
functional.networkTestsOVS.OVSNetworkTest
    testHonorBlockingDhcp                                       SKIP: Slow tests are disabled
    testLowerMtuDoesNotOverride                                 OK
    testNoBridgeLeftovers                                       FAIL
    testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
    testSetupNetworkOutboundQos(True)                           OK
    testSetupNetworksAddBadParams(True)                         OK
    testSetupNetworksAddBondWithManyVlans(True)                 FAIL
    testSetupNetworksAddDelBondedNetwork(True)                  FAIL
    testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
    testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
    testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
    testSetupNetworksAddManyVlans(True)                         OK
    testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
    testSetupNetworksAddVlan(True)                              FAIL
    testSetupNetworksConnectivityCheck                          OK
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow tests are disabled
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow tests are disabled
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
    testSetupNetworksMtus(True)                                 SKIP: This test fails because the 2 different networks are getting configured with the same MTU. The test should assert that the reported MTUs are equal to the requested ones.
    testSetupNetworksMultiMTUsOverBond(True)                    FAIL
    testSetupNetworksMultiMTUsOverNic(True)                     FAIL
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
    testSetupNetworksNicless                                    OK
    testSetupNetworksOverExistingBridge                         OK
    testSetupNetworksReconfigureBridge                          FAIL
    testStaticNetworkConfig((4, 6))                             FAIL
    testStaticNetworkConfig((4,))                               FAIL
    testStaticNetworkConfig((6,))                               FAIL
    testStaticSourceRouting(True)                               FAIL
    test_drop_initial_bond_slaves_ip_config                     FAIL
    test_ovirtmgmtm_to_ovs                                      FAIL
    test_remove_bond_under_network                              SKIP: OVS does not support bridgeless networks
    test_remove_initial_network_nic_ip_config                   OK
    test_rollback                                               FAIL


On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew@redhat.com> wrote:
I like the idea but I have the same feelings as Francesco. I think that we need to make sure that functional tests for each vertical are stable before enabling this process.

On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri@redhat.com> wrote:
adding also infra team for visibility on the change in CI.
also inline.

On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani@redhat.com> wrote:
----- Original Message -----
> From: "Yaniv Bronheim" <ybronhei@redhat.com>
> To: devel@ovirt.org, "Francesco Romani" <fromani@redhat.com>, "Nir Soffer" <nsoffer@redhat.com>, "Piotr Kliczewski"
> <pkliczew@redhat.com>
> Cc: "danken" <danken@redhat.com>, "David Caro" <dcaroest@redhat.com>, "Eyal Edri" <eedri@redhat.com>
> Sent: Thursday, December 10, 2015 6:46:37 PM
> Subject: Automation CI for vdsm

[...]
> We want to allow developers to trigger the script once reviews and
> verification are ready (last step before merge). To do so we agreed to add
> Continues Integration flag for each vdsm patch.

This flag will be called 'Workflow' or we can name it otherwise, we just need to choose what makes sense. 
David/Yaniv - Please correct me if I'm wrong.
 
Once this flag will be
> signed with +1 it will trigger Jenkins CI to run the check-merged script
> (adding new button to gerrit is not an option - you can image that flag as
> a trigger button), on success Jenkins CI flag will turn to +2. on fail
> we'll get -1 and once new patchset is ready the developer will remove the
> +1 and add it back to the Continues Integration flag to re-trigger the job.
>
> Please ack the process before we move on with that

Sounds good, even though I'm a little scared (just gut feeling, no evidence
whatsoever) that this could add even more complexity and fragility to the jenkins
fleet.

In the long run, when this is reliable, it will help greatly.
In the short term, I'm scared because this can lead to false positives and bogus
failures.

Let me stress I don't have concrete item to share or specific flaws.

As action item on me, I will find some time next week to check virt functional tests,
to see if they need some fixes, work reliably and so forth

> The patch for those scripts still under review and testing -
> https://gerrit.ovirt.org/#/c/48268

Will review asap.

--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani



--
Eyal Edri
Supervisor, RHEV CI
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)




--
Yaniv Bronhaim.