Nice, but we cannot enable this until all the tests pass or disabled.
There is no point in broken or flaky functional tests.
On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei@redhat.com> wrote:
> So its not stable. It won't block merges and at least give us report after
> each merge. It takes really long time to run it (because of the tests
> themselves. Lago things takes maximum 15minutes, but the run last for more
> than 2hrs right now and I suspect functional/storageTests.py gets stuck)
>
> Bellow you can see where we stand (before I added python-rtslib package).
>
> Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ -
> which enables this run after merges, and I still want you to consider the
> addition of Automation CI flag to our gerrit so that developer will be able
> to use it as a trigger for the check-merged.sh script run, just to see if
> their patch fixes\brakes something realted to the functional tests
>
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is
> an example of how the run looks like. I still work to improve the output
>
>
> Please reply and let me know if the idea around the automation flag is
> acceptable by you.. and please review the patch for comments and acks.
> We can ask dcaro to add the flag until Friday, otherwise we'll need to delay
> this effort after the holiday..
>
>
> functional.sosPluginTests.SosPluginTest
> testSosPlugin OK
> functional.vmRecoveryTests.RecoveryTests
> test_vm_recovery FAIL
> functional.vmQoSTests.VMQosTests
> testSmallVMBallooning FAIL
> functional.virtTests.VirtTest
> testComplexVm FAIL
> testHeadlessVm OK
> testSimpleVm FAIL
> testVmDefinitionGraphics('spice') FAIL
> testVmDefinitionGraphics('vnc') OK
> testVmDefinitionLegacyGraphics('qxl') FAIL
> testVmDefinitionLegacyGraphics('vnc') OK
> testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL
> testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL
> testVmWithCdrom('self') FAIL
> testVmWithCdrom('specParams') FAIL
> testVmWithCdrom('vmPayload') FAIL
> testVmWithDevice('hotplugDisk') FAIL
> testVmWithDevice('hotplugNic') FAIL
> testVmWithDevice('smartcard') FAIL
> testVmWithDevice('virtioNic') FAIL
> testVmWithDevice('virtioRng') FAIL
> testVmWithSla FAIL
> testVmWithStorage('iscsi') SKIP:
> python-rtslib is not installed.
> testVmWithStorage('localfs') FAIL
> testVmWithStorage('nfs') FAIL
> functional.storageTests.StorageTest
> testCreatePoolErrors OK
> testStorage('glusterfs', 0) ERROR
> testStorage('glusterfs', 3) ERROR
> testStorage('iscsi', 0) SKIP:
> python-rtslib is not installed.
> testStorage('iscsi', 3) SKIP:
> python-rtslib is not installed.
> testStorage('localfs', 0) FAIL
> testStorage('localfs', 3) FAIL
> testStorage('nfs', 0) FAIL
> testStorage('nfs', 3) FAIL
> functional.networkTests.NetworkTest
> testAddVlanedBridgeless ERROR
> testAddVlanedBridgeless_oneCommand ERROR
> testAfterNetworkSetupHook ERROR
> testBeforeNetworkSetupHook ERROR
> testBondHwAddress(False) ERROR
> testBondHwAddress(True) ERROR
> testBrokenNetworkReplacement(False) ERROR
> testBrokenNetworkReplacement(True) ERROR
> testDelNetworkBondAccumulation ERROR
> testDelNetworkWithMTU(False) ERROR
> testDelNetworkWithMTU(True) ERROR
> testDelWithoutAdd ERROR
> testDhclientLeases(4, 'default') ERROR
> testDhclientLeases(4, 'local') ERROR
> testDhclientLeases(6, None) ERROR
> testDhcpReplaceNicWithBridge ERROR
> testFailWithInvalidBondingName(False) ERROR
> testFailWithInvalidBondingName(True) ERROR
> testFailWithInvalidBridgeName ERROR
> testFailWithInvalidIpConfig ERROR
> testFailWithInvalidNic(False) ERROR
> testFailWithInvalidNic(True) ERROR
> testFailWithInvalidParams(False) ERROR
> testFailWithInvalidParams(True) ERROR
> testGetRouteDeviceTo ERROR
> testHonorBlockingDhcp SKIP: Slow
> tests are disabled
> testIpLinkWrapper ERROR
> testLowerMtuDoesNotOverride FAIL
> testNoBridgeLeftovers FAIL
> testReconfigureBrNetWithVanishedPort ERROR
> testRedefineBondedNetworkIPs ERROR
> testRemovingBridgeDoesNotLeaveBridge ERROR
> testReorderBondingOptions(False) ERROR
> testReorderBondingOptions(True) ERROR
> testRestoreNetworksOnlyRestoreUnchangedDevices ERROR
> testRestoreToBlockingDHCP ERROR
> testSafeNetworkConfig(False) ERROR
> testSafeNetworkConfig(True) ERROR
> testSelectiveRestoreDuringUpgrade ERROR
> testSelectiveRestoreIgnoresVdsmRegParams FAIL
> testSetupNetworkOutboundQos(False) ERROR
> testSetupNetworkOutboundQos(True) FAIL
> testSetupNetworksActiveSlave ERROR
> testSetupNetworksAddBadParams(False) ERROR
> testSetupNetworksAddBadParams(True) FAIL
> testSetupNetworksAddBondWithManyVlans(False) ERROR
> testSetupNetworksAddBondWithManyVlans(True) FAIL
> testSetupNetworksAddDelBondedNetwork(False) ERROR
> testSetupNetworksAddDelBondedNetwork(True) FAIL
> testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR
> testSetupNetworksAddDelDhcp(False, (4,)) ERROR
> testSetupNetworksAddDelDhcp(False, (6,)) ERROR
> testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> testSetupNetworksAddManyVlans(False) ERROR
> testSetupNetworksAddManyVlans(True) FAIL
> testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR
> testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR
> testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR
> testSetupNetworksAddOverExistingBond(False) ERROR
> testSetupNetworksAddOverExistingBond(True) ERROR
> testSetupNetworksAddVlan(False) ERROR
> testSetupNetworksAddVlan(True) FAIL
> testSetupNetworksConnectivityCheck FAIL
> testSetupNetworksConvertVlanNetBridgeness ERROR
> testSetupNetworksDelOneOfBondNets ERROR
> testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> testSetupNetworksMtus(False) ERROR
> testSetupNetworksMtus(True) SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
> testSetupNetworksMultiMTUsOverBond(False) ERROR
> testSetupNetworksMultiMTUsOverBond(True) FAIL
> testSetupNetworksMultiMTUsOverNic(False) ERROR
> testSetupNetworksMultiMTUsOverNic(True) FAIL
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> testSetupNetworksNicless FAIL
> testSetupNetworksNiclessBridgeless ERROR
> testSetupNetworksOverDhcpIface ERROR
> testSetupNetworksOverExistingBridge FAIL
> testSetupNetworksReconfigureBridge FAIL
> testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR
> testSetupNetworksRemoveSlavelessBond ERROR
> testSetupNetworksResizeBond(False) ERROR
> testSetupNetworksResizeBond(True) ERROR
> testSetupNetworksStableBond(False) ERROR
> testSetupNetworksStableBond(True) ERROR
> testStaticNetworkConfig((4, 6)) FAIL
> testStaticNetworkConfig((4,)) FAIL
> testStaticNetworkConfig((6,)) FAIL
> testStaticSourceRouting(False) ERROR
> testStaticSourceRouting(True) FAIL
> testTwiceAdd(False) ERROR
> testTwiceAdd(True) ERROR
> testVolatileConfig(False) ERROR
> testVolatileConfig(True) ERROR
> test_drop_initial_bond_slaves_ip_config FAIL
> test_getVdsStats ERROR
> test_remove_bond_under_network FAIL
> test_remove_initial_network_nic_ip_config FAIL
> test_rollback FAIL
> test_setupNetworks_bond_with_custom_option ERROR
> test_setupNetworks_on_external_bond ERROR
> test_setupNetworks_on_external_vlaned_bond ERROR
> functional.networkTestsOVS.OVSNetworkTest
> testHonorBlockingDhcp SKIP: Slow
> tests are disabled
> testLowerMtuDoesNotOverride OK
> testNoBridgeLeftovers FAIL
> testSelectiveRestoreIgnoresVdsmRegParams FAIL
> testSetupNetworkOutboundQos(True) OK
> testSetupNetworksAddBadParams(True) OK
> testSetupNetworksAddBondWithManyVlans(True) FAIL
> testSetupNetworksAddDelBondedNetwork(True) FAIL
> testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> testSetupNetworksAddManyVlans(True) OK
> testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> testSetupNetworksAddVlan(True) FAIL
> testSetupNetworksConnectivityCheck OK
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> testSetupNetworksMtus(True) SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
> testSetupNetworksMultiMTUsOverBond(True) FAIL
> testSetupNetworksMultiMTUsOverNic(True) FAIL
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> testSetupNetworksNicless OK
> testSetupNetworksOverExistingBridge OK
> testSetupNetworksReconfigureBridge FAIL
> testStaticNetworkConfig((4, 6)) FAIL
> testStaticNetworkConfig((4,)) FAIL
> testStaticNetworkConfig((6,)) FAIL
> testStaticSourceRouting(True) FAIL
> test_drop_initial_bond_slaves_ip_config FAIL
> test_ovirtmgmtm_to_ovs FAIL
> test_remove_bond_under_network SKIP: OVS
> does not support bridgeless networks
> test_remove_initial_network_nic_ip_config OK
> test_rollback FAIL
>
>
> On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew@redhat.com>
> wrote:
>>
>> I like the idea but I have the same feelings as Francesco. I think that we
>> need to make sure that functional tests for each vertical are stable before
>> enabling this process.
>>
>> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri@redhat.com> wrote:
>>>
>>> adding also infra team for visibility on the change in CI.
>>> also inline.
>>>
>>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani@redhat.com>
>>> wrote:
>>>>
>>>> ----- Original Message -----
>>>> > From: "Yaniv Bronheim" <ybronhei@redhat.com>
>>>> > To: devel@ovirt.org, "Francesco Romani" <fromani@redhat.com>, "Nir
>>>> > Soffer" <nsoffer@redhat.com>, "Piotr Kliczewski"
>>>> > <pkliczew@redhat.com>
>>>> > Cc: "danken" <danken@redhat.com>, "David Caro" <dcaroest@redhat.com>,
>>>> > "Eyal Edri" <eedri@redhat.com>
>>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
>>>> > Subject: Automation CI for vdsm
>>>>
>>>> [...]
>>>> > We want to allow developers to trigger the script once reviews and
>>>> > verification are ready (last step before merge). To do so we agreed to
>>>> > add
>>>> > Continues Integration flag for each vdsm patch.
>>>
>>>
>>> This flag will be called 'Workflow' or we can name it otherwise, we just
>>> need to choose what makes sense.
>>> David/Yaniv - Please correct me if I'm wrong.
>>>
>>>>
>>>> Once this flag will be
>>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
>>>> > script
>>>> > (adding new button to gerrit is not an option - you can image that
>>>> > flag as
>>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail
>>>> > we'll get -1 and once new patchset is ready the developer will remove
>>>> > the
>>>> > +1 and add it back to the Continues Integration flag to re-trigger the
>>>> > job.
>>>> >
>>>> > Please ack the process before we move on with that
>>>>
>>>> Sounds good, even though I'm a little scared (just gut feeling, no
>>>> evidence
>>>> whatsoever) that this could add even more complexity and fragility to
>>>> the jenkins
>>>> fleet.
>>>>
>>>> In the long run, when this is reliable, it will help greatly.
>>>> In the short term, I'm scared because this can lead to false positives
>>>> and bogus
>>>> failures.
>>>>
>>>> Let me stress I don't have concrete item to share or specific flaws.
>>>>
>>>> As action item on me, I will find some time next week to check virt
>>>> functional tests,
>>>> to see if they need some fixes, work reliably and so forth
>>>>
>>>> > The patch for those scripts still under review and testing -
>>>> > https://gerrit.ovirt.org/#/c/48268
>>>>
>>>> Will review asap.
>>>>
>>>> --
>>>> Francesco Romani
>>>> RedHat Engineering Virtualization R & D
>>>> Phone: 8261328
>>>> IRC: fromani
>>>
>>>
>>>
>>>
>>> --
>>> Eyal Edri
>>> Supervisor, RHEV CI
>>> EMEA ENG Virtualization R&D
>>> Red Hat Israel
>>>
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>>
>
>
>
> --
> Yaniv Bronhaim.