Automation CI for vdsm

Nir Soffer nsoffer at redhat.com
Wed Dec 16 16:19:20 UTC 2015


Nice, but we cannot enable this until all the tests pass or disabled.

There is no point in broken or flaky functional tests.

On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei at redhat.com> wrote:
> So its not stable. It won't block merges and at least give us report after
> each merge. It takes really long time to run it (because of the tests
> themselves. Lago things takes maximum 15minutes, but the run last for more
> than 2hrs right now and I suspect functional/storageTests.py gets stuck)
>
> Bellow you can see where we stand (before I added python-rtslib package).
>
> Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ -
> which enables this run after merges, and I still want you to consider the
> addition of Automation CI flag to our gerrit so that developer will be able
> to use it as a trigger for the check-merged.sh script run, just to see if
> their patch fixes\brakes something realted to the functional tests
>
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is
> an example of how the run looks like. I still work to improve the output
>
>
> Please reply and let me know if the idea around the automation flag is
> acceptable by you.. and please review the patch for comments and acks.
> We can ask dcaro to add the flag until Friday, otherwise we'll need to delay
> this effort after the holiday..
>
>
> functional.sosPluginTests.SosPluginTest
>     testSosPlugin                                               OK
> functional.vmRecoveryTests.RecoveryTests
>     test_vm_recovery                                            FAIL
> functional.vmQoSTests.VMQosTests
>     testSmallVMBallooning                                       FAIL
> functional.virtTests.VirtTest
>     testComplexVm                                               FAIL
>     testHeadlessVm                                              OK
>     testSimpleVm                                                FAIL
>     testVmDefinitionGraphics('spice')                           FAIL
>     testVmDefinitionGraphics('vnc')                             OK
>     testVmDefinitionLegacyGraphics('qxl')                       FAIL
>     testVmDefinitionLegacyGraphics('vnc')                       OK
>     testVmDefinitionMultipleGraphics('spice', 'vnc')            FAIL
>     testVmDefinitionMultipleGraphics('vnc', 'spice')            FAIL
>     testVmWithCdrom('self')                                     FAIL
>     testVmWithCdrom('specParams')                               FAIL
>     testVmWithCdrom('vmPayload')                                FAIL
>     testVmWithDevice('hotplugDisk')                             FAIL
>     testVmWithDevice('hotplugNic')                              FAIL
>     testVmWithDevice('smartcard')                               FAIL
>     testVmWithDevice('virtioNic')                               FAIL
>     testVmWithDevice('virtioRng')                               FAIL
>     testVmWithSla                                               FAIL
>     testVmWithStorage('iscsi')                                  SKIP:
> python-rtslib is not installed.
>     testVmWithStorage('localfs')                                FAIL
>     testVmWithStorage('nfs')                                    FAIL
> functional.storageTests.StorageTest
>     testCreatePoolErrors                                        OK
>     testStorage('glusterfs', 0)                                 ERROR
>     testStorage('glusterfs', 3)                                 ERROR
>     testStorage('iscsi', 0)                                     SKIP:
> python-rtslib is not installed.
>     testStorage('iscsi', 3)                                     SKIP:
> python-rtslib is not installed.
>     testStorage('localfs', 0)                                   FAIL
>     testStorage('localfs', 3)                                   FAIL
>     testStorage('nfs', 0)                                       FAIL
>     testStorage('nfs', 3)                                       FAIL
> functional.networkTests.NetworkTest
>     testAddVlanedBridgeless                                     ERROR
>     testAddVlanedBridgeless_oneCommand                          ERROR
>     testAfterNetworkSetupHook                                   ERROR
>     testBeforeNetworkSetupHook                                  ERROR
>     testBondHwAddress(False)                                    ERROR
>     testBondHwAddress(True)                                     ERROR
>     testBrokenNetworkReplacement(False)                         ERROR
>     testBrokenNetworkReplacement(True)                          ERROR
>     testDelNetworkBondAccumulation                              ERROR
>     testDelNetworkWithMTU(False)                                ERROR
>     testDelNetworkWithMTU(True)                                 ERROR
>     testDelWithoutAdd                                           ERROR
>     testDhclientLeases(4, 'default')                            ERROR
>     testDhclientLeases(4, 'local')                              ERROR
>     testDhclientLeases(6, None)                                 ERROR
>     testDhcpReplaceNicWithBridge                                ERROR
>     testFailWithInvalidBondingName(False)                       ERROR
>     testFailWithInvalidBondingName(True)                        ERROR
>     testFailWithInvalidBridgeName                               ERROR
>     testFailWithInvalidIpConfig                                 ERROR
>     testFailWithInvalidNic(False)                               ERROR
>     testFailWithInvalidNic(True)                                ERROR
>     testFailWithInvalidParams(False)                            ERROR
>     testFailWithInvalidParams(True)                             ERROR
>     testGetRouteDeviceTo                                        ERROR
>     testHonorBlockingDhcp                                       SKIP: Slow
> tests are disabled
>     testIpLinkWrapper                                           ERROR
>     testLowerMtuDoesNotOverride                                 FAIL
>     testNoBridgeLeftovers                                       FAIL
>     testReconfigureBrNetWithVanishedPort                        ERROR
>     testRedefineBondedNetworkIPs                                ERROR
>     testRemovingBridgeDoesNotLeaveBridge                        ERROR
>     testReorderBondingOptions(False)                            ERROR
>     testReorderBondingOptions(True)                             ERROR
>     testRestoreNetworksOnlyRestoreUnchangedDevices              ERROR
>     testRestoreToBlockingDHCP                                   ERROR
>     testSafeNetworkConfig(False)                                ERROR
>     testSafeNetworkConfig(True)                                 ERROR
>     testSelectiveRestoreDuringUpgrade                           ERROR
>     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
>     testSetupNetworkOutboundQos(False)                          ERROR
>     testSetupNetworkOutboundQos(True)                           FAIL
>     testSetupNetworksActiveSlave                                ERROR
>     testSetupNetworksAddBadParams(False)                        ERROR
>     testSetupNetworksAddBadParams(True)                         FAIL
>     testSetupNetworksAddBondWithManyVlans(False)                ERROR
>     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
>     testSetupNetworksAddDelBondedNetwork(False)                 ERROR
>     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
>     testSetupNetworksAddDelDhcp(False, (4, 6))                  ERROR
>     testSetupNetworksAddDelDhcp(False, (4,))                    ERROR
>     testSetupNetworksAddDelDhcp(False, (6,))                    ERROR
>     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
>     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
>     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
>     testSetupNetworksAddManyVlans(False)                        ERROR
>     testSetupNetworksAddManyVlans(True)                         FAIL
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(False)    ERROR
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
>     testSetupNetworksAddNetworkToNicAfterBondResizing(False)    ERROR
>     testSetupNetworksAddNetworkToNicAfterBondResizing(True)     ERROR
>     testSetupNetworksAddOverExistingBond(False)                 ERROR
>     testSetupNetworksAddOverExistingBond(True)                  ERROR
>     testSetupNetworksAddVlan(False)                             ERROR
>     testSetupNetworksAddVlan(True)                              FAIL
>     testSetupNetworksConnectivityCheck                          FAIL
>     testSetupNetworksConvertVlanNetBridgeness                   ERROR
>     testSetupNetworksDelOneOfBondNets                           ERROR
>     testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured   ERROR
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(False)  ERROR
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
>     testSetupNetworksMtus(False)                                ERROR
>     testSetupNetworksMtus(True)                                 SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
>     testSetupNetworksMultiMTUsOverBond(False)                   ERROR
>     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
>     testSetupNetworksMultiMTUsOverNic(False)                    ERROR
>     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
>     testSetupNetworksNicless                                    FAIL
>     testSetupNetworksNiclessBridgeless                          ERROR
>     testSetupNetworksOverDhcpIface                              ERROR
>     testSetupNetworksOverExistingBridge                         FAIL
>     testSetupNetworksReconfigureBridge                          FAIL
>     testSetupNetworksRemoveBondWithKilledEnslavedNics           ERROR
>     testSetupNetworksRemoveSlavelessBond                        ERROR
>     testSetupNetworksResizeBond(False)                          ERROR
>     testSetupNetworksResizeBond(True)                           ERROR
>     testSetupNetworksStableBond(False)                          ERROR
>     testSetupNetworksStableBond(True)                           ERROR
>     testStaticNetworkConfig((4, 6))                             FAIL
>     testStaticNetworkConfig((4,))                               FAIL
>     testStaticNetworkConfig((6,))                               FAIL
>     testStaticSourceRouting(False)                              ERROR
>     testStaticSourceRouting(True)                               FAIL
>     testTwiceAdd(False)                                         ERROR
>     testTwiceAdd(True)                                          ERROR
>     testVolatileConfig(False)                                   ERROR
>     testVolatileConfig(True)                                    ERROR
>     test_drop_initial_bond_slaves_ip_config                     FAIL
>     test_getVdsStats                                            ERROR
>     test_remove_bond_under_network                              FAIL
>     test_remove_initial_network_nic_ip_config                   FAIL
>     test_rollback                                               FAIL
>     test_setupNetworks_bond_with_custom_option                  ERROR
>     test_setupNetworks_on_external_bond                         ERROR
>     test_setupNetworks_on_external_vlaned_bond                  ERROR
> functional.networkTestsOVS.OVSNetworkTest
>     testHonorBlockingDhcp                                       SKIP: Slow
> tests are disabled
>     testLowerMtuDoesNotOverride                                 OK
>     testNoBridgeLeftovers                                       FAIL
>     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
>     testSetupNetworkOutboundQos(True)                           OK
>     testSetupNetworksAddBadParams(True)                         OK
>     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
>     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
>     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
>     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
>     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
>     testSetupNetworksAddManyVlans(True)                         OK
>     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
>     testSetupNetworksAddVlan(True)                              FAIL
>     testSetupNetworksConnectivityCheck                          OK
>     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
> tests are disabled
>     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
>     testSetupNetworksMtus(True)                                 SKIP: This
> test fails because the 2 different networks are getting configured with the
> same MTU. The test should assert that the reported MTUs are equal to the
> requested ones.
>     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
>     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
>     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
>     testSetupNetworksNicless                                    OK
>     testSetupNetworksOverExistingBridge                         OK
>     testSetupNetworksReconfigureBridge                          FAIL
>     testStaticNetworkConfig((4, 6))                             FAIL
>     testStaticNetworkConfig((4,))                               FAIL
>     testStaticNetworkConfig((6,))                               FAIL
>     testStaticSourceRouting(True)                               FAIL
>     test_drop_initial_bond_slaves_ip_config                     FAIL
>     test_ovirtmgmtm_to_ovs                                      FAIL
>     test_remove_bond_under_network                              SKIP: OVS
> does not support bridgeless networks
>     test_remove_initial_network_nic_ip_config                   OK
>     test_rollback                                               FAIL
>
>
> On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew at redhat.com>
> wrote:
>>
>> I like the idea but I have the same feelings as Francesco. I think that we
>> need to make sure that functional tests for each vertical are stable before
>> enabling this process.
>>
>> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
>>>
>>> adding also infra team for visibility on the change in CI.
>>> also inline.
>>>
>>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani at redhat.com>
>>> wrote:
>>>>
>>>> ----- Original Message -----
>>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
>>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>, "Nir
>>>> > Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
>>>> > <pkliczew at redhat.com>
>>>> > Cc: "danken" <danken at redhat.com>, "David Caro" <dcaroest at redhat.com>,
>>>> > "Eyal Edri" <eedri at redhat.com>
>>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
>>>> > Subject: Automation CI for vdsm
>>>>
>>>> [...]
>>>> > We want to allow developers to trigger the script once reviews and
>>>> > verification are ready (last step before merge). To do so we agreed to
>>>> > add
>>>> > Continues Integration flag for each vdsm patch.
>>>
>>>
>>> This flag will be called 'Workflow' or we can name it otherwise, we just
>>> need to choose what makes sense.
>>> David/Yaniv - Please correct me if I'm wrong.
>>>
>>>>
>>>> Once this flag will be
>>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
>>>> > script
>>>> > (adding new button to gerrit is not an option - you can image that
>>>> > flag as
>>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail
>>>> > we'll get -1 and once new patchset is ready the developer will remove
>>>> > the
>>>> > +1 and add it back to the Continues Integration flag to re-trigger the
>>>> > job.
>>>> >
>>>> > Please ack the process before we move on with that
>>>>
>>>> Sounds good, even though I'm a little scared (just gut feeling, no
>>>> evidence
>>>> whatsoever) that this could add even more complexity and fragility to
>>>> the jenkins
>>>> fleet.
>>>>
>>>> In the long run, when this is reliable, it will help greatly.
>>>> In the short term, I'm scared because this can lead to false positives
>>>> and bogus
>>>> failures.
>>>>
>>>> Let me stress I don't have concrete item to share or specific flaws.
>>>>
>>>> As action item on me, I will find some time next week to check virt
>>>> functional tests,
>>>> to see if they need some fixes, work reliably and so forth
>>>>
>>>> > The patch for those scripts still under review and testing -
>>>> > https://gerrit.ovirt.org/#/c/48268
>>>>
>>>> Will review asap.
>>>>
>>>> --
>>>> Francesco Romani
>>>> RedHat Engineering Virtualization R & D
>>>> Phone: 8261328
>>>> IRC: fromani
>>>
>>>
>>>
>>>
>>> --
>>> Eyal Edri
>>> Supervisor, RHEV CI
>>> EMEA ENG Virtualization R&D
>>> Red Hat Israel
>>>
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>>
>
>
>
> --
> Yaniv Bronhaim.



More information about the Infra mailing list