[ovirt-devel] Automation CI for vdsm

Yaniv Bronheim ybronhei at redhat.com
Wed Dec 16 15:34:54 UTC 2015


So its not stable. It won't block merges and at least give us report after
each merge. It takes really long time to run it (because of the tests
themselves. Lago things takes maximum 15minutes, but the run last for more
than 2hrs right now and I suspect functional/storageTests.py gets stuck)

Bellow you can see where we stand (before I added python-rtslib package).

Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ -
which enables this run after merges, and I still want you to consider the
addition of Automation CI flag to our gerrit so that developer will be able
to use it as a trigger for the check-merged.sh script run, just to see if
their patch fixes\brakes something realted to the functional tests

http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is
an example of how the run looks like. I still work to improve the output


Please reply and let me know if the idea around the automation flag is
acceptable by you.. and please review the patch for comments and acks.
We can ask dcaro to add the flag until Friday, otherwise we'll need to
delay this effort after the holiday..


functional.sosPluginTests.SosPluginTest
    testSosPlugin                                               OK
functional.vmRecoveryTests.RecoveryTests
    test_vm_recovery                                            FAIL
functional.vmQoSTests.VMQosTests
    testSmallVMBallooning                                       FAIL
functional.virtTests.VirtTest
    testComplexVm                                               FAIL
    testHeadlessVm                                              OK
    testSimpleVm                                                FAIL
    testVmDefinitionGraphics('spice')                           FAIL
    testVmDefinitionGraphics('vnc')                             OK
    testVmDefinitionLegacyGraphics('qxl')                       FAIL
    testVmDefinitionLegacyGraphics('vnc')                       OK
    testVmDefinitionMultipleGraphics('spice', 'vnc')            FAIL
    testVmDefinitionMultipleGraphics('vnc', 'spice')            FAIL
    testVmWithCdrom('self')                                     FAIL
    testVmWithCdrom('specParams')                               FAIL
    testVmWithCdrom('vmPayload')                                FAIL
    testVmWithDevice('hotplugDisk')                             FAIL
    testVmWithDevice('hotplugNic')                              FAIL
    testVmWithDevice('smartcard')                               FAIL
    testVmWithDevice('virtioNic')                               FAIL
    testVmWithDevice('virtioRng')                               FAIL
    testVmWithSla                                               FAIL
    testVmWithStorage('iscsi')                                  SKIP:
python-rtslib is not installed.
    testVmWithStorage('localfs')                                FAIL
    testVmWithStorage('nfs')                                    FAIL
functional.storageTests.StorageTest
    testCreatePoolErrors                                        OK
    testStorage('glusterfs', 0)                                 ERROR
    testStorage('glusterfs', 3)                                 ERROR
    testStorage('iscsi', 0)                                     SKIP:
python-rtslib is not installed.
    testStorage('iscsi', 3)                                     SKIP:
python-rtslib is not installed.
    testStorage('localfs', 0)                                   FAIL
    testStorage('localfs', 3)                                   FAIL
    testStorage('nfs', 0)                                       FAIL
    testStorage('nfs', 3)                                       FAIL
functional.networkTests.NetworkTest
    testAddVlanedBridgeless                                     ERROR
    testAddVlanedBridgeless_oneCommand                          ERROR
    testAfterNetworkSetupHook                                   ERROR
    testBeforeNetworkSetupHook                                  ERROR
    testBondHwAddress(False)                                    ERROR
    testBondHwAddress(True)                                     ERROR
    testBrokenNetworkReplacement(False)                         ERROR
    testBrokenNetworkReplacement(True)                          ERROR
    testDelNetworkBondAccumulation                              ERROR
    testDelNetworkWithMTU(False)                                ERROR
    testDelNetworkWithMTU(True)                                 ERROR
    testDelWithoutAdd                                           ERROR
    testDhclientLeases(4, 'default')                            ERROR
    testDhclientLeases(4, 'local')                              ERROR
    testDhclientLeases(6, None)                                 ERROR
    testDhcpReplaceNicWithBridge                                ERROR
    testFailWithInvalidBondingName(False)                       ERROR
    testFailWithInvalidBondingName(True)                        ERROR
    testFailWithInvalidBridgeName                               ERROR
    testFailWithInvalidIpConfig                                 ERROR
    testFailWithInvalidNic(False)                               ERROR
    testFailWithInvalidNic(True)                                ERROR
    testFailWithInvalidParams(False)                            ERROR
    testFailWithInvalidParams(True)                             ERROR
    testGetRouteDeviceTo                                        ERROR
    testHonorBlockingDhcp                                       SKIP: Slow
tests are disabled
    testIpLinkWrapper                                           ERROR
    testLowerMtuDoesNotOverride                                 FAIL
    testNoBridgeLeftovers                                       FAIL
    testReconfigureBrNetWithVanishedPort                        ERROR
    testRedefineBondedNetworkIPs                                ERROR
    testRemovingBridgeDoesNotLeaveBridge                        ERROR
    testReorderBondingOptions(False)                            ERROR
    testReorderBondingOptions(True)                             ERROR
    testRestoreNetworksOnlyRestoreUnchangedDevices              ERROR
    testRestoreToBlockingDHCP                                   ERROR
    testSafeNetworkConfig(False)                                ERROR
    testSafeNetworkConfig(True)                                 ERROR
    testSelectiveRestoreDuringUpgrade                           ERROR
    testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
    testSetupNetworkOutboundQos(False)                          ERROR
    testSetupNetworkOutboundQos(True)                           FAIL
    testSetupNetworksActiveSlave                                ERROR
    testSetupNetworksAddBadParams(False)                        ERROR
    testSetupNetworksAddBadParams(True)                         FAIL
    testSetupNetworksAddBondWithManyVlans(False)                ERROR
    testSetupNetworksAddBondWithManyVlans(True)                 FAIL
    testSetupNetworksAddDelBondedNetwork(False)                 ERROR
    testSetupNetworksAddDelBondedNetwork(True)                  FAIL
    testSetupNetworksAddDelDhcp(False, (4, 6))                  ERROR
    testSetupNetworksAddDelDhcp(False, (4,))                    ERROR
    testSetupNetworksAddDelDhcp(False, (6,))                    ERROR
    testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
    testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
    testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
    testSetupNetworksAddManyVlans(False)                        ERROR
    testSetupNetworksAddManyVlans(True)                         FAIL
    testSetupNetworksAddNetworkToNicAfterBondBreaking(False)    ERROR
    testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
    testSetupNetworksAddNetworkToNicAfterBondResizing(False)    ERROR
    testSetupNetworksAddNetworkToNicAfterBondResizing(True)     ERROR
    testSetupNetworksAddOverExistingBond(False)                 ERROR
    testSetupNetworksAddOverExistingBond(True)                  ERROR
    testSetupNetworksAddVlan(False)                             ERROR
    testSetupNetworksAddVlan(True)                              FAIL
    testSetupNetworksConnectivityCheck                          FAIL
    testSetupNetworksConvertVlanNetBridgeness                   ERROR
    testSetupNetworksDelOneOfBondNets                           ERROR
    testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured   ERROR
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
tests are disabled
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
tests are disabled
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(False)  ERROR
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
    testSetupNetworksMtus(False)                                ERROR
    testSetupNetworksMtus(True)                                 SKIP: This
test fails because the 2 different networks are getting configured with the
same MTU. The test should assert that the reported MTUs are equal to the
requested ones.
    testSetupNetworksMultiMTUsOverBond(False)                   ERROR
    testSetupNetworksMultiMTUsOverBond(True)                    FAIL
    testSetupNetworksMultiMTUsOverNic(False)                    ERROR
    testSetupNetworksMultiMTUsOverNic(True)                     FAIL
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
    testSetupNetworksNicless                                    FAIL
    testSetupNetworksNiclessBridgeless                          ERROR
    testSetupNetworksOverDhcpIface                              ERROR
    testSetupNetworksOverExistingBridge                         FAIL
    testSetupNetworksReconfigureBridge                          FAIL
    testSetupNetworksRemoveBondWithKilledEnslavedNics           ERROR
    testSetupNetworksRemoveSlavelessBond                        ERROR
    testSetupNetworksResizeBond(False)                          ERROR
    testSetupNetworksResizeBond(True)                           ERROR
    testSetupNetworksStableBond(False)                          ERROR
    testSetupNetworksStableBond(True)                           ERROR
    testStaticNetworkConfig((4, 6))                             FAIL
    testStaticNetworkConfig((4,))                               FAIL
    testStaticNetworkConfig((6,))                               FAIL
    testStaticSourceRouting(False)                              ERROR
    testStaticSourceRouting(True)                               FAIL
    testTwiceAdd(False)                                         ERROR
    testTwiceAdd(True)                                          ERROR
    testVolatileConfig(False)                                   ERROR
    testVolatileConfig(True)                                    ERROR
    test_drop_initial_bond_slaves_ip_config                     FAIL
    test_getVdsStats                                            ERROR
    test_remove_bond_under_network                              FAIL
    test_remove_initial_network_nic_ip_config                   FAIL
    test_rollback                                               FAIL
    test_setupNetworks_bond_with_custom_option                  ERROR
    test_setupNetworks_on_external_bond                         ERROR
    test_setupNetworks_on_external_vlaned_bond                  ERROR
functional.networkTestsOVS.OVSNetworkTest
    testHonorBlockingDhcp                                       SKIP: Slow
tests are disabled
    testLowerMtuDoesNotOverride                                 OK
    testNoBridgeLeftovers                                       FAIL
    testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
    testSetupNetworkOutboundQos(True)                           OK
    testSetupNetworksAddBadParams(True)                         OK
    testSetupNetworksAddBondWithManyVlans(True)                 FAIL
    testSetupNetworksAddDelBondedNetwork(True)                  FAIL
    testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
    testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
    testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
    testSetupNetworksAddManyVlans(True)                         OK
    testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
    testSetupNetworksAddVlan(True)                              FAIL
    testSetupNetworksConnectivityCheck                          OK
    testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
tests are disabled
    testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
tests are disabled
    testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
    testSetupNetworksMtus(True)                                 SKIP: This
test fails because the 2 different networks are getting configured with the
same MTU. The test should assert that the reported MTUs are equal to the
requested ones.
    testSetupNetworksMultiMTUsOverBond(True)                    FAIL
    testSetupNetworksMultiMTUsOverNic(True)                     FAIL
    testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
    testSetupNetworksNicless                                    OK
    testSetupNetworksOverExistingBridge                         OK
    testSetupNetworksReconfigureBridge                          FAIL
    testStaticNetworkConfig((4, 6))                             FAIL
    testStaticNetworkConfig((4,))                               FAIL
    testStaticNetworkConfig((6,))                               FAIL
    testStaticSourceRouting(True)                               FAIL
    test_drop_initial_bond_slaves_ip_config                     FAIL
    test_ovirtmgmtm_to_ovs                                      FAIL
    test_remove_bond_under_network                              SKIP: OVS
does not support bridgeless networks
    test_remove_initial_network_nic_ip_config                   OK
    test_rollback                                               FAIL


On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew at redhat.com>
wrote:

> I like the idea but I have the same feelings as Francesco. I think that we
> need to make sure that functional tests for each vertical are stable before
> enabling this process.
>
> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
>
>> adding also infra team for visibility on the change in CI.
>> also inline.
>>
>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani at redhat.com>
>> wrote:
>>
>>> ----- Original Message -----
>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>, "Nir
>>> Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
>>> > <pkliczew at redhat.com>
>>> > Cc: "danken" <danken at redhat.com>, "David Caro" <dcaroest at redhat.com>,
>>> "Eyal Edri" <eedri at redhat.com>
>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
>>> > Subject: Automation CI for vdsm
>>>
>>> [...]
>>> > We want to allow developers to trigger the script once reviews and
>>> > verification are ready (last step before merge). To do so we agreed to
>>> add
>>> > Continues Integration flag for each vdsm patch.
>>
>>
>> This flag will be called 'Workflow' or we can name it otherwise, we just
>> need to choose what makes sense.
>> David/Yaniv - Please correct me if I'm wrong.
>>
>>
>>> Once this flag will be
>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
>>> script
>>> > (adding new button to gerrit is not an option - you can image that
>>> flag as
>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail
>>> > we'll get -1 and once new patchset is ready the developer will remove
>>> the
>>> > +1 and add it back to the Continues Integration flag to re-trigger the
>>> job.
>>> >
>>> > Please ack the process before we move on with that
>>>
>>> Sounds good, even though I'm a little scared (just gut feeling, no
>>> evidence
>>> whatsoever) that this could add even more complexity and fragility to
>>> the jenkins
>>> fleet.
>>>
>>> In the long run, when this is reliable, it will help greatly.
>>> In the short term, I'm scared because this can lead to false positives
>>> and bogus
>>> failures.
>>>
>>> Let me stress I don't have concrete item to share or specific flaws.
>>>
>>> As action item on me, I will find some time next week to check virt
>>> functional tests,
>>> to see if they need some fixes, work reliably and so forth
>>>
>>> > The patch for those scripts still under review and testing -
>>> > https://gerrit.ovirt.org/#/c/48268
>>>
>>> Will review asap.
>>>
>>> --
>>> Francesco Romani
>>> RedHat Engineering Virtualization R & D
>>> Phone: 8261328
>>> IRC: fromani
>>>
>>
>>
>>
>> --
>> Eyal Edri
>> Supervisor, RHEV CI
>> EMEA ENG Virtualization R&D
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>
>


-- 
*Yaniv Bronhaim.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151216/bcc050d9/attachment-0001.html>


More information about the Devel mailing list