[ovirt-devel] Automation CI for vdsm

Yaniv Bronheim ybronhei at redhat.com
Wed Dec 16 16:25:17 UTC 2015


exactly - just posted a patch again that sings all fails as broken .
we'll get the report soon and I'll publish it as well. hope the run will
take less time now

On Wed, Dec 16, 2015 at 6:19 PM, Nir Soffer <nsoffer at redhat.com> wrote:

> Nice, but we cannot enable this until all the tests pass or disabled.
>
> There is no point in broken or flaky functional tests.
>
> On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei at redhat.com>
> wrote:
> > So its not stable. It won't block merges and at least give us report
> after
> > each merge. It takes really long time to run it (because of the tests
> > themselves. Lago things takes maximum 15minutes, but the run last for
> more
> > than 2hrs right now and I suspect functional/storageTests.py gets stuck)
> >
> > Bellow you can see where we stand (before I added python-rtslib package).
> >
> > Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/
> -
> > which enables this run after merges, and I still want you to consider the
> > addition of Automation CI flag to our gerrit so that developer will be
> able
> > to use it as a trigger for the check-merged.sh script run, just to see if
> > their patch fixes\brakes something realted to the functional tests
> >
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/
> - is
> > an example of how the run looks like. I still work to improve the output
> >
> >
> > Please reply and let me know if the idea around the automation flag is
> > acceptable by you.. and please review the patch for comments and acks.
> > We can ask dcaro to add the flag until Friday, otherwise we'll need to
> delay
> > this effort after the holiday..
> >
> >
> > functional.sosPluginTests.SosPluginTest
> >     testSosPlugin                                               OK
> > functional.vmRecoveryTests.RecoveryTests
> >     test_vm_recovery                                            FAIL
> > functional.vmQoSTests.VMQosTests
> >     testSmallVMBallooning                                       FAIL
> > functional.virtTests.VirtTest
> >     testComplexVm                                               FAIL
> >     testHeadlessVm                                              OK
> >     testSimpleVm                                                FAIL
> >     testVmDefinitionGraphics('spice')                           FAIL
> >     testVmDefinitionGraphics('vnc')                             OK
> >     testVmDefinitionLegacyGraphics('qxl')                       FAIL
> >     testVmDefinitionLegacyGraphics('vnc')                       OK
> >     testVmDefinitionMultipleGraphics('spice', 'vnc')            FAIL
> >     testVmDefinitionMultipleGraphics('vnc', 'spice')            FAIL
> >     testVmWithCdrom('self')                                     FAIL
> >     testVmWithCdrom('specParams')                               FAIL
> >     testVmWithCdrom('vmPayload')                                FAIL
> >     testVmWithDevice('hotplugDisk')                             FAIL
> >     testVmWithDevice('hotplugNic')                              FAIL
> >     testVmWithDevice('smartcard')                               FAIL
> >     testVmWithDevice('virtioNic')                               FAIL
> >     testVmWithDevice('virtioRng')                               FAIL
> >     testVmWithSla                                               FAIL
> >     testVmWithStorage('iscsi')                                  SKIP:
> > python-rtslib is not installed.
> >     testVmWithStorage('localfs')                                FAIL
> >     testVmWithStorage('nfs')                                    FAIL
> > functional.storageTests.StorageTest
> >     testCreatePoolErrors                                        OK
> >     testStorage('glusterfs', 0)                                 ERROR
> >     testStorage('glusterfs', 3)                                 ERROR
> >     testStorage('iscsi', 0)                                     SKIP:
> > python-rtslib is not installed.
> >     testStorage('iscsi', 3)                                     SKIP:
> > python-rtslib is not installed.
> >     testStorage('localfs', 0)                                   FAIL
> >     testStorage('localfs', 3)                                   FAIL
> >     testStorage('nfs', 0)                                       FAIL
> >     testStorage('nfs', 3)                                       FAIL
> > functional.networkTests.NetworkTest
> >     testAddVlanedBridgeless                                     ERROR
> >     testAddVlanedBridgeless_oneCommand                          ERROR
> >     testAfterNetworkSetupHook                                   ERROR
> >     testBeforeNetworkSetupHook                                  ERROR
> >     testBondHwAddress(False)                                    ERROR
> >     testBondHwAddress(True)                                     ERROR
> >     testBrokenNetworkReplacement(False)                         ERROR
> >     testBrokenNetworkReplacement(True)                          ERROR
> >     testDelNetworkBondAccumulation                              ERROR
> >     testDelNetworkWithMTU(False)                                ERROR
> >     testDelNetworkWithMTU(True)                                 ERROR
> >     testDelWithoutAdd                                           ERROR
> >     testDhclientLeases(4, 'default')                            ERROR
> >     testDhclientLeases(4, 'local')                              ERROR
> >     testDhclientLeases(6, None)                                 ERROR
> >     testDhcpReplaceNicWithBridge                                ERROR
> >     testFailWithInvalidBondingName(False)                       ERROR
> >     testFailWithInvalidBondingName(True)                        ERROR
> >     testFailWithInvalidBridgeName                               ERROR
> >     testFailWithInvalidIpConfig                                 ERROR
> >     testFailWithInvalidNic(False)                               ERROR
> >     testFailWithInvalidNic(True)                                ERROR
> >     testFailWithInvalidParams(False)                            ERROR
> >     testFailWithInvalidParams(True)                             ERROR
> >     testGetRouteDeviceTo                                        ERROR
> >     testHonorBlockingDhcp                                       SKIP:
> Slow
> > tests are disabled
> >     testIpLinkWrapper                                           ERROR
> >     testLowerMtuDoesNotOverride                                 FAIL
> >     testNoBridgeLeftovers                                       FAIL
> >     testReconfigureBrNetWithVanishedPort                        ERROR
> >     testRedefineBondedNetworkIPs                                ERROR
> >     testRemovingBridgeDoesNotLeaveBridge                        ERROR
> >     testReorderBondingOptions(False)                            ERROR
> >     testReorderBondingOptions(True)                             ERROR
> >     testRestoreNetworksOnlyRestoreUnchangedDevices              ERROR
> >     testRestoreToBlockingDHCP                                   ERROR
> >     testSafeNetworkConfig(False)                                ERROR
> >     testSafeNetworkConfig(True)                                 ERROR
> >     testSelectiveRestoreDuringUpgrade                           ERROR
> >     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
> >     testSetupNetworkOutboundQos(False)                          ERROR
> >     testSetupNetworkOutboundQos(True)                           FAIL
> >     testSetupNetworksActiveSlave                                ERROR
> >     testSetupNetworksAddBadParams(False)                        ERROR
> >     testSetupNetworksAddBadParams(True)                         FAIL
> >     testSetupNetworksAddBondWithManyVlans(False)                ERROR
> >     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
> >     testSetupNetworksAddDelBondedNetwork(False)                 ERROR
> >     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
> >     testSetupNetworksAddDelDhcp(False, (4, 6))                  ERROR
> >     testSetupNetworksAddDelDhcp(False, (4,))                    ERROR
> >     testSetupNetworksAddDelDhcp(False, (6,))                    ERROR
> >     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
> >     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
> >     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
> >     testSetupNetworksAddManyVlans(False)                        ERROR
> >     testSetupNetworksAddManyVlans(True)                         FAIL
> >     testSetupNetworksAddNetworkToNicAfterBondBreaking(False)    ERROR
> >     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
> >     testSetupNetworksAddNetworkToNicAfterBondResizing(False)    ERROR
> >     testSetupNetworksAddNetworkToNicAfterBondResizing(True)     ERROR
> >     testSetupNetworksAddOverExistingBond(False)                 ERROR
> >     testSetupNetworksAddOverExistingBond(True)                  ERROR
> >     testSetupNetworksAddVlan(False)                             ERROR
> >     testSetupNetworksAddVlan(True)                              FAIL
> >     testSetupNetworksConnectivityCheck                          FAIL
> >     testSetupNetworksConvertVlanNetBridgeness                   ERROR
> >     testSetupNetworksDelOneOfBondNets                           ERROR
> >     testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured   ERROR
> >     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
> >     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> Slow
> > tests are disabled
> >     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
> >     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> Slow
> > tests are disabled
> >     testSetupNetworksKeepNetworkOnBondAfterBondResizing(False)  ERROR
> >     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
> >     testSetupNetworksMtus(False)                                ERROR
> >     testSetupNetworksMtus(True)                                 SKIP:
> This
> > test fails because the 2 different networks are getting configured with
> the
> > same MTU. The test should assert that the reported MTUs are equal to the
> > requested ones.
> >     testSetupNetworksMultiMTUsOverBond(False)                   ERROR
> >     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
> >     testSetupNetworksMultiMTUsOverNic(False)                    ERROR
> >     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
> >     testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
> >     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
> >     testSetupNetworksNicless                                    FAIL
> >     testSetupNetworksNiclessBridgeless                          ERROR
> >     testSetupNetworksOverDhcpIface                              ERROR
> >     testSetupNetworksOverExistingBridge                         FAIL
> >     testSetupNetworksReconfigureBridge                          FAIL
> >     testSetupNetworksRemoveBondWithKilledEnslavedNics           ERROR
> >     testSetupNetworksRemoveSlavelessBond                        ERROR
> >     testSetupNetworksResizeBond(False)                          ERROR
> >     testSetupNetworksResizeBond(True)                           ERROR
> >     testSetupNetworksStableBond(False)                          ERROR
> >     testSetupNetworksStableBond(True)                           ERROR
> >     testStaticNetworkConfig((4, 6))                             FAIL
> >     testStaticNetworkConfig((4,))                               FAIL
> >     testStaticNetworkConfig((6,))                               FAIL
> >     testStaticSourceRouting(False)                              ERROR
> >     testStaticSourceRouting(True)                               FAIL
> >     testTwiceAdd(False)                                         ERROR
> >     testTwiceAdd(True)                                          ERROR
> >     testVolatileConfig(False)                                   ERROR
> >     testVolatileConfig(True)                                    ERROR
> >     test_drop_initial_bond_slaves_ip_config                     FAIL
> >     test_getVdsStats                                            ERROR
> >     test_remove_bond_under_network                              FAIL
> >     test_remove_initial_network_nic_ip_config                   FAIL
> >     test_rollback                                               FAIL
> >     test_setupNetworks_bond_with_custom_option                  ERROR
> >     test_setupNetworks_on_external_bond                         ERROR
> >     test_setupNetworks_on_external_vlaned_bond                  ERROR
> > functional.networkTestsOVS.OVSNetworkTest
> >     testHonorBlockingDhcp                                       SKIP:
> Slow
> > tests are disabled
> >     testLowerMtuDoesNotOverride                                 OK
> >     testNoBridgeLeftovers                                       FAIL
> >     testSelectiveRestoreIgnoresVdsmRegParams                    FAIL
> >     testSetupNetworkOutboundQos(True)                           OK
> >     testSetupNetworksAddBadParams(True)                         OK
> >     testSetupNetworksAddBondWithManyVlans(True)                 FAIL
> >     testSetupNetworksAddDelBondedNetwork(True)                  FAIL
> >     testSetupNetworksAddDelDhcp(True, (4, 6))                   FAIL
> >     testSetupNetworksAddDelDhcp(True, (4,))                     FAIL
> >     testSetupNetworksAddDelDhcp(True, (6,))                     FAIL
> >     testSetupNetworksAddManyVlans(True)                         OK
> >     testSetupNetworksAddNetworkToNicAfterBondBreaking(True)     FAIL
> >     testSetupNetworksAddVlan(True)                              FAIL
> >     testSetupNetworksConnectivityCheck                          OK
> >     testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> Slow
> > tests are disabled
> >     testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> Slow
> > tests are disabled
> >     testSetupNetworksKeepNetworkOnBondAfterBondResizing(True)   FAIL
> >     testSetupNetworksMtus(True)                                 SKIP:
> This
> > test fails because the 2 different networks are getting configured with
> the
> > same MTU. The test should assert that the reported MTUs are equal to the
> > requested ones.
> >     testSetupNetworksMultiMTUsOverBond(True)                    FAIL
> >     testSetupNetworksMultiMTUsOverNic(True)                     FAIL
> >     testSetupNetworksNetCompatibilityMultipleNetsSameNic(True)  FAIL
> >     testSetupNetworksNicless                                    OK
> >     testSetupNetworksOverExistingBridge                         OK
> >     testSetupNetworksReconfigureBridge                          FAIL
> >     testStaticNetworkConfig((4, 6))                             FAIL
> >     testStaticNetworkConfig((4,))                               FAIL
> >     testStaticNetworkConfig((6,))                               FAIL
> >     testStaticSourceRouting(True)                               FAIL
> >     test_drop_initial_bond_slaves_ip_config                     FAIL
> >     test_ovirtmgmtm_to_ovs                                      FAIL
> >     test_remove_bond_under_network                              SKIP: OVS
> > does not support bridgeless networks
> >     test_remove_initial_network_nic_ip_config                   OK
> >     test_rollback                                               FAIL
> >
> >
> > On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew at redhat.com>
> > wrote:
> >>
> >> I like the idea but I have the same feelings as Francesco. I think that
> we
> >> need to make sure that functional tests for each vertical are stable
> before
> >> enabling this process.
> >>
> >> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
> >>>
> >>> adding also infra team for visibility on the change in CI.
> >>> also inline.
> >>>
> >>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani at redhat.com>
> >>> wrote:
> >>>>
> >>>> ----- Original Message -----
> >>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
> >>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>, "Nir
> >>>> > Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
> >>>> > <pkliczew at redhat.com>
> >>>> > Cc: "danken" <danken at redhat.com>, "David Caro" <dcaroest at redhat.com
> >,
> >>>> > "Eyal Edri" <eedri at redhat.com>
> >>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
> >>>> > Subject: Automation CI for vdsm
> >>>>
> >>>> [...]
> >>>> > We want to allow developers to trigger the script once reviews and
> >>>> > verification are ready (last step before merge). To do so we agreed
> to
> >>>> > add
> >>>> > Continues Integration flag for each vdsm patch.
> >>>
> >>>
> >>> This flag will be called 'Workflow' or we can name it otherwise, we
> just
> >>> need to choose what makes sense.
> >>> David/Yaniv - Please correct me if I'm wrong.
> >>>
> >>>>
> >>>> Once this flag will be
> >>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
> >>>> > script
> >>>> > (adding new button to gerrit is not an option - you can image that
> >>>> > flag as
> >>>> > a trigger button), on success Jenkins CI flag will turn to +2. on
> fail
> >>>> > we'll get -1 and once new patchset is ready the developer will
> remove
> >>>> > the
> >>>> > +1 and add it back to the Continues Integration flag to re-trigger
> the
> >>>> > job.
> >>>> >
> >>>> > Please ack the process before we move on with that
> >>>>
> >>>> Sounds good, even though I'm a little scared (just gut feeling, no
> >>>> evidence
> >>>> whatsoever) that this could add even more complexity and fragility to
> >>>> the jenkins
> >>>> fleet.
> >>>>
> >>>> In the long run, when this is reliable, it will help greatly.
> >>>> In the short term, I'm scared because this can lead to false positives
> >>>> and bogus
> >>>> failures.
> >>>>
> >>>> Let me stress I don't have concrete item to share or specific flaws.
> >>>>
> >>>> As action item on me, I will find some time next week to check virt
> >>>> functional tests,
> >>>> to see if they need some fixes, work reliably and so forth
> >>>>
> >>>> > The patch for those scripts still under review and testing -
> >>>> > https://gerrit.ovirt.org/#/c/48268
> >>>>
> >>>> Will review asap.
> >>>>
> >>>> --
> >>>> Francesco Romani
> >>>> RedHat Engineering Virtualization R & D
> >>>> Phone: 8261328
> >>>> IRC: fromani
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Eyal Edri
> >>> Supervisor, RHEV CI
> >>> EMEA ENG Virtualization R&D
> >>> Red Hat Israel
> >>>
> >>> phone: +972-9-7692018
> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >>
> >
> >
> >
> > --
> > Yaniv Bronhaim.
>



-- 
*Yaniv Bronhaim.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20151216/9364bf68/attachment-0001.html>


More information about the Devel mailing list