Automation CI for vdsm
Yaniv Bronheim
ybronhei at redhat.com
Wed Dec 16 16:25:17 UTC 2015
exactly - just posted a patch again that sings all fails as broken .
we'll get the report soon and I'll publish it as well. hope the run will
take less time now
On Wed, Dec 16, 2015 at 6:19 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> Nice, but we cannot enable this until all the tests pass or disabled.
>
> There is no point in broken or flaky functional tests.
>
> On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei at redhat.com>
> wrote:
> > So its not stable. It won't block merges and at least give us report
> after
> > each merge. It takes really long time to run it (because of the tests
> > themselves. Lago things takes maximum 15minutes, but the run last for
> more
> > than 2hrs right now and I suspect functional/storageTests.py gets stuck)
> >
> > Bellow you can see where we stand (before I added python-rtslib package).
> >
> > Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/
> -
> > which enables this run after merges, and I still want you to consider the
> > addition of Automation CI flag to our gerrit so that developer will be
> able
> > to use it as a trigger for the check-merged.sh script run, just to see if
> > their patch fixes\brakes something realted to the functional tests
> >
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/
> - is
> > an example of how the run looks like. I still work to improve the output
> >
> >
> > Please reply and let me know if the idea around the automation flag is
> > acceptable by you.. and please review the patch for comments and acks.
> > We can ask dcaro to add the flag until Friday, otherwise we'll need to
> delay
> > this effort after the holiday..
> >
> >
> > functional.sosPluginTests.SosPluginTest
> > testSosPlugin OK
> > functional.vmRecoveryTests.RecoveryTests
> > test_vm_recovery FAIL
> > functional.vmQoSTests.VMQosTests
> > testSmallVMBallooning FAIL
> > functional.virtTests.VirtTest
> > testComplexVm FAIL
> > testHeadlessVm OK
> > testSimpleVm FAIL
> > testVmDefinitionGraphics('spice') FAIL
> > testVmDefinitionGraphics('vnc') OK
> > testVmDefinitionLegacyGraphics('qxl') FAIL
> > testVmDefinitionLegacyGraphics('vnc') OK
> > testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL
> > testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL
> > testVmWithCdrom('self') FAIL
> > testVmWithCdrom('specParams') FAIL
> > testVmWithCdrom('vmPayload') FAIL
> > testVmWithDevice('hotplugDisk') FAIL
> > testVmWithDevice('hotplugNic') FAIL
> > testVmWithDevice('smartcard') FAIL
> > testVmWithDevice('virtioNic') FAIL
> > testVmWithDevice('virtioRng') FAIL
> > testVmWithSla FAIL
> > testVmWithStorage('iscsi') SKIP:
> > python-rtslib is not installed.
> > testVmWithStorage('localfs') FAIL
> > testVmWithStorage('nfs') FAIL
> > functional.storageTests.StorageTest
> > testCreatePoolErrors OK
> > testStorage('glusterfs', 0) ERROR
> > testStorage('glusterfs', 3) ERROR
> > testStorage('iscsi', 0) SKIP:
> > python-rtslib is not installed.
> > testStorage('iscsi', 3) SKIP:
> > python-rtslib is not installed.
> > testStorage('localfs', 0) FAIL
> > testStorage('localfs', 3) FAIL
> > testStorage('nfs', 0) FAIL
> > testStorage('nfs', 3) FAIL
> > functional.networkTests.NetworkTest
> > testAddVlanedBridgeless ERROR
> > testAddVlanedBridgeless_oneCommand ERROR
> > testAfterNetworkSetupHook ERROR
> > testBeforeNetworkSetupHook ERROR
> > testBondHwAddress(False) ERROR
> > testBondHwAddress(True) ERROR
> > testBrokenNetworkReplacement(False) ERROR
> > testBrokenNetworkReplacement(True) ERROR
> > testDelNetworkBondAccumulation ERROR
> > testDelNetworkWithMTU(False) ERROR
> > testDelNetworkWithMTU(True) ERROR
> > testDelWithoutAdd ERROR
> > testDhclientLeases(4, 'default') ERROR
> > testDhclientLeases(4, 'local') ERROR
> > testDhclientLeases(6, None) ERROR
> > testDhcpReplaceNicWithBridge ERROR
> > testFailWithInvalidBondingName(False) ERROR
> > testFailWithInvalidBondingName(True) ERROR
> > testFailWithInvalidBridgeName ERROR
> > testFailWithInvalidIpConfig ERROR
> > testFailWithInvalidNic(False) ERROR
> > testFailWithInvalidNic(True) ERROR
> > testFailWithInvalidParams(False) ERROR
> > testFailWithInvalidParams(True) ERROR
> > testGetRouteDeviceTo ERROR
> > testHonorBlockingDhcp SKIP:
> Slow
> > tests are disabled
> > testIpLinkWrapper ERROR
> > testLowerMtuDoesNotOverride FAIL
> > testNoBridgeLeftovers FAIL
> > testReconfigureBrNetWithVanishedPort ERROR
> > testRedefineBondedNetworkIPs ERROR
> > testRemovingBridgeDoesNotLeaveBridge ERROR
> > testReorderBondingOptions(False) ERROR
> > testReorderBondingOptions(True) ERROR
> > testRestoreNetworksOnlyRestoreUnchangedDevices ERROR
> > testRestoreToBlockingDHCP ERROR
> > testSafeNetworkConfig(False) ERROR
> > testSafeNetworkConfig(True) ERROR
> > testSelectiveRestoreDuringUpgrade ERROR
> > testSelectiveRestoreIgnoresVdsmRegParams FAIL
> > testSetupNetworkOutboundQos(False) ERROR
> > testSetupNetworkOutboundQos(True) FAIL
> > testSetupNetworksActiveSlave ERROR
> > testSetupNetworksAddBadParams(False) ERROR
> > testSetupNetworksAddBadParams(True) FAIL
> > testSetupNetworksAddBondWithManyVlans(False) ERROR
> > testSetupNetworksAddBondWithManyVlans(True) FAIL
> > testSetupNetworksAddDelBondedNetwork(False) ERROR
> > testSetupNetworksAddDelBondedNetwork(True) FAIL
> > testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR
> > testSetupNetworksAddDelDhcp(False, (4,)) ERROR
> > testSetupNetworksAddDelDhcp(False, (6,)) ERROR
> > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> > testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> > testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> > testSetupNetworksAddManyVlans(False) ERROR
> > testSetupNetworksAddManyVlans(True) FAIL
> > testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR
> > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> > testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR
> > testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR
> > testSetupNetworksAddOverExistingBond(False) ERROR
> > testSetupNetworksAddOverExistingBond(True) ERROR
> > testSetupNetworksAddVlan(False) ERROR
> > testSetupNetworksAddVlan(True) FAIL
> > testSetupNetworksConnectivityCheck FAIL
> > testSetupNetworksConvertVlanNetBridgeness ERROR
> > testSetupNetworksDelOneOfBondNets ERROR
> > testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR
> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> Slow
> > tests are disabled
> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> Slow
> > tests are disabled
> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR
> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> > testSetupNetworksMtus(False) ERROR
> > testSetupNetworksMtus(True) SKIP:
> This
> > test fails because the 2 different networks are getting configured with
> the
> > same MTU. The test should assert that the reported MTUs are equal to the
> > requested ones.
> > testSetupNetworksMultiMTUsOverBond(False) ERROR
> > testSetupNetworksMultiMTUsOverBond(True) FAIL
> > testSetupNetworksMultiMTUsOverNic(False) ERROR
> > testSetupNetworksMultiMTUsOverNic(True) FAIL
> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> > testSetupNetworksNicless FAIL
> > testSetupNetworksNiclessBridgeless ERROR
> > testSetupNetworksOverDhcpIface ERROR
> > testSetupNetworksOverExistingBridge FAIL
> > testSetupNetworksReconfigureBridge FAIL
> > testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR
> > testSetupNetworksRemoveSlavelessBond ERROR
> > testSetupNetworksResizeBond(False) ERROR
> > testSetupNetworksResizeBond(True) ERROR
> > testSetupNetworksStableBond(False) ERROR
> > testSetupNetworksStableBond(True) ERROR
> > testStaticNetworkConfig((4, 6)) FAIL
> > testStaticNetworkConfig((4,)) FAIL
> > testStaticNetworkConfig((6,)) FAIL
> > testStaticSourceRouting(False) ERROR
> > testStaticSourceRouting(True) FAIL
> > testTwiceAdd(False) ERROR
> > testTwiceAdd(True) ERROR
> > testVolatileConfig(False) ERROR
> > testVolatileConfig(True) ERROR
> > test_drop_initial_bond_slaves_ip_config FAIL
> > test_getVdsStats ERROR
> > test_remove_bond_under_network FAIL
> > test_remove_initial_network_nic_ip_config FAIL
> > test_rollback FAIL
> > test_setupNetworks_bond_with_custom_option ERROR
> > test_setupNetworks_on_external_bond ERROR
> > test_setupNetworks_on_external_vlaned_bond ERROR
> > functional.networkTestsOVS.OVSNetworkTest
> > testHonorBlockingDhcp SKIP:
> Slow
> > tests are disabled
> > testLowerMtuDoesNotOverride OK
> > testNoBridgeLeftovers FAIL
> > testSelectiveRestoreIgnoresVdsmRegParams FAIL
> > testSetupNetworkOutboundQos(True) OK
> > testSetupNetworksAddBadParams(True) OK
> > testSetupNetworksAddBondWithManyVlans(True) FAIL
> > testSetupNetworksAddDelBondedNetwork(True) FAIL
> > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> > testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> > testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> > testSetupNetworksAddManyVlans(True) OK
> > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> > testSetupNetworksAddVlan(True) FAIL
> > testSetupNetworksConnectivityCheck OK
> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> Slow
> > tests are disabled
> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> Slow
> > tests are disabled
> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> > testSetupNetworksMtus(True) SKIP:
> This
> > test fails because the 2 different networks are getting configured with
> the
> > same MTU. The test should assert that the reported MTUs are equal to the
> > requested ones.
> > testSetupNetworksMultiMTUsOverBond(True) FAIL
> > testSetupNetworksMultiMTUsOverNic(True) FAIL
> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> > testSetupNetworksNicless OK
> > testSetupNetworksOverExistingBridge OK
> > testSetupNetworksReconfigureBridge FAIL
> > testStaticNetworkConfig((4, 6)) FAIL
> > testStaticNetworkConfig((4,)) FAIL
> > testStaticNetworkConfig((6,)) FAIL
> > testStaticSourceRouting(True) FAIL
> > test_drop_initial_bond_slaves_ip_config FAIL
> > test_ovirtmgmtm_to_ovs FAIL
> > test_remove_bond_under_network SKIP: OVS
> > does not support bridgeless networks
> > test_remove_initial_network_nic_ip_config OK
> > test_rollback FAIL
> >
> >
> > On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew at redhat.com>
> > wrote:
> >>
> >> I like the idea but I have the same feelings as Francesco. I think that
> we
> >> need to make sure that functional tests for each vertical are stable
> before
> >> enabling this process.
> >>
> >> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
> >>>
> >>> adding also infra team for visibility on the change in CI.
> >>> also inline.
> >>>
> >>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani at redhat.com>
> >>> wrote:
> >>>>
> >>>> ----- Original Message -----
> >>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
> >>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>, "Nir
> >>>> > Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
> >>>> > <pkliczew at redhat.com>
> >>>> > Cc: "danken" <danken at redhat.com>, "David Caro" <dcaroest at redhat.com
> >,
> >>>> > "Eyal Edri" <eedri at redhat.com>
> >>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
> >>>> > Subject: Automation CI for vdsm
> >>>>
> >>>> [...]
> >>>> > We want to allow developers to trigger the script once reviews and
> >>>> > verification are ready (last step before merge). To do so we agreed
> to
> >>>> > add
> >>>> > Continues Integration flag for each vdsm patch.
> >>>
> >>>
> >>> This flag will be called 'Workflow' or we can name it otherwise, we
> just
> >>> need to choose what makes sense.
> >>> David/Yaniv - Please correct me if I'm wrong.
> >>>
> >>>>
> >>>> Once this flag will be
> >>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
> >>>> > script
> >>>> > (adding new button to gerrit is not an option - you can image that
> >>>> > flag as
> >>>> > a trigger button), on success Jenkins CI flag will turn to +2. on
> fail
> >>>> > we'll get -1 and once new patchset is ready the developer will
> remove
> >>>> > the
> >>>> > +1 and add it back to the Continues Integration flag to re-trigger
> the
> >>>> > job.
> >>>> >
> >>>> > Please ack the process before we move on with that
> >>>>
> >>>> Sounds good, even though I'm a little scared (just gut feeling, no
> >>>> evidence
> >>>> whatsoever) that this could add even more complexity and fragility to
> >>>> the jenkins
> >>>> fleet.
> >>>>
> >>>> In the long run, when this is reliable, it will help greatly.
> >>>> In the short term, I'm scared because this can lead to false positives
> >>>> and bogus
> >>>> failures.
> >>>>
> >>>> Let me stress I don't have concrete item to share or specific flaws.
> >>>>
> >>>> As action item on me, I will find some time next week to check virt
> >>>> functional tests,
> >>>> to see if they need some fixes, work reliably and so forth
> >>>>
> >>>> > The patch for those scripts still under review and testing -
> >>>> > https://gerrit.ovirt.org/#/c/48268
> >>>>
> >>>> Will review asap.
> >>>>
> >>>> --
> >>>> Francesco Romani
> >>>> RedHat Engineering Virtualization R & D
> >>>> Phone: 8261328
> >>>> IRC: fromani
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Eyal Edri
> >>> Supervisor, RHEV CI
> >>> EMEA ENG Virtualization R&D
> >>> Red Hat Israel
> >>>
> >>> phone: +972-9-7692018
> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >>
> >
> >
> >
> > --
> > Yaniv Bronhaim.
>
--
*Yaniv Bronhaim.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20151216/9364bf68/attachment.html>
More information about the Infra
mailing list