[ovirt-devel] Automation CI for vdsm
Yaniv Bronheim
ybronhei at redhat.com
Tue Dec 22 12:50:19 UTC 2015
OK. so for now the patch [1] skips all storage and OVSNetwork tests
Which still doesn't help much, single run takes more than 3hrs :/ (1:17hr,
for the setup :\ the rest for the tests)
The output looks pretty good though :)
[1] https://gerrit.ovirt.org/#/c/48268/
On Thu, Dec 17, 2015 at 12:13 PM, Petr Horacek <phoracek at redhat.com> wrote:
> Hi,
>
> OVSNetworkTest requires vdsm-hook-ovs installed (tests are not skipped
> if the package is not installed) and openvswitch service running (not
> started automatically). NetworkTest are not able to run together with
> OVSNetworkTest since we are doing ugly inheritance hacking there. My
> apologize for network tests difficulties, it will be resolved soon. If
> you want your life easier, temporary skip OVSNetworkTest suite and it
> should be OK.
>
> Best regards,
> Petr
>
> 2015-12-16 17:25 GMT+01:00 Yaniv Bronheim <ybronhei at redhat.com>:
> > exactly - just posted a patch again that sings all fails as broken .
> > we'll get the report soon and I'll publish it as well. hope the run will
> > take less time now
> >
> > On Wed, Dec 16, 2015 at 6:19 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> >>
> >> Nice, but we cannot enable this until all the tests pass or disabled.
> >>
> >> There is no point in broken or flaky functional tests.
> >>
> >> On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <ybronhei at redhat.com>
> >> wrote:
> >> > So its not stable. It won't block merges and at least give us report
> >> > after
> >> > each merge. It takes really long time to run it (because of the tests
> >> > themselves. Lago things takes maximum 15minutes, but the run last for
> >> > more
> >> > than 2hrs right now and I suspect functional/storageTests.py gets
> stuck)
> >> >
> >> > Bellow you can see where we stand (before I added python-rtslib
> >> > package).
> >> >
> >> > Now, I still want to merge the patch
> https://gerrit.ovirt.org/#/c/48268/
> >> > -
> >> > which enables this run after merges, and I still want you to consider
> >> > the
> >> > addition of Automation CI flag to our gerrit so that developer will be
> >> > able
> >> > to use it as a trigger for the check-merged.sh script run, just to see
> >> > if
> >> > their patch fixes\brakes something realted to the functional tests
> >> >
> >> >
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ -
> >> > is
> >> > an example of how the run looks like. I still work to improve the
> output
> >> >
> >> >
> >> > Please reply and let me know if the idea around the automation flag is
> >> > acceptable by you.. and please review the patch for comments and acks.
> >> > We can ask dcaro to add the flag until Friday, otherwise we'll need to
> >> > delay
> >> > this effort after the holiday..
> >> >
> >> >
> >> > functional.sosPluginTests.SosPluginTest
> >> > testSosPlugin OK
> >> > functional.vmRecoveryTests.RecoveryTests
> >> > test_vm_recovery FAIL
> >> > functional.vmQoSTests.VMQosTests
> >> > testSmallVMBallooning FAIL
> >> > functional.virtTests.VirtTest
> >> > testComplexVm FAIL
> >> > testHeadlessVm OK
> >> > testSimpleVm FAIL
> >> > testVmDefinitionGraphics('spice') FAIL
> >> > testVmDefinitionGraphics('vnc') OK
> >> > testVmDefinitionLegacyGraphics('qxl') FAIL
> >> > testVmDefinitionLegacyGraphics('vnc') OK
> >> > testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL
> >> > testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL
> >> > testVmWithCdrom('self') FAIL
> >> > testVmWithCdrom('specParams') FAIL
> >> > testVmWithCdrom('vmPayload') FAIL
> >> > testVmWithDevice('hotplugDisk') FAIL
> >> > testVmWithDevice('hotplugNic') FAIL
> >> > testVmWithDevice('smartcard') FAIL
> >> > testVmWithDevice('virtioNic') FAIL
> >> > testVmWithDevice('virtioRng') FAIL
> >> > testVmWithSla FAIL
> >> > testVmWithStorage('iscsi') SKIP:
> >> > python-rtslib is not installed.
> >> > testVmWithStorage('localfs') FAIL
> >> > testVmWithStorage('nfs') FAIL
> >> > functional.storageTests.StorageTest
> >> > testCreatePoolErrors OK
> >> > testStorage('glusterfs', 0) ERROR
> >> > testStorage('glusterfs', 3) ERROR
> >> > testStorage('iscsi', 0) SKIP:
> >> > python-rtslib is not installed.
> >> > testStorage('iscsi', 3) SKIP:
> >> > python-rtslib is not installed.
> >> > testStorage('localfs', 0) FAIL
> >> > testStorage('localfs', 3) FAIL
> >> > testStorage('nfs', 0) FAIL
> >> > testStorage('nfs', 3) FAIL
> >> > functional.networkTests.NetworkTest
> >> > testAddVlanedBridgeless ERROR
> >> > testAddVlanedBridgeless_oneCommand ERROR
> >> > testAfterNetworkSetupHook ERROR
> >> > testBeforeNetworkSetupHook ERROR
> >> > testBondHwAddress(False) ERROR
> >> > testBondHwAddress(True) ERROR
> >> > testBrokenNetworkReplacement(False) ERROR
> >> > testBrokenNetworkReplacement(True) ERROR
> >> > testDelNetworkBondAccumulation ERROR
> >> > testDelNetworkWithMTU(False) ERROR
> >> > testDelNetworkWithMTU(True) ERROR
> >> > testDelWithoutAdd ERROR
> >> > testDhclientLeases(4, 'default') ERROR
> >> > testDhclientLeases(4, 'local') ERROR
> >> > testDhclientLeases(6, None) ERROR
> >> > testDhcpReplaceNicWithBridge ERROR
> >> > testFailWithInvalidBondingName(False) ERROR
> >> > testFailWithInvalidBondingName(True) ERROR
> >> > testFailWithInvalidBridgeName ERROR
> >> > testFailWithInvalidIpConfig ERROR
> >> > testFailWithInvalidNic(False) ERROR
> >> > testFailWithInvalidNic(True) ERROR
> >> > testFailWithInvalidParams(False) ERROR
> >> > testFailWithInvalidParams(True) ERROR
> >> > testGetRouteDeviceTo ERROR
> >> > testHonorBlockingDhcp SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testIpLinkWrapper ERROR
> >> > testLowerMtuDoesNotOverride FAIL
> >> > testNoBridgeLeftovers FAIL
> >> > testReconfigureBrNetWithVanishedPort ERROR
> >> > testRedefineBondedNetworkIPs ERROR
> >> > testRemovingBridgeDoesNotLeaveBridge ERROR
> >> > testReorderBondingOptions(False) ERROR
> >> > testReorderBondingOptions(True) ERROR
> >> > testRestoreNetworksOnlyRestoreUnchangedDevices ERROR
> >> > testRestoreToBlockingDHCP ERROR
> >> > testSafeNetworkConfig(False) ERROR
> >> > testSafeNetworkConfig(True) ERROR
> >> > testSelectiveRestoreDuringUpgrade ERROR
> >> > testSelectiveRestoreIgnoresVdsmRegParams FAIL
> >> > testSetupNetworkOutboundQos(False) ERROR
> >> > testSetupNetworkOutboundQos(True) FAIL
> >> > testSetupNetworksActiveSlave ERROR
> >> > testSetupNetworksAddBadParams(False) ERROR
> >> > testSetupNetworksAddBadParams(True) FAIL
> >> > testSetupNetworksAddBondWithManyVlans(False) ERROR
> >> > testSetupNetworksAddBondWithManyVlans(True) FAIL
> >> > testSetupNetworksAddDelBondedNetwork(False) ERROR
> >> > testSetupNetworksAddDelBondedNetwork(True) FAIL
> >> > testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR
> >> > testSetupNetworksAddDelDhcp(False, (4,)) ERROR
> >> > testSetupNetworksAddDelDhcp(False, (6,)) ERROR
> >> > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> >> > testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> >> > testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> >> > testSetupNetworksAddManyVlans(False) ERROR
> >> > testSetupNetworksAddManyVlans(True) FAIL
> >> > testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR
> >> > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> >> > testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR
> >> > testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR
> >> > testSetupNetworksAddOverExistingBond(False) ERROR
> >> > testSetupNetworksAddOverExistingBond(True) ERROR
> >> > testSetupNetworksAddVlan(False) ERROR
> >> > testSetupNetworksAddVlan(True) FAIL
> >> > testSetupNetworksConnectivityCheck FAIL
> >> > testSetupNetworksConvertVlanNetBridgeness ERROR
> >> > testSetupNetworksDelOneOfBondNets ERROR
> >> > testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR
> >> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
> >> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
> >> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR
> >> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> >> > testSetupNetworksMtus(False) ERROR
> >> > testSetupNetworksMtus(True) SKIP:
> >> > This
> >> > test fails because the 2 different networks are getting configured
> with
> >> > the
> >> > same MTU. The test should assert that the reported MTUs are equal to
> the
> >> > requested ones.
> >> > testSetupNetworksMultiMTUsOverBond(False) ERROR
> >> > testSetupNetworksMultiMTUsOverBond(True) FAIL
> >> > testSetupNetworksMultiMTUsOverNic(False) ERROR
> >> > testSetupNetworksMultiMTUsOverNic(True) FAIL
> >> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
> >> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> >> > testSetupNetworksNicless FAIL
> >> > testSetupNetworksNiclessBridgeless ERROR
> >> > testSetupNetworksOverDhcpIface ERROR
> >> > testSetupNetworksOverExistingBridge FAIL
> >> > testSetupNetworksReconfigureBridge FAIL
> >> > testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR
> >> > testSetupNetworksRemoveSlavelessBond ERROR
> >> > testSetupNetworksResizeBond(False) ERROR
> >> > testSetupNetworksResizeBond(True) ERROR
> >> > testSetupNetworksStableBond(False) ERROR
> >> > testSetupNetworksStableBond(True) ERROR
> >> > testStaticNetworkConfig((4, 6)) FAIL
> >> > testStaticNetworkConfig((4,)) FAIL
> >> > testStaticNetworkConfig((6,)) FAIL
> >> > testStaticSourceRouting(False) ERROR
> >> > testStaticSourceRouting(True) FAIL
> >> > testTwiceAdd(False) ERROR
> >> > testTwiceAdd(True) ERROR
> >> > testVolatileConfig(False) ERROR
> >> > testVolatileConfig(True) ERROR
> >> > test_drop_initial_bond_slaves_ip_config FAIL
> >> > test_getVdsStats ERROR
> >> > test_remove_bond_under_network FAIL
> >> > test_remove_initial_network_nic_ip_config FAIL
> >> > test_rollback FAIL
> >> > test_setupNetworks_bond_with_custom_option ERROR
> >> > test_setupNetworks_on_external_bond ERROR
> >> > test_setupNetworks_on_external_vlaned_bond ERROR
> >> > functional.networkTestsOVS.OVSNetworkTest
> >> > testHonorBlockingDhcp SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testLowerMtuDoesNotOverride OK
> >> > testNoBridgeLeftovers FAIL
> >> > testSelectiveRestoreIgnoresVdsmRegParams FAIL
> >> > testSetupNetworkOutboundQos(True) OK
> >> > testSetupNetworksAddBadParams(True) OK
> >> > testSetupNetworksAddBondWithManyVlans(True) FAIL
> >> > testSetupNetworksAddDelBondedNetwork(True) FAIL
> >> > testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
> >> > testSetupNetworksAddDelDhcp(True, (4,)) FAIL
> >> > testSetupNetworksAddDelDhcp(True, (6,)) FAIL
> >> > testSetupNetworksAddManyVlans(True) OK
> >> > testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
> >> > testSetupNetworksAddVlan(True) FAIL
> >> > testSetupNetworksConnectivityCheck OK
> >> > testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP:
> >> > Slow
> >> > tests are disabled
> >> > testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
> >> > testSetupNetworksMtus(True) SKIP:
> >> > This
> >> > test fails because the 2 different networks are getting configured
> with
> >> > the
> >> > same MTU. The test should assert that the reported MTUs are equal to
> the
> >> > requested ones.
> >> > testSetupNetworksMultiMTUsOverBond(True) FAIL
> >> > testSetupNetworksMultiMTUsOverNic(True) FAIL
> >> > testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
> >> > testSetupNetworksNicless OK
> >> > testSetupNetworksOverExistingBridge OK
> >> > testSetupNetworksReconfigureBridge FAIL
> >> > testStaticNetworkConfig((4, 6)) FAIL
> >> > testStaticNetworkConfig((4,)) FAIL
> >> > testStaticNetworkConfig((6,)) FAIL
> >> > testStaticSourceRouting(True) FAIL
> >> > test_drop_initial_bond_slaves_ip_config FAIL
> >> > test_ovirtmgmtm_to_ovs FAIL
> >> > test_remove_bond_under_network SKIP:
> >> > OVS
> >> > does not support bridgeless networks
> >> > test_remove_initial_network_nic_ip_config OK
> >> > test_rollback FAIL
> >> >
> >> >
> >> > On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <
> pkliczew at redhat.com>
> >> > wrote:
> >> >>
> >> >> I like the idea but I have the same feelings as Francesco. I think
> that
> >> >> we
> >> >> need to make sure that functional tests for each vertical are stable
> >> >> before
> >> >> enabling this process.
> >> >>
> >> >> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
> >> >>>
> >> >>> adding also infra team for visibility on the change in CI.
> >> >>> also inline.
> >> >>>
> >> >>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <
> fromani at redhat.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> ----- Original Message -----
> >> >>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
> >> >>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>,
> "Nir
> >> >>>> > Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
> >> >>>> > <pkliczew at redhat.com>
> >> >>>> > Cc: "danken" <danken at redhat.com>, "David Caro"
> >> >>>> > <dcaroest at redhat.com>,
> >> >>>> > "Eyal Edri" <eedri at redhat.com>
> >> >>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
> >> >>>> > Subject: Automation CI for vdsm
> >> >>>>
> >> >>>> [...]
> >> >>>> > We want to allow developers to trigger the script once reviews
> and
> >> >>>> > verification are ready (last step before merge). To do so we
> agreed
> >> >>>> > to
> >> >>>> > add
> >> >>>> > Continues Integration flag for each vdsm patch.
> >> >>>
> >> >>>
> >> >>> This flag will be called 'Workflow' or we can name it otherwise, we
> >> >>> just
> >> >>> need to choose what makes sense.
> >> >>> David/Yaniv - Please correct me if I'm wrong.
> >> >>>
> >> >>>>
> >> >>>> Once this flag will be
> >> >>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
> >> >>>> > script
> >> >>>> > (adding new button to gerrit is not an option - you can image
> that
> >> >>>> > flag as
> >> >>>> > a trigger button), on success Jenkins CI flag will turn to +2. on
> >> >>>> > fail
> >> >>>> > we'll get -1 and once new patchset is ready the developer will
> >> >>>> > remove
> >> >>>> > the
> >> >>>> > +1 and add it back to the Continues Integration flag to
> re-trigger
> >> >>>> > the
> >> >>>> > job.
> >> >>>> >
> >> >>>> > Please ack the process before we move on with that
> >> >>>>
> >> >>>> Sounds good, even though I'm a little scared (just gut feeling, no
> >> >>>> evidence
> >> >>>> whatsoever) that this could add even more complexity and fragility
> to
> >> >>>> the jenkins
> >> >>>> fleet.
> >> >>>>
> >> >>>> In the long run, when this is reliable, it will help greatly.
> >> >>>> In the short term, I'm scared because this can lead to false
> >> >>>> positives
> >> >>>> and bogus
> >> >>>> failures.
> >> >>>>
> >> >>>> Let me stress I don't have concrete item to share or specific
> flaws.
> >> >>>>
> >> >>>> As action item on me, I will find some time next week to check virt
> >> >>>> functional tests,
> >> >>>> to see if they need some fixes, work reliably and so forth
> >> >>>>
> >> >>>> > The patch for those scripts still under review and testing -
> >> >>>> > https://gerrit.ovirt.org/#/c/48268
> >> >>>>
> >> >>>> Will review asap.
> >> >>>>
> >> >>>> --
> >> >>>> Francesco Romani
> >> >>>> RedHat Engineering Virtualization R & D
> >> >>>> Phone: 8261328
> >> >>>> IRC: fromani
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Eyal Edri
> >> >>> Supervisor, RHEV CI
> >> >>> EMEA ENG Virtualization R&D
> >> >>> Red Hat Israel
> >> >>>
> >> >>> phone: +972-9-7692018
> >> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Yaniv Bronhaim.
> >
> >
> >
> >
> > --
> > Yaniv Bronhaim.
> >
> > _______________________________________________
> > Devel mailing list
> > Devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
>
--
*Yaniv Bronhaim.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20151222/5a03bcf6/attachment.html>
More information about the Infra
mailing list