Automation CI for vdsm
Yaniv Bronheim
ybronhei at redhat.com
Wed Dec 16 15:34:54 UTC 2015
So its not stable. It won't block merges and at least give us report after
each merge. It takes really long time to run it (because of the tests
themselves. Lago things takes maximum 15minutes, but the run last for more
than 2hrs right now and I suspect functional/storageTests.py gets stuck)
Bellow you can see where we stand (before I added python-rtslib package).
Now, I still want to merge the patch https://gerrit.ovirt.org/#/c/48268/ -
which enables this run after merges, and I still want you to consider the
addition of Automation CI flag to our gerrit so that developer will be able
to use it as a trigger for the check-merged.sh script run, just to see if
their patch fixes\brakes something realted to the functional tests
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/ - is
an example of how the run looks like. I still work to improve the output
Please reply and let me know if the idea around the automation flag is
acceptable by you.. and please review the patch for comments and acks.
We can ask dcaro to add the flag until Friday, otherwise we'll need to
delay this effort after the holiday..
functional.sosPluginTests.SosPluginTest
testSosPlugin OK
functional.vmRecoveryTests.RecoveryTests
test_vm_recovery FAIL
functional.vmQoSTests.VMQosTests
testSmallVMBallooning FAIL
functional.virtTests.VirtTest
testComplexVm FAIL
testHeadlessVm OK
testSimpleVm FAIL
testVmDefinitionGraphics('spice') FAIL
testVmDefinitionGraphics('vnc') OK
testVmDefinitionLegacyGraphics('qxl') FAIL
testVmDefinitionLegacyGraphics('vnc') OK
testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL
testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL
testVmWithCdrom('self') FAIL
testVmWithCdrom('specParams') FAIL
testVmWithCdrom('vmPayload') FAIL
testVmWithDevice('hotplugDisk') FAIL
testVmWithDevice('hotplugNic') FAIL
testVmWithDevice('smartcard') FAIL
testVmWithDevice('virtioNic') FAIL
testVmWithDevice('virtioRng') FAIL
testVmWithSla FAIL
testVmWithStorage('iscsi') SKIP:
python-rtslib is not installed.
testVmWithStorage('localfs') FAIL
testVmWithStorage('nfs') FAIL
functional.storageTests.StorageTest
testCreatePoolErrors OK
testStorage('glusterfs', 0) ERROR
testStorage('glusterfs', 3) ERROR
testStorage('iscsi', 0) SKIP:
python-rtslib is not installed.
testStorage('iscsi', 3) SKIP:
python-rtslib is not installed.
testStorage('localfs', 0) FAIL
testStorage('localfs', 3) FAIL
testStorage('nfs', 0) FAIL
testStorage('nfs', 3) FAIL
functional.networkTests.NetworkTest
testAddVlanedBridgeless ERROR
testAddVlanedBridgeless_oneCommand ERROR
testAfterNetworkSetupHook ERROR
testBeforeNetworkSetupHook ERROR
testBondHwAddress(False) ERROR
testBondHwAddress(True) ERROR
testBrokenNetworkReplacement(False) ERROR
testBrokenNetworkReplacement(True) ERROR
testDelNetworkBondAccumulation ERROR
testDelNetworkWithMTU(False) ERROR
testDelNetworkWithMTU(True) ERROR
testDelWithoutAdd ERROR
testDhclientLeases(4, 'default') ERROR
testDhclientLeases(4, 'local') ERROR
testDhclientLeases(6, None) ERROR
testDhcpReplaceNicWithBridge ERROR
testFailWithInvalidBondingName(False) ERROR
testFailWithInvalidBondingName(True) ERROR
testFailWithInvalidBridgeName ERROR
testFailWithInvalidIpConfig ERROR
testFailWithInvalidNic(False) ERROR
testFailWithInvalidNic(True) ERROR
testFailWithInvalidParams(False) ERROR
testFailWithInvalidParams(True) ERROR
testGetRouteDeviceTo ERROR
testHonorBlockingDhcp SKIP: Slow
tests are disabled
testIpLinkWrapper ERROR
testLowerMtuDoesNotOverride FAIL
testNoBridgeLeftovers FAIL
testReconfigureBrNetWithVanishedPort ERROR
testRedefineBondedNetworkIPs ERROR
testRemovingBridgeDoesNotLeaveBridge ERROR
testReorderBondingOptions(False) ERROR
testReorderBondingOptions(True) ERROR
testRestoreNetworksOnlyRestoreUnchangedDevices ERROR
testRestoreToBlockingDHCP ERROR
testSafeNetworkConfig(False) ERROR
testSafeNetworkConfig(True) ERROR
testSelectiveRestoreDuringUpgrade ERROR
testSelectiveRestoreIgnoresVdsmRegParams FAIL
testSetupNetworkOutboundQos(False) ERROR
testSetupNetworkOutboundQos(True) FAIL
testSetupNetworksActiveSlave ERROR
testSetupNetworksAddBadParams(False) ERROR
testSetupNetworksAddBadParams(True) FAIL
testSetupNetworksAddBondWithManyVlans(False) ERROR
testSetupNetworksAddBondWithManyVlans(True) FAIL
testSetupNetworksAddDelBondedNetwork(False) ERROR
testSetupNetworksAddDelBondedNetwork(True) FAIL
testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR
testSetupNetworksAddDelDhcp(False, (4,)) ERROR
testSetupNetworksAddDelDhcp(False, (6,)) ERROR
testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
testSetupNetworksAddDelDhcp(True, (4,)) FAIL
testSetupNetworksAddDelDhcp(True, (6,)) FAIL
testSetupNetworksAddManyVlans(False) ERROR
testSetupNetworksAddManyVlans(True) FAIL
testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR
testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR
testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR
testSetupNetworksAddOverExistingBond(False) ERROR
testSetupNetworksAddOverExistingBond(True) ERROR
testSetupNetworksAddVlan(False) ERROR
testSetupNetworksAddVlan(True) FAIL
testSetupNetworksConnectivityCheck FAIL
testSetupNetworksConvertVlanNetBridgeness ERROR
testSetupNetworksDelOneOfBondNets ERROR
testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR
testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR
testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
tests are disabled
testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR
testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
tests are disabled
testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR
testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
testSetupNetworksMtus(False) ERROR
testSetupNetworksMtus(True) SKIP: This
test fails because the 2 different networks are getting configured with the
same MTU. The test should assert that the reported MTUs are equal to the
requested ones.
testSetupNetworksMultiMTUsOverBond(False) ERROR
testSetupNetworksMultiMTUsOverBond(True) FAIL
testSetupNetworksMultiMTUsOverNic(False) ERROR
testSetupNetworksMultiMTUsOverNic(True) FAIL
testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR
testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
testSetupNetworksNicless FAIL
testSetupNetworksNiclessBridgeless ERROR
testSetupNetworksOverDhcpIface ERROR
testSetupNetworksOverExistingBridge FAIL
testSetupNetworksReconfigureBridge FAIL
testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR
testSetupNetworksRemoveSlavelessBond ERROR
testSetupNetworksResizeBond(False) ERROR
testSetupNetworksResizeBond(True) ERROR
testSetupNetworksStableBond(False) ERROR
testSetupNetworksStableBond(True) ERROR
testStaticNetworkConfig((4, 6)) FAIL
testStaticNetworkConfig((4,)) FAIL
testStaticNetworkConfig((6,)) FAIL
testStaticSourceRouting(False) ERROR
testStaticSourceRouting(True) FAIL
testTwiceAdd(False) ERROR
testTwiceAdd(True) ERROR
testVolatileConfig(False) ERROR
testVolatileConfig(True) ERROR
test_drop_initial_bond_slaves_ip_config FAIL
test_getVdsStats ERROR
test_remove_bond_under_network FAIL
test_remove_initial_network_nic_ip_config FAIL
test_rollback FAIL
test_setupNetworks_bond_with_custom_option ERROR
test_setupNetworks_on_external_bond ERROR
test_setupNetworks_on_external_vlaned_bond ERROR
functional.networkTestsOVS.OVSNetworkTest
testHonorBlockingDhcp SKIP: Slow
tests are disabled
testLowerMtuDoesNotOverride OK
testNoBridgeLeftovers FAIL
testSelectiveRestoreIgnoresVdsmRegParams FAIL
testSetupNetworkOutboundQos(True) OK
testSetupNetworksAddBadParams(True) OK
testSetupNetworksAddBondWithManyVlans(True) FAIL
testSetupNetworksAddDelBondedNetwork(True) FAIL
testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL
testSetupNetworksAddDelDhcp(True, (4,)) FAIL
testSetupNetworksAddDelDhcp(True, (6,)) FAIL
testSetupNetworksAddManyVlans(True) OK
testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL
testSetupNetworksAddVlan(True) FAIL
testSetupNetworksConnectivityCheck OK
testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow
tests are disabled
testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow
tests are disabled
testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL
testSetupNetworksMtus(True) SKIP: This
test fails because the 2 different networks are getting configured with the
same MTU. The test should assert that the reported MTUs are equal to the
requested ones.
testSetupNetworksMultiMTUsOverBond(True) FAIL
testSetupNetworksMultiMTUsOverNic(True) FAIL
testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL
testSetupNetworksNicless OK
testSetupNetworksOverExistingBridge OK
testSetupNetworksReconfigureBridge FAIL
testStaticNetworkConfig((4, 6)) FAIL
testStaticNetworkConfig((4,)) FAIL
testStaticNetworkConfig((6,)) FAIL
testStaticSourceRouting(True) FAIL
test_drop_initial_bond_slaves_ip_config FAIL
test_ovirtmgmtm_to_ovs FAIL
test_remove_bond_under_network SKIP: OVS
does not support bridgeless networks
test_remove_initial_network_nic_ip_config OK
test_rollback FAIL
On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <pkliczew at redhat.com>
wrote:
> I like the idea but I have the same feelings as Francesco. I think that we
> need to make sure that functional tests for each vertical are stable before
> enabling this process.
>
> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <eedri at redhat.com> wrote:
>
>> adding also infra team for visibility on the change in CI.
>> also inline.
>>
>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <fromani at redhat.com>
>> wrote:
>>
>>> ----- Original Message -----
>>> > From: "Yaniv Bronheim" <ybronhei at redhat.com>
>>> > To: devel at ovirt.org, "Francesco Romani" <fromani at redhat.com>, "Nir
>>> Soffer" <nsoffer at redhat.com>, "Piotr Kliczewski"
>>> > <pkliczew at redhat.com>
>>> > Cc: "danken" <danken at redhat.com>, "David Caro" <dcaroest at redhat.com>,
>>> "Eyal Edri" <eedri at redhat.com>
>>> > Sent: Thursday, December 10, 2015 6:46:37 PM
>>> > Subject: Automation CI for vdsm
>>>
>>> [...]
>>> > We want to allow developers to trigger the script once reviews and
>>> > verification are ready (last step before merge). To do so we agreed to
>>> add
>>> > Continues Integration flag for each vdsm patch.
>>
>>
>> This flag will be called 'Workflow' or we can name it otherwise, we just
>> need to choose what makes sense.
>> David/Yaniv - Please correct me if I'm wrong.
>>
>>
>>> Once this flag will be
>>> > signed with +1 it will trigger Jenkins CI to run the check-merged
>>> script
>>> > (adding new button to gerrit is not an option - you can image that
>>> flag as
>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail
>>> > we'll get -1 and once new patchset is ready the developer will remove
>>> the
>>> > +1 and add it back to the Continues Integration flag to re-trigger the
>>> job.
>>> >
>>> > Please ack the process before we move on with that
>>>
>>> Sounds good, even though I'm a little scared (just gut feeling, no
>>> evidence
>>> whatsoever) that this could add even more complexity and fragility to
>>> the jenkins
>>> fleet.
>>>
>>> In the long run, when this is reliable, it will help greatly.
>>> In the short term, I'm scared because this can lead to false positives
>>> and bogus
>>> failures.
>>>
>>> Let me stress I don't have concrete item to share or specific flaws.
>>>
>>> As action item on me, I will find some time next week to check virt
>>> functional tests,
>>> to see if they need some fixes, work reliably and so forth
>>>
>>> > The patch for those scripts still under review and testing -
>>> > https://gerrit.ovirt.org/#/c/48268
>>>
>>> Will review asap.
>>>
>>> --
>>> Francesco Romani
>>> RedHat Engineering Virtualization R & D
>>> Phone: 8261328
>>> IRC: fromani
>>>
>>
>>
>>
>> --
>> Eyal Edri
>> Supervisor, RHEV CI
>> EMEA ENG Virtualization R&D
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>
>
--
*Yaniv Bronhaim.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20151216/bcc050d9/attachment.html>
More information about the Infra
mailing list