<div dir="ltr">exactly - just posted a patch again that sings all fails as broken . <div>we'll get the report soon and I'll publish it as well. hope the run will take less time now</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 16, 2015 at 6:19 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Nice, but we cannot enable this until all the tests pass or disabled.<br>
<br>
There is no point in broken or flaky functional tests.<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Dec 16, 2015 at 5:34 PM, Yaniv Bronheim <<a href="mailto:ybronhei@redhat.com">ybronhei@redhat.com</a>> wrote:<br>
> So its not stable. It won't block merges and at least give us report after<br>
> each merge. It takes really long time to run it (because of the tests<br>
> themselves. Lago things takes maximum 15minutes, but the run last for more<br>
> than 2hrs right now and I suspect functional/storageTests.py gets stuck)<br>
><br>
> Bellow you can see where we stand (before I added python-rtslib package).<br>
><br>
> Now, I still want to merge the patch <a href="https://gerrit.ovirt.org/#/c/48268/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/48268/</a> -<br>
> which enables this run after merges, and I still want you to consider the<br>
> addition of Automation CI flag to our gerrit so that developer will be able<br>
> to use it as a trigger for the check-merged.sh script run, just to see if<br>
> their patch fixes\brakes something realted to the functional tests<br>
><br>
> <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/1480/</a> - is<br>
> an example of how the run looks like. I still work to improve the output<br>
><br>
><br>
> Please reply and let me know if the idea around the automation flag is<br>
> acceptable by you.. and please review the patch for comments and acks.<br>
> We can ask dcaro to add the flag until Friday, otherwise we'll need to delay<br>
> this effort after the holiday..<br>
><br>
><br>
> functional.sosPluginTests.SosPluginTest<br>
> testSosPlugin OK<br>
> functional.vmRecoveryTests.RecoveryTests<br>
> test_vm_recovery FAIL<br>
> functional.vmQoSTests.VMQosTests<br>
> testSmallVMBallooning FAIL<br>
> functional.virtTests.VirtTest<br>
> testComplexVm FAIL<br>
> testHeadlessVm OK<br>
> testSimpleVm FAIL<br>
> testVmDefinitionGraphics('spice') FAIL<br>
> testVmDefinitionGraphics('vnc') OK<br>
> testVmDefinitionLegacyGraphics('qxl') FAIL<br>
> testVmDefinitionLegacyGraphics('vnc') OK<br>
> testVmDefinitionMultipleGraphics('spice', 'vnc') FAIL<br>
> testVmDefinitionMultipleGraphics('vnc', 'spice') FAIL<br>
> testVmWithCdrom('self') FAIL<br>
> testVmWithCdrom('specParams') FAIL<br>
> testVmWithCdrom('vmPayload') FAIL<br>
> testVmWithDevice('hotplugDisk') FAIL<br>
> testVmWithDevice('hotplugNic') FAIL<br>
> testVmWithDevice('smartcard') FAIL<br>
> testVmWithDevice('virtioNic') FAIL<br>
> testVmWithDevice('virtioRng') FAIL<br>
> testVmWithSla FAIL<br>
> testVmWithStorage('iscsi') SKIP:<br>
> python-rtslib is not installed.<br>
> testVmWithStorage('localfs') FAIL<br>
> testVmWithStorage('nfs') FAIL<br>
> functional.storageTests.StorageTest<br>
> testCreatePoolErrors OK<br>
> testStorage('glusterfs', 0) ERROR<br>
> testStorage('glusterfs', 3) ERROR<br>
> testStorage('iscsi', 0) SKIP:<br>
> python-rtslib is not installed.<br>
> testStorage('iscsi', 3) SKIP:<br>
> python-rtslib is not installed.<br>
> testStorage('localfs', 0) FAIL<br>
> testStorage('localfs', 3) FAIL<br>
> testStorage('nfs', 0) FAIL<br>
> testStorage('nfs', 3) FAIL<br>
> functional.networkTests.NetworkTest<br>
> testAddVlanedBridgeless ERROR<br>
> testAddVlanedBridgeless_oneCommand ERROR<br>
> testAfterNetworkSetupHook ERROR<br>
> testBeforeNetworkSetupHook ERROR<br>
> testBondHwAddress(False) ERROR<br>
> testBondHwAddress(True) ERROR<br>
> testBrokenNetworkReplacement(False) ERROR<br>
> testBrokenNetworkReplacement(True) ERROR<br>
> testDelNetworkBondAccumulation ERROR<br>
> testDelNetworkWithMTU(False) ERROR<br>
> testDelNetworkWithMTU(True) ERROR<br>
> testDelWithoutAdd ERROR<br>
> testDhclientLeases(4, 'default') ERROR<br>
> testDhclientLeases(4, 'local') ERROR<br>
> testDhclientLeases(6, None) ERROR<br>
> testDhcpReplaceNicWithBridge ERROR<br>
> testFailWithInvalidBondingName(False) ERROR<br>
> testFailWithInvalidBondingName(True) ERROR<br>
> testFailWithInvalidBridgeName ERROR<br>
> testFailWithInvalidIpConfig ERROR<br>
> testFailWithInvalidNic(False) ERROR<br>
> testFailWithInvalidNic(True) ERROR<br>
> testFailWithInvalidParams(False) ERROR<br>
> testFailWithInvalidParams(True) ERROR<br>
> testGetRouteDeviceTo ERROR<br>
> testHonorBlockingDhcp SKIP: Slow<br>
> tests are disabled<br>
> testIpLinkWrapper ERROR<br>
> testLowerMtuDoesNotOverride FAIL<br>
> testNoBridgeLeftovers FAIL<br>
> testReconfigureBrNetWithVanishedPort ERROR<br>
> testRedefineBondedNetworkIPs ERROR<br>
> testRemovingBridgeDoesNotLeaveBridge ERROR<br>
> testReorderBondingOptions(False) ERROR<br>
> testReorderBondingOptions(True) ERROR<br>
> testRestoreNetworksOnlyRestoreUnchangedDevices ERROR<br>
> testRestoreToBlockingDHCP ERROR<br>
> testSafeNetworkConfig(False) ERROR<br>
> testSafeNetworkConfig(True) ERROR<br>
> testSelectiveRestoreDuringUpgrade ERROR<br>
> testSelectiveRestoreIgnoresVdsmRegParams FAIL<br>
> testSetupNetworkOutboundQos(False) ERROR<br>
> testSetupNetworkOutboundQos(True) FAIL<br>
> testSetupNetworksActiveSlave ERROR<br>
> testSetupNetworksAddBadParams(False) ERROR<br>
> testSetupNetworksAddBadParams(True) FAIL<br>
> testSetupNetworksAddBondWithManyVlans(False) ERROR<br>
> testSetupNetworksAddBondWithManyVlans(True) FAIL<br>
> testSetupNetworksAddDelBondedNetwork(False) ERROR<br>
> testSetupNetworksAddDelBondedNetwork(True) FAIL<br>
> testSetupNetworksAddDelDhcp(False, (4, 6)) ERROR<br>
> testSetupNetworksAddDelDhcp(False, (4,)) ERROR<br>
> testSetupNetworksAddDelDhcp(False, (6,)) ERROR<br>
> testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL<br>
> testSetupNetworksAddDelDhcp(True, (4,)) FAIL<br>
> testSetupNetworksAddDelDhcp(True, (6,)) FAIL<br>
> testSetupNetworksAddManyVlans(False) ERROR<br>
> testSetupNetworksAddManyVlans(True) FAIL<br>
> testSetupNetworksAddNetworkToNicAfterBondBreaking(False) ERROR<br>
> testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL<br>
> testSetupNetworksAddNetworkToNicAfterBondResizing(False) ERROR<br>
> testSetupNetworksAddNetworkToNicAfterBondResizing(True) ERROR<br>
> testSetupNetworksAddOverExistingBond(False) ERROR<br>
> testSetupNetworksAddOverExistingBond(True) ERROR<br>
> testSetupNetworksAddVlan(False) ERROR<br>
> testSetupNetworksAddVlan(True) FAIL<br>
> testSetupNetworksConnectivityCheck FAIL<br>
> testSetupNetworksConvertVlanNetBridgeness ERROR<br>
> testSetupNetworksDelOneOfBondNets ERROR<br>
> testSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ERROR<br>
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(False)ERROR<br>
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow<br>
> tests are disabled<br>
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(False)ERROR<br>
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow<br>
> tests are disabled<br>
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(False) ERROR<br>
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL<br>
> testSetupNetworksMtus(False) ERROR<br>
> testSetupNetworksMtus(True) SKIP: This<br>
> test fails because the 2 different networks are getting configured with the<br>
> same MTU. The test should assert that the reported MTUs are equal to the<br>
> requested ones.<br>
> testSetupNetworksMultiMTUsOverBond(False) ERROR<br>
> testSetupNetworksMultiMTUsOverBond(True) FAIL<br>
> testSetupNetworksMultiMTUsOverNic(False) ERROR<br>
> testSetupNetworksMultiMTUsOverNic(True) FAIL<br>
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(False) ERROR<br>
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL<br>
> testSetupNetworksNicless FAIL<br>
> testSetupNetworksNiclessBridgeless ERROR<br>
> testSetupNetworksOverDhcpIface ERROR<br>
> testSetupNetworksOverExistingBridge FAIL<br>
> testSetupNetworksReconfigureBridge FAIL<br>
> testSetupNetworksRemoveBondWithKilledEnslavedNics ERROR<br>
> testSetupNetworksRemoveSlavelessBond ERROR<br>
> testSetupNetworksResizeBond(False) ERROR<br>
> testSetupNetworksResizeBond(True) ERROR<br>
> testSetupNetworksStableBond(False) ERROR<br>
> testSetupNetworksStableBond(True) ERROR<br>
> testStaticNetworkConfig((4, 6)) FAIL<br>
> testStaticNetworkConfig((4,)) FAIL<br>
> testStaticNetworkConfig((6,)) FAIL<br>
> testStaticSourceRouting(False) ERROR<br>
> testStaticSourceRouting(True) FAIL<br>
> testTwiceAdd(False) ERROR<br>
> testTwiceAdd(True) ERROR<br>
> testVolatileConfig(False) ERROR<br>
> testVolatileConfig(True) ERROR<br>
> test_drop_initial_bond_slaves_ip_config FAIL<br>
> test_getVdsStats ERROR<br>
> test_remove_bond_under_network FAIL<br>
> test_remove_initial_network_nic_ip_config FAIL<br>
> test_rollback FAIL<br>
> test_setupNetworks_bond_with_custom_option ERROR<br>
> test_setupNetworks_on_external_bond ERROR<br>
> test_setupNetworks_on_external_vlaned_bond ERROR<br>
> functional.networkTestsOVS.OVSNetworkTest<br>
> testHonorBlockingDhcp SKIP: Slow<br>
> tests are disabled<br>
> testLowerMtuDoesNotOverride OK<br>
> testNoBridgeLeftovers FAIL<br>
> testSelectiveRestoreIgnoresVdsmRegParams FAIL<br>
> testSetupNetworkOutboundQos(True) OK<br>
> testSetupNetworksAddBadParams(True) OK<br>
> testSetupNetworksAddBondWithManyVlans(True) FAIL<br>
> testSetupNetworksAddDelBondedNetwork(True) FAIL<br>
> testSetupNetworksAddDelDhcp(True, (4, 6)) FAIL<br>
> testSetupNetworksAddDelDhcp(True, (4,)) FAIL<br>
> testSetupNetworksAddDelDhcp(True, (6,)) FAIL<br>
> testSetupNetworksAddManyVlans(True) OK<br>
> testSetupNetworksAddNetworkToNicAfterBondBreaking(True) FAIL<br>
> testSetupNetworksAddVlan(True) FAIL<br>
> testSetupNetworksConnectivityCheck OK<br>
> testSetupNetworksEmergencyDevicesCleanupBondOverwrite(True) SKIP: Slow<br>
> tests are disabled<br>
> testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(True) SKIP: Slow<br>
> tests are disabled<br>
> testSetupNetworksKeepNetworkOnBondAfterBondResizing(True) FAIL<br>
> testSetupNetworksMtus(True) SKIP: This<br>
> test fails because the 2 different networks are getting configured with the<br>
> same MTU. The test should assert that the reported MTUs are equal to the<br>
> requested ones.<br>
> testSetupNetworksMultiMTUsOverBond(True) FAIL<br>
> testSetupNetworksMultiMTUsOverNic(True) FAIL<br>
> testSetupNetworksNetCompatibilityMultipleNetsSameNic(True) FAIL<br>
> testSetupNetworksNicless OK<br>
> testSetupNetworksOverExistingBridge OK<br>
> testSetupNetworksReconfigureBridge FAIL<br>
> testStaticNetworkConfig((4, 6)) FAIL<br>
> testStaticNetworkConfig((4,)) FAIL<br>
> testStaticNetworkConfig((6,)) FAIL<br>
> testStaticSourceRouting(True) FAIL<br>
> test_drop_initial_bond_slaves_ip_config FAIL<br>
> test_ovirtmgmtm_to_ovs FAIL<br>
> test_remove_bond_under_network SKIP: OVS<br>
> does not support bridgeless networks<br>
> test_remove_initial_network_nic_ip_config OK<br>
> test_rollback FAIL<br>
><br>
><br>
> On Sun, Dec 13, 2015 at 10:07 PM, Piotr Kliczewski <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
> wrote:<br>
>><br>
>> I like the idea but I have the same feelings as Francesco. I think that we<br>
>> need to make sure that functional tests for each vertical are stable before<br>
>> enabling this process.<br>
>><br>
>> On Sun, Dec 13, 2015 at 8:34 AM, Eyal Edri <<a href="mailto:eedri@redhat.com">eedri@redhat.com</a>> wrote:<br>
>>><br>
>>> adding also infra team for visibility on the change in CI.<br>
>>> also inline.<br>
>>><br>
>>> On Fri, Dec 11, 2015 at 4:19 PM, Francesco Romani <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> ----- Original Message -----<br>
>>>> > From: "Yaniv Bronheim" <<a href="mailto:ybronhei@redhat.com">ybronhei@redhat.com</a>><br>
>>>> > To: <a href="mailto:devel@ovirt.org">devel@ovirt.org</a>, "Francesco Romani" <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>>, "Nir<br>
>>>> > Soffer" <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>>, "Piotr Kliczewski"<br>
>>>> > <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
>>>> > Cc: "danken" <<a href="mailto:danken@redhat.com">danken@redhat.com</a>>, "David Caro" <<a href="mailto:dcaroest@redhat.com">dcaroest@redhat.com</a>>,<br>
>>>> > "Eyal Edri" <<a href="mailto:eedri@redhat.com">eedri@redhat.com</a>><br>
>>>> > Sent: Thursday, December 10, 2015 6:46:37 PM<br>
>>>> > Subject: Automation CI for vdsm<br>
>>>><br>
>>>> [...]<br>
>>>> > We want to allow developers to trigger the script once reviews and<br>
>>>> > verification are ready (last step before merge). To do so we agreed to<br>
>>>> > add<br>
>>>> > Continues Integration flag for each vdsm patch.<br>
>>><br>
>>><br>
>>> This flag will be called 'Workflow' or we can name it otherwise, we just<br>
>>> need to choose what makes sense.<br>
>>> David/Yaniv - Please correct me if I'm wrong.<br>
>>><br>
>>>><br>
>>>> Once this flag will be<br>
>>>> > signed with +1 it will trigger Jenkins CI to run the check-merged<br>
>>>> > script<br>
>>>> > (adding new button to gerrit is not an option - you can image that<br>
>>>> > flag as<br>
>>>> > a trigger button), on success Jenkins CI flag will turn to +2. on fail<br>
>>>> > we'll get -1 and once new patchset is ready the developer will remove<br>
>>>> > the<br>
>>>> > +1 and add it back to the Continues Integration flag to re-trigger the<br>
>>>> > job.<br>
>>>> ><br>
>>>> > Please ack the process before we move on with that<br>
>>>><br>
>>>> Sounds good, even though I'm a little scared (just gut feeling, no<br>
>>>> evidence<br>
>>>> whatsoever) that this could add even more complexity and fragility to<br>
>>>> the jenkins<br>
>>>> fleet.<br>
>>>><br>
>>>> In the long run, when this is reliable, it will help greatly.<br>
>>>> In the short term, I'm scared because this can lead to false positives<br>
>>>> and bogus<br>
>>>> failures.<br>
>>>><br>
>>>> Let me stress I don't have concrete item to share or specific flaws.<br>
>>>><br>
>>>> As action item on me, I will find some time next week to check virt<br>
>>>> functional tests,<br>
>>>> to see if they need some fixes, work reliably and so forth<br>
>>>><br>
>>>> > The patch for those scripts still under review and testing -<br>
>>>> > <a href="https://gerrit.ovirt.org/#/c/48268" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/48268</a><br>
>>>><br>
>>>> Will review asap.<br>
>>>><br>
>>>> --<br>
>>>> Francesco Romani<br>
>>>> RedHat Engineering Virtualization R & D<br>
>>>> Phone: 8261328<br>
>>>> IRC: fromani<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Eyal Edri<br>
>>> Supervisor, RHEV CI<br>
>>> EMEA ENG Virtualization R&D<br>
>>> Red Hat Israel<br>
>>><br>
>>> phone: <a href="tel:%2B972-9-7692018" value="+97297692018">+972-9-7692018</a><br>
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br>
>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Yaniv Bronhaim.<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px"><b>Yaniv Bronhaim.</b></span><br></div></div></div></div></div>
</div>