[OST] Network suites fails, 6+ days runtime?
I have a trivial patch: https://gerrit.ovirt.org/c/112068 This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results! Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable. Can someone look at the network suite failure? Should we remove it from the test matrix? Nir
Hello Ehud, can you help us here? Thanks On Mon, Nov 9, 2020 at 2:43 PM Nir Soffer <nsoffer@redhat.com> wrote:
I have a trivial patch: https://gerrit.ovirt.org/c/112068
This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...
The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results!
Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable.
Can someone look at the network suite failure? Should we remove it from the test matrix?
Nir
On Mon, Nov 9, 2020 at 4:15 PM Dominik Holler <dholler@redhat.com> wrote:
Hello Ehud, can you help us here? Thanks
On Mon, Nov 9, 2020 at 2:43 PM Nir Soffer <nsoffer@redhat.com> wrote:
I have a trivial patch: https://gerrit.ovirt.org/c/112068
This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...
The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results!
Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable.
Can someone look at the network suite failure? Should we remove it from the test matrix?
I do not think it's a matter of resources, but perhaps some other infra issue causing VMs to be hung or inaccessible or something like this. See also e.g. [1][2] and the thread (about [2]): [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1812 - Failure! [1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1812/ Ehud (or anyone): Can we (also) have relevant host-side logs collected per run, so that it's easier to try and debug such issues? Thanks and best regards,
Nir
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4IVGZDJMF5APGE...
-- Didi
On Mon, Nov 9, 2020 at 4:53 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Nov 9, 2020 at 4:15 PM Dominik Holler <dholler@redhat.com> wrote:
Hello Ehud, can you help us here? Thanks
On Mon, Nov 9, 2020 at 2:43 PM Nir Soffer <nsoffer@redhat.com> wrote:
I have a trivial patch: https://gerrit.ovirt.org/c/112068
This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...
The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results!
Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable.
Can someone look at the network suite failure? Should we remove it from the test matrix?
I do not think it's a matter of resources, but perhaps some other infra issue causing VMs to be hung or inaccessible or something like this. See also e.g. [1][2] and the thread (about [2]):
[oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1812 - Failure!
I cannot find this thread, can you link to it?
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/...
[2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1812/
Ehud (or anyone): Can we (also) have relevant host-side logs collected per run, so that it's easier to try and debug such issues?
Thanks and best regards,
Nir
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4IVGZDJMF5APGE...
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RS7KC7LII2TL4S...
On Mon, Nov 9, 2020 at 5:22 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Nov 9, 2020 at 4:53 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Nov 9, 2020 at 4:15 PM Dominik Holler <dholler@redhat.com> wrote:
Hello Ehud, can you help us here? Thanks
On Mon, Nov 9, 2020 at 2:43 PM Nir Soffer <nsoffer@redhat.com> wrote:
I have a trivial patch: https://gerrit.ovirt.org/c/112068
This should not have any effect on the network tests, but I ran the tests twice, and the network suite fails in both runs: - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...
The first run waited in the queue, 1 day 12 hours then ran for 11 hours. The second run waited in the queue for 2 days 21 hours and then ran for 19 hours. This is a total of 135 hours waiting for test results!
Looks like we don't have the resources to test OST patches. The current way testing 8 suites for every patch is not workable.
Can someone look at the network suite failure? Should we remove it from the test matrix?
I do not think it's a matter of resources, but perhaps some other infra issue causing VMs to be hung or inaccessible or something like this. See also e.g. [1][2] and the thread (about [2]):
[oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1812 - Failure!
I cannot find this thread, can you link to it?
Sorry, it was a private thread :-(. I meant: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 561 - Failure!
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/...
[2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1812/
Ehud (or anyone): Can we (also) have relevant host-side logs collected per run, so that it's easier to try and debug such issues?
Thanks and best regards,
Nir
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4IVGZDJMF5APGE...
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RS7KC7LII2TL4S...
-- Didi
participants (3)
-
Dominik Holler -
Nir Soffer -
Yedidyah Bar David