Jenkins build became unstable: ovirt_4.0_he-system-tests #288
Is this the usual Sanlock issue? Error Message status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up. Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up. On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Mon, Sep 19, 2016 at 7:49 PM, Eyal Edri <eedri@redhat.com> wrote:
Is this the usual Sanlock issue?
Not that sure, however build 289 come back to stable by itself.
Error Message
status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
I really want to add this job to the experimental flow to catch regressions early, but we have to iron out any false positives or errors we're seeing before that. Until now I was only aware of the Sanlock issue, which should be resolved soon once its available in centos. Are we aware of other known issues in the tests? On Tue, Sep 20, 2016 at 11:22 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Sep 19, 2016 at 7:49 PM, Eyal Edri <eedri@redhat.com> wrote:
Is this the usual Sanlock issue?
Not that sure, however build 289 come back to stable by itself.
Error Message
status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi Eyal, I'm not 100% sure about that one. Checking the logs I see that the first host has been setup without any issues, yet setting up the second host gives: 17:16:31 [ ERROR ] Failed to execute stage 'Closing up': HTTP Error 500: Internal Server Error 17:16:31 [ INFO ] Stage: Clean up 17:16:31 [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160919131631.conf' 17:16:31 [ INFO ] Stage: Pre-termination 17:16:31 [ INFO ] Stage: Termination 17:16:31 [ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy 17:16:31 Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160919131556-qufq5z.log However as we don't have the ovirt-engine and ovirt-hosted-engine-setup logs we can't be 100% certain. Thanks in advance, Lev Veyde. ----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Lev Veyde" <lveyde@redhat.com> Cc: "Sandro Bonazzola" <sbonazzo@redhat.com>, "infra" <infra@ovirt.org>, "Simone Tiraboschi" <stirabos@redhat.com> Sent: Monday, September 19, 2016 8:49:59 PM Subject: Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288 Is this the usual Sanlock issue? Error Message status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up. Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up. On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Sep 20, 2016 at 10:44 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
I'm not 100% sure about that one. Checking the logs I see that the first host has been setup without any issues, yet setting up the second host gives:
17:16:31 [ ERROR ] Failed to execute stage 'Closing up': HTTP Error 500: Internal Server Error 17:16:31 [ INFO ] Stage: Clean up 17:16:31 [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine- setup/answers/answers-20160919131631.conf' 17:16:31 [ INFO ] Stage: Pre-termination 17:16:31 [ INFO ] Stage: Termination 17:16:31 [ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy 17:16:31 Log file is located at /var/log/ovirt-hosted-engine- setup/ovirt-hosted-engine-setup-20160919131556-qufq5z.log
However as we don't have the ovirt-engine and ovirt-hosted-engine-setup logs we can't be 100% certain.
We have them, they are in http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/artifact/*zip*/ar... under /archive/exported-artifacts/test_logs/he_basic_suite_4.0/ post-002_bootstrap.py/
Thanks in advance, Lev Veyde.
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Lev Veyde" <lveyde@redhat.com> Cc: "Sandro Bonazzola" <sbonazzo@redhat.com>, "infra" <infra@ovirt.org>, "Simone Tiraboschi" <stirabos@redhat.com> Sent: Monday, September 19, 2016 8:49:59 PM Subject: Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288
Is this the usual Sanlock issue?
Error Message
status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/ connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/ connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Sep 20, 2016 at 10:44 AM, Lev Veyde <lveyde@redhat.com> wrote:
Hi Eyal,
I'm not 100% sure about that one. Checking the logs I see that the first host has been setup without any issues, yet setting up the second host gives:
17:16:31 [ ERROR ] Failed to execute stage 'Closing up': HTTP Error 500: Internal Server Error 17:16:31 [ INFO ] Stage: Clean up 17:16:31 [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine- setup/answers/answers-20160919131631.conf' 17:16:31 [ INFO ] Stage: Pre-termination 17:16:31 [ INFO ] Stage: Termination 17:16:31 [ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy 17:16:31 Log file is located at /var/log/ovirt-hosted-engine- setup/ovirt-hosted-engine-setup-20160919131556-qufq5z.log
Can we run sosreport / log collector and keep the logs on failre?
However as we don't have the ovirt-engine and ovirt-hosted-engine-setup logs we can't be 100% certain.
Thanks in advance, Lev Veyde.
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Lev Veyde" <lveyde@redhat.com> Cc: "Sandro Bonazzola" <sbonazzo@redhat.com>, "infra" <infra@ovirt.org>, "Simone Tiraboschi" <stirabos@redhat.com> Sent: Monday, September 19, 2016 8:49:59 PM Subject: Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288
Is this the usual Sanlock issue?
Error Message
status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/ infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/ connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/ connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com <https://www.redhat.com/it/about/events/red-hat-open-source-day-2016>
On Mon, Sep 19, 2016 at 8:49 PM, Eyal Edri <eedri@redhat.com> wrote:
Is this the usual Sanlock issue?
"Usual" sanlock issue is alarming. I don't know about any issues with Sanlock. Do we have a bug about this?
Error Message
status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 367, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py", line 78, in add_vm_blank api.vms.add(vm_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35701, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request detail: Cannot add VM: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up.
On Mon, Sep 19, 2016 at 8:22 PM, <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
participants (6)
-
Eyal Edri -
jenkins＠jenkins.phx.ovirt.org -
Lev Veyde -
Nir Soffer -
Sandro Bonazzola -
Simone Tiraboschi