[JIRA] (OVIRT-2252) The s390x slave is used in parallel by both the
staging and the production CI systems
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2252?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-2252:
--------------------------------
Component/s: (was: CI Mirrors)
Jenkins Slaves
> The s390x slave is used in parallel by both the staging and the production CI systems
> -------------------------------------------------------------------------------------
>
> Key: OVIRT-2252
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2252
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: Jenkins Slaves
> Reporter: Barak Korren
> Assignee: infra
>
> Since we only have one s390x slave, it is currently attached to both the staging and the production CI systems, and while they use separate user accounts, it turns out this is not enough to isolate them from one another.
> The are several issues that are caused by this configuration:
> # Tests that allocate a fixed network port can fail if they are run by both systems at the same time - this happens in practice when sending Python patchs to the '{{jenkins}}' repo because the {{mirror_client.py}} tests start a web server on port 8675.
> # The {{mock_cleanup.sh}} script that is being run by one system can time out trying to umount things from a mock environment that was created and is being used by the other system.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100087)
6 years, 6 months
[JIRA] (OVIRT-2252) The s390x slave is used in parallel by both the
staging and the production CI systems
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2252?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-2252:
--------------------------------
Description: "Since we only have one s390x slave, it is currently attached to both the staging and the production CI systems, and while they use separate user accounts, it turns out this is not enough to isolate them from one another.\r\n\r\nThe are several issues that are caused by this configuration:\r\n# Tests that allocate a fixed network port can fail if they are run by both systems at the same time - this happens in practice when sending Python patchs to the '{{jenkins}}' repo because the {{mirror_client.py}} tests start a web server on port 8675.\r\n# The {{mock_cleanup.sh}} script that is being run by one system can time out trying to umount things from a mock environment that was created and is being used by the other system.\r\n\r\n\r\n\r\n" (was: One of the tests of the CI mirrors python package is starting up an HTTP server on a fixed port number.
This can cause an issue if the same test is running in parallel on the same machine. This typically cannot be an issue on the CI infra because it never runs more then one STDCI thread at a time on a given slave.
This issue does arise specifically on the s390x slave because there the same slave is used both by the staging and the production CI systems. In that case the two systems might be trying to run the same test at the same time, so this can lead to port allocation exceptions.
)
> The s390x slave is used in parallel by both the staging and the production CI systems
> -------------------------------------------------------------------------------------
>
> Key: OVIRT-2252
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2252
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: CI Mirrors
> Reporter: Barak Korren
> Assignee: infra
>
> Since we only have one s390x slave, it is currently attached to both the staging and the production CI systems, and while they use separate user accounts, it turns out this is not enough to isolate them from one another.
> The are several issues that are caused by this configuration:
> # Tests that allocate a fixed network port can fail if they are run by both systems at the same time - this happens in practice when sending Python patchs to the '{{jenkins}}' repo because the {{mirror_client.py}} tests start a web server on port 8675.
> # The {{mock_cleanup.sh}} script that is being run by one system can time out trying to umount things from a mock environment that was created and is being used by the other system.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100087)
6 years, 6 months
[JIRA] (OVIRT-2252) The s390x slave is used in parallel by both the
staging and the production CI systems
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2252?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-2252:
--------------------------------
Summary: The s390x slave is used in parallel by both the staging and the production CI systems (was: Rece condition in python tests in jenkins repo on s390x)
> The s390x slave is used in parallel by both the staging and the production CI systems
> -------------------------------------------------------------------------------------
>
> Key: OVIRT-2252
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2252
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: CI Mirrors
> Reporter: Barak Korren
> Assignee: infra
>
> One of the tests of the CI mirrors python package is starting up an HTTP server on a fixed port number.
> This can cause an issue if the same test is running in parallel on the same machine. This typically cannot be an issue on the CI infra because it never runs more then one STDCI thread at a time on a given slave.
> This issue does arise specifically on the s390x slave because there the same slave is used both by the staging and the production CI systems. In that case the two systems might be trying to run the same test at the same time, so this can lead to port allocation exceptions.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100087)
6 years, 6 months
[JIRA] (OVIRT-2252) Rece condition in python tests in jenkins repo
on s390x
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2252?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-2252:
--------------------------------
Description: "One of the tests of the CI mirrors python package is starting up an HTTP server on a fixed port number.\r\n\r\nThis can cause an issue if the same test is running in parallel on the same machine. This typically cannot be an issue on the CI infra because it never runs more then one STDCI thread at a time on a given slave.\r\n\r\nThis issue does arise specifically on the s390x slave because there the same slave is used both by the staging and the production CI systems. In that case the two systems might be trying to run the same test at the same time, so this can lead to port allocation exceptions.\r\n\r\n\r\n" (was: One of the tests of the CI mirrors python package is starting up an HTTP server on a fixed port number.
This can cause an issue if the same test is running in parallel on the same machine. This typically cannot be an issue on the CI infra because it never runs more then one STDCI thread at a time on a given slave.
This issue does arise specifically on the s390x slave because there the same slave is used both by the staging and the production CI systems. In that case the two systems might be trying to run the same test at the same time, so this can lead to port allocation exceptions.)
> Rece condition in python tests in jenkins repo on s390x
> -------------------------------------------------------
>
> Key: OVIRT-2252
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2252
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: CI Mirrors
> Reporter: Barak Korren
> Assignee: infra
>
> One of the tests of the CI mirrors python package is starting up an HTTP server on a fixed port number.
> This can cause an issue if the same test is running in parallel on the same machine. This typically cannot be an issue on the CI infra because it never runs more then one STDCI thread at a time on a given slave.
> This issue does arise specifically on the s390x slave because there the same slave is used both by the staging and the production CI systems. In that case the two systems might be trying to run the same test at the same time, so this can lead to port allocation exceptions.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100087)
6 years, 6 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 292
- Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/292/
Build Number: 292
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #292
[Sandro Bonazzola] yum-repos: master: adjusted for new packages
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 004_basic_sanity.snapshot_merge
Error Message:
status: 409
reason: Conflict
detail: Cannot remove Snapshot. Snapshot is currently being created for VM vm0.
Stack Trace:
Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 200, in snapshot_merge
api.vms.get(VM0_NAME).snapshots.list()[-2].delete()
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34001, in delete
headers={"Correlation-Id":correlation_id,"Content-type":None}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 57, in delete
return self.request('DELETE', url, body, headers, cls=cls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
persistent_auth=self.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 409
reason: Conflict
detail: Cannot remove Snapshot. Snapshot is currently being created for VM vm0.
6 years, 6 months
