[JIRA] (OVIRT-1712) Re: Manual OST fails
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-1712:
-------------------------------------
I've created the [issue #27|https://github.com/lago-project/lago-ost-plugin/issues/27] on [lago-ost-plugin|https://github.com/lago-project/lago-ost-plugin] to track improving the '{{lago ovirt serve}}' mechanism.
> Re: Manual OST fails
> --------------------
>
> Key: OVIRT-1712
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: infra
>
> Evgheni,
> Was there any change recently to Lago slaves?
> On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski <
> piotr.kliczewski(a)gmail.com> wrote:
> > I attempted to run manual OST twice and both failed with below issue.
> > Can someone take a look?
> >
> > Thanks,
> > Piotr
> >
> > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix:
> > :DEBUG::
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> >
> > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error
> > occured, aborting
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in
> > do_run
> > self.cli_plugins[args.ovirtverb].do_run(args)
> > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
> > 184, in do_run
> > self._do_run(**vars(args))
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> > wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> > wrapper
> > return func(*args, prefix=prefix, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,
> > in do_deploy
> > prefix.deploy()
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> > error: [Errno 98] Address already in use
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100066)
7 years
[JIRA] (OVIRT-1712) Re: Manual OST fails
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-1712:
-------------------------------------
{quote}
They also had leftovers of ovirt-master_change-queue-tester in the Jenkins work directory, so this may be the job causing the issue.
{quote}
No, that last job to run on a slave always leaves its $WORKSPACE behind on that slave so if it runs on it again, some stuff is already cached for it.
We need to check the OST cleanup code and the jobs that previously ran on the slaves to see why the '{{lago serve}} process on the port was not killed. We should probably also modify how '{{lago serve}}' works so its less likely to influence other Lago environments trying to run on the same node, and less likely to stay behind.
> Re: Manual OST fails
> --------------------
>
> Key: OVIRT-1712
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: infra
>
> Evgheni,
> Was there any change recently to Lago slaves?
> On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski <
> piotr.kliczewski(a)gmail.com> wrote:
> > I attempted to run manual OST twice and both failed with below issue.
> > Can someone take a look?
> >
> > Thanks,
> > Piotr
> >
> > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix:
> > :DEBUG::
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> >
> > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error
> > occured, aborting
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in
> > do_run
> > self.cli_plugins[args.ovirtverb].do_run(args)
> > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
> > 184, in do_run
> > self._do_run(**vars(args))
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> > wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> > wrapper
> > return func(*args, prefix=prefix, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,
> > in do_deploy
> > prefix.deploy()
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> > error: [Errno 98] Address already in use
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100066)
7 years
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.1 - Build # 67 - Failure!
by jenkins＠jenkins.phx.ovirt.org
------=_Part_129_2057358532.1508555780829
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.1=
/=20
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.1/6=
7/
Build Number: 67
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #67
[Eyal Edri] Initial 4.2 suite ( using mostly 4.1 code )
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
status: 404
reason: Not Found
detail:=20
<html><head><title>Error</title></head><body>404 - Not Found</body></html>
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_u=
p at 0x4093ed8>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i=
n assert_equals_within
res =3D func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.1/ovirt=
-system-tests/hc-basic-suite-4.1/test-scenarios/002_bootstrap.py", line 144=
, in _host_is_up
cur_state =3D api.hosts.get(host.name()).status.state
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py=
", line 18338, in get
headers=3D{"All-Content":all_content}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",=
line 46, in get
return self.request(method=3D'GET', url=3Durl, headers=3Dheaders, cls=
=3Dcls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",=
line 122, in request
persistent_auth=3Dself.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connection=
spool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connection=
spool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body=
)
RequestError:=20
status: 404
reason: Not Found
detail:=20
<html><head><title>Error</title></head><body>404 - Not Found</body></html>
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTes=
t
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, i=
n wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in=
wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.1/ovirt=
-system-tests/hc-basic-suite-4.1/test-scenarios/002_bootstrap.py", line 163=
, in add_hosts
testlib.assert_true_within(_host_is_up, timeout=3D15 * 60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, i=
n assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i=
n assert_equals_within
res =3D func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.1/ovirt=
-system-tests/hc-basic-suite-4.1/test-scenarios/002_bootstrap.py", line 144=
, in _host_is_up
cur_state =3D api.hosts.get(host.name()).status.state
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py=
", line 18338, in get
headers=3D{"All-Content":all_content}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",=
line 46, in get
return self.request(method=3D'GET', url=3Durl, headers=3Dheaders, cls=
=3Dcls)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",=
line 122, in request
persistent_auth=3Dself.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connection=
spool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connection=
spool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body=
)
'\r\nstatus: 404\r\nreason: Not Found\r\ndetail: \n<html><head><title>Error=
</title></head><body>404 - Not Found</body></html>\n-------------------- >>=
begin captured logging << --------------------\novirtlago.testlib: ERROR: =
* Unhandled exception in <function _host_is_up at 0x4093ed8>\nTraceback=
(most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtla=
go/testlib.py", line 219, in assert_equals_within\n res =3D func()\n Fi=
le "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.1/ovirt-sys=
tem-tests/hc-basic-suite-4.1/test-scenarios/002_bootstrap.py", line 144, in=
_host_is_up\n cur_state =3D api.hosts.get(host.name()).status.state\n =
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py",=
line 18338, in get\n headers=3D{"All-Content":all_content}\n File "/us=
r/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 46, i=
n get\n return self.request(method=3D\'GET\', url=3Durl, headers=3Dheade=
rs, cls=3Dcls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastru=
cture/proxy.py", line 122, in request\n persistent_auth=3Dself.__persist=
ent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/=
connectionspool.py", line 79, in do_request\n persistent_auth)\n File "=
/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py=
", line 162, in __do_request\n raise errors.RequestError(response_code, =
response_reason, response_body)\nRequestError: \r\nstatus: 404\r\nreason: N=
ot Found\r\ndetail: \n<html><head><title>Error</title></head><body>404 - No=
t Found</body></html>\n--------------------- >> end captured logging << ---=
------------------'
------=_Part_129_2057358532.1508555780829--
7 years
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #909
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 50b3e94db35ecaa83eed2b32ea1629ced82b7f0e (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 50b3e94db35ecaa83eed2b32ea1629ced82b7f0e
Commit message: "python-windows: add upcoming fedoras"
> git rev-list 50b3e94db35ecaa83eed2b32ea1629ced82b7f0e # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins230390054820086134.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML
self._revertOldRepoXML()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
Build step 'Execute shell' marked build as failure
7 years
Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #834
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/834/disp...>
Changes:
[Sandro Bonazzola] python-windows: add upcoming fedoras
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 50b3e94db35ecaa83eed2b32ea1629ced82b7f0e (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 50b3e94db35ecaa83eed2b32ea1629ced82b7f0e
Commit message: "python-windows: add upcoming fedoras"
> git rev-list 4b1ee562e0bd553c241954b36e00edef4b90c51a # timeout=10
[system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins5864837049272072669.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
if self._latestRepoXML(local):
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1434, in _latestRepoXML
oxml = self._saveOldRepoXML(local)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1292, in _saveOldRepoXML
shutil.copy2(local, old_local)
File "/usr/lib64/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-base-fc25/repomd.xml.old.tmp'
Build step 'Execute shell' marked build as failure
7 years, 1 month
Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #832
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/832/disp...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4b1ee562e0bd553c241954b36e00edef4b90c51a (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 4b1ee562e0bd553c241954b36e00edef4b90c51a
Commit message: "ovirt-host: add 4.1 jobs"
> git rev-list 4b1ee562e0bd553c241954b36e00edef4b90c51a # timeout=10
[system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins268482957124770089.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML
self._revertOldRepoXML()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
Build step 'Execute shell' marked build as failure
7 years, 1 month
oVirt infra daily report - unstable production jobs - 480
by jenkins＠jenkins.phx.ovirt.org
------=_Part_124_1965220657.1508540401853
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/480//artifact/exported...
Cheers,
Jenkins
------=_Part_124_1965220657.1508540401853
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 20/10/2017
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/">system-sync_mirrors-epel-el6-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_124_1965220657.1508540401853--
7 years, 1 month
[JIRA] (OVIRT-1712) Re: Manual OST fails
by Evgheni Dereveanchin (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1508515087-31944-155
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]
Evgheni Dereveanchin edited comment on OVIRT-1712 at 10/20/17 3:58 PM:
-----------------------------------------------------------------------
I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished. I went through all of the bare metals and a few of them had port 8585 still occupied. They also had leftovers of ovirt-master_change-queue-tester in the Jenkins work directory, so this may be the job causing the issue.
was (Author: ederevea):
I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished.. It may be some other job leaving this web server behind but it's definitely affected several bare metals so far. I'll try to log in to the idle ones and kill the python process if present but the root cause is yet to be identified.
> Re: Manual OST fails
> --------------------
>
> Key: OVIRT-1712
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: infra
>
> Evgheni,
> Was there any change recently to Lago slaves?
> On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski <
> piotr.kliczewski(a)gmail.com> wrote:
> > I attempted to run manual OST twice and both failed with below issue.
> > Can someone take a look?
> >
> > Thanks,
> > Piotr
> >
> > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix:
> > :DEBUG::
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> >
> > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error
> > occured, aborting
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in
> > do_run
> > self.cli_plugins[args.ovirtverb].do_run(args)
> > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
> > 184, in do_run
> > self._do_run(**vars(args))
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> > wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> > wrapper
> > return func(*args, prefix=prefix, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,
> > in do_deploy
> > prefix.deploy()
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> > error: [Errno 98] Address already in use
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100066)
------------=_1508515087-31944-155
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]</pre>
<h3>Evgheni Dereveanchin edited comment on OVIRT-1712 at 10/20/17 3:58 PM:</h3>
<p>I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished. I went through all of the bare metals and a few of them had port 8585 still occupied. They also had leftovers of ovirt-master_change-queue-tester in the Jenkins work directory, so this may be the job causing the issue.</p>
<p>was (Author: ederevea): I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished.. It may be some other job leaving this web server behind but it's definitely affected several bare metals so far. I'll try to log in to the idle ones and kill the python process if present but the root cause is yet to be identified.</p>
<blockquote><h3>Re: Manual OST fails</h3>
<pre> Key: OVIRT-1712
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: eyal edri
Assignee: infra</pre>
<p>Evgheni, Was there any change recently to Lago slaves? On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski < piotr.kliczewski(a)gmail.com> wrote:</p>
<blockquote><p>I attempted to run manual OST twice and both failed with below issue. Can someone take a look?</p>
<p>Thanks, Piotr</p>
<p>2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix: :DEBUG::</p>
<pre>File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,</pre>
<p>in wrapper</p>
<pre> return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line</pre>
<p>111, in wrapper</p>
<pre> with utils.repo_server_context(args[0]):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line</pre>
<p>100, in repo_server_context</p>
<pre> root_dir=prefix.paths.internal_repo(),
File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,</pre>
<p>in _create_http_server</p>
<pre> generate_request_handler(root_dir),
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
self.server_bind()
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
SocketServer.TCPServer.server_bind(self)
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
self.socket.bind(self.server_address)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)</pre>
<p>2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error occured, aborting Traceback (most recent call last):</p>
<pre>File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in</pre>
<p>do_run</p>
<pre> self.cli_plugins[args.ovirtverb].do_run(args)
File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line</pre>
<p>184, in do_run</p>
<pre> self._do_run(**vars(args))
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in</pre>
<p>wrapper</p>
<pre> return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in</pre>
<p>wrapper</p>
<pre> return func(*args, prefix=prefix, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,</pre>
<p>in do_deploy</p>
<pre> prefix.deploy()
File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,</pre>
<p>in wrapper</p>
<pre> return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line</pre>
<p>111, in wrapper</p>
<pre> with utils.repo_server_context(args[0]):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line</pre>
<p>100, in repo_server_context</p>
<pre> root_dir=prefix.paths.internal_repo(),
File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,</pre>
<p>in _create_http_server</p>
<pre> generate_request_handler(root_dir),
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
self.server_bind()
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
SocketServer.TCPServer.server_bind(self)
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
self.socket.bind(self.server_address)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)</pre>
<p>error: [Errno 98] Address already in use <em>____________________________________________</em>_ Infra mailing list Infra(a)ovirt.org <a href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a></p></blockquote>
<p>— Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <<a href="https://www.redhat.com/">https://www.redhat.com/</a>> <<a href="https://red.ht/sig">https://red.ht/sig</a>> TRIED. TESTED. TRUSTED. <<a href="https://redhat.com/trusted">https://redhat.com/trusted</a>> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)</p></blockquote>
<p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100066)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYL..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1508515087-31944-155--
7 years, 1 month
