[ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
4:06 a.m.
This is a multi-part message in MIME format.
--------------C2726ECDFD4C62F2178F2F20
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
*
Hi,
**
We have failed cq with ovirt-imageio failing to start on upgrade suite*.
*I can still only see errors**in the messages log.
**
I'm writing the reported patch but I don't think it has anything to do
with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster -
https://gerrit.ovirt.org/#/c/84814/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
**
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
(Relevant) error snippet from the log:
<error>
*From messages log
*Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Started oVirt Engine. Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28
12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked
executable. Please remove executable permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt Engine Data Warehouse... Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine Data Warehouse. Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28
12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked
executable. Please remove executable permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt ImageIO Proxy... Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last): Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line 21, in main Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line
45, in start Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov
28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line
430, in server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited,
status=1/FAILURE Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov
28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt ImageIO Proxy... Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last): Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line 21, in main Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line
45, in start Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov
28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line
430, in server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited,
status=1/FAILURE Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
*
--------------C2726ECDFD4C62F2178F2F20
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans:
2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing:
0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial;
text-decoration-color: initial; word-wrap: break-word; white-space:
pre-wrap;"><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Hi,
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p></b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">We
have failed cq with ovirt-imageio failing to start on upgrade suite<b>. </b>I
can still only see errors<b> </b>in the messages log.
</span></p></b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I'm
writing the reported patch but I don't think it has anything to do with this issue.
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
and headline of suspected patches: </span></p>restapi: Enable update to no
default network provider of cluster - <a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/84814/">https://gerrit.ov...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job:</span></p><a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs:</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p></b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
</b>From messages log
<b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66">
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable
permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
Engine Data Warehouse...
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine Data Warehouse.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable
permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p></b>
</pre>
</body>
</html>
--------------C2726ECDFD4C62F2178F2F20--
4:18 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
*Hi,
*
*We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only
see errors in the messages log.
*
*I'm writing the reported patch but I don't think it has anything to do with this
issue.
Link and headline of suspected patches: restapi: Enable update to no default network
provider of cluster - https://gerrit.ovirt.org/#/c/84814/
<https://gerrit.ovirt.org/#/c/84814/>
Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
*
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
(Relevant) error snippet from the log: <error>
*From messages log
*
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable
permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
Engine Data Warehouse...
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine Data Warehouse.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable
permission bits. Proceeding anyway.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>*
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
4:59 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
On Wed, Nov 29, 2017 at 12:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
Do we have more info from Didi's debug plugin now?
Not yet. Patch to install the debug plugins was merged this morning,
build below failed yesterday evening.
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
> *Hi,
>
> *
>
> *We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still
only see errors in the messages log.
> *
>
> *I'm writing the reported patch but I don't think it has anything to do with
this issue.
>
> Link and headline of suspected patches: restapi: Enable update to no default network
provider of cluster - https://gerrit.ovirt.org/#/c/84814/
<https://gerrit.ovirt.org/#/c/84814/>
>
> Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
>
> Link to all logs:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
> *
>
>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
> (Relevant) error snippet from the log: <error>
> *From messages log
>
> *
>
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration
file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove
executable permission bits. Proceeding anyway.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
Engine Data Warehouse...
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine Data Warehouse.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration
file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove
executable permission bits. Proceeding anyway.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> </error>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
--
Didi
5 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
This is a multi-part message in MIME format.
--------------98CB106BD8B4230B34117EA6
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
this is the plugin info from steup log but I don't see anything more
than we have seen except a timeout.
https://pastebin.com/QVtNRNWV
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
Didi, is there anywhere else I should look?
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com
<mailto:dron@redhat.com>> wrote:
*
Hi,
**
We have failed cq with ovirt-imageio failing to start on upgrade
suite*. *I can still only see errors**in the messages log.
**
I'm writing the reported patch but I don't think it has anything
to do with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster -
https://gerrit.ovirt.org/#/c/84814/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
**
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
(Relevant) error snippet from the log:
<error>
*From messages log
*Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine
systemd: Started oVirt Engine. Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine
systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable.
Please remove executable permission bits. Proceeding anyway. Nov
28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt Engine Data Warehouse... Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Started
oVirt Engine Data Warehouse. Nov 28 12:32:13
lago-upgrade-from-release-suite-master-engine systemd: Reloading.
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine
systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable.
Please remove executable permission bits. Proceeding anyway. Nov
28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt ImageIO Proxy... Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last): Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov
28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line 21, in main Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45, in start Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in
use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service: main process exited,
code=exited, status=1/FAILURE Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last): Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov
28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line 21, in main Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45, in start Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address) Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28
12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in
use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service: main process exited,
code=exited, status=1/FAILURE Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed. Nov 28 12:32:14
lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
*
_______________________________________________
Devel mailing list
Devel(a)ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel
--------------98CB106BD8B4230B34117EA6
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl
eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp
dGUtcHJlZml4Ij50aGlzIGlzIHRoZSBwbHVnaW4gaW5mbyBmcm9tIHN0ZXVwIGxvZwogICAg
ICBidXQgSSBkb24ndCBzZWUgYW55dGhpbmcgbW9yZSB0aGFuIHdlIGhhdmUgc2VlbiBleGNl
cHQgYSB0aW1lb3V0LgogICAgICA8YnI+CiAgICAgIDxicj4KICAgICAgPGEgY2xhc3M9Im1v
ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cHM6Ly9wYXN0ZWJpbi5jb20vUVZ0TlJO
V1YiPmh0dHBzOi8vcGFzdGViaW4uY29tL1FWdE5STldWPC9hPjxicj4KICAgICAgPGJyPgo8
YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5v
dmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0
aWZhY3QvZXhwb3J0ZWQtYXJ0aWZhY3RzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXQtbWFz
dGVyLWVsNy90ZXN0X2xvZ3MvdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyL3Bv
c3QtMDAxX2luaXRpYWxpemVfZW5naW5lLnB5L2xhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut
c3VpdGUtbWFzdGVyLWVuZ2luZS9fdmFyX2xvZy9vdmlydC1lbmdpbmUvc2V0dXAvb3ZpcnQt
ZW5naW5lLXNldHVwLTIwMTcxMTI4MTIzMTE2LW1tamVuMy5sb2ciPmh0dHA6Ly9qZW5raW5z
Lm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDE5NC9h
cnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdC1t
YXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIv
cG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJfbG9nL292aXJ0LWVuZ2luZS9zZXR1cC9vdmly
dC1lbmdpbmUtc2V0dXAtMjAxNzExMjgxMjMxMTYtbW1qZW4zLmxvZzwvYT48YnI+CiAgICAg
IDxicj4KICAgICAgRGlkaSwgaXMgdGhlcmUgYW55d2hlcmUgZWxzZSBJIHNob3VsZCBsb29r
PyA8YnI+CiAgICAgIDxicj4KICAgICAgPGJyPgogICAgICBPbiAxMS8yOS8yMDE3IDEwOjE4
IEFNLCBOaXIgU29mZmVyIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUg
dHlwZT0iY2l0ZSIKY2l0ZT0ibWlkOkNBTVJieXl2Z1RIc2k3czNQX2I9RmhEN2czSG1QTTlB
UXM0cGRBcHRoaVNNcktOSzdfQUBtYWlsLmdtYWlsLmNvbSI+CiAgICAgIDxkaXYgZGlyPSJs
dHIiPkRvIHdlIGhhdmUgbW9yZSBpbmZvIGZyb20gRGlkaSdzIGRlYnVnIHBsdWdpbiBub3c/
PGJyPgogICAgICAgIDxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAg
ICAgICAgICA8ZGl2IGRpcj0ibHRyIj5PbiBXZWQsIE5vdiAyOSwgMjAxNyBhdCAxMjowNyBQ
TSBEYWZuYSBSb24gJmx0OzxhCiAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOmRyb25AcmVk
aGF0LmNvbSIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5kcm9uQHJlZGhhdC5jb208L2E+Jmd0
OwogICAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxi
bG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMAogICAg
ICAgICAgICAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDox
ZXgiPgogICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYi
PgogICAgICAgICAgICAgIDxwcmUgc3R5bGU9ImNvbG9yOnJnYigwLDAsMCk7Zm9udC1zdHls
ZTpub3JtYWw7Zm9udC12YXJpYW50LWxpZ2F0dXJlczpub3JtYWw7Zm9udC12YXJpYW50LWNh
cHM6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3JtYWw7dGV4
dC1hbGlnbjpzdGFydDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3b3Jk
LXNwYWNpbmc6MHB4O3RleHQtZGVjb3JhdGlvbi1zdHlsZTppbml0aWFsO3RleHQtZGVjb3Jh
dGlvbi1jb2xvcjppbml0aWFsO3dvcmQtd3JhcDpicmVhay13b3JkO3doaXRlLXNwYWNlOnBy
ZS13cmFwIj48YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIiBpZD0ibV8tNzk2Mzg5Mjc2
MzM4MTgyNDI4NGRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0z
YTZlNGUxZDlhNjYiPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJn
aW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox
MXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0
cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJp
YW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGlu
ZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+SGksIAo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz
dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw
dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y
OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7
Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246
bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwv
c3Bhbj48L3A+PC9iPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXy03OTYz
ODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0y
OWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4
O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1z
aXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNv
bG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250
LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJh
c2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5XZSBoYXZlIGZhaWxlZCBjcSB3aXRoIG92
aXJ0LWltYWdlaW8gZmFpbGluZyB0byBzdGFydCBvbiB1cGdyYWRlIHN1aXRlPGI+LiA8L2I+
SSBjYW4gc3RpbGwgb25seSBzZWUgZXJyb3JzPGI+IDwvYj5pbiB0aGUgbWVzc2FnZXMgbG9n
LiAKPC9zcGFuPjwvcD48L2I+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIgaWQ9Im1f
LTc5NjM4OTI3NjMzODE4MjQyODRkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtMDczNi1j
N2ZjLTI5YzAtM2E2ZTRlMWQ5YTY2Ij48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0
OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91
bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFs
O2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxp
Z246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPkknbSB3cml0aW5nIHRoZSByZXBv
cnRlZCBwYXRjaCBidXQgSSBkb24ndCB0aGluayBpdCBoYXMgYW55dGhpbmcgdG8gZG8gd2l0
aCB0aGlzIGlzc3VlLiAKPC9zcGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVp
Z2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxl
PSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tn
cm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9y
bWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwt
YWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPgo8L3NwYW4+PC9wPjxwIGRp
cj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4t
Ym90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFy
aWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdl
aWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRl
Y29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUt
d3JhcCI+TGluayBhbmQgaGVhZGxpbmUgb2Ygc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48
L3A+cmVzdGFwaTogRW5hYmxlIHVwZGF0ZSB0byBubyBkZWZhdWx0IG5ldHdvcmsgcHJvdmlk
ZXIgb2YgY2x1c3RlciAtIDxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96LXR4
dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzg0
ODE0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9n
ZXJyaXQub3ZpcnQub3JnLyMvYy84NDgxNC88L2E+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGlu
ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4g
c3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7
YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHls
ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0
aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+
PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21h
cmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1p
bHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2Zv
bnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3Rl
eHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNl
OnByZS13cmFwIj5MaW5rIHRvIEpvYjo8L3NwYW4+PC9wPjxhIGNsYXNzPSJtXy03OTYzODky
NzYzMzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lu
cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv
IiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lu
cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv
PC9hPgoKPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6
MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9u
dC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFy
ZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9y
bWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRl
LXNwYWNlOnByZS13cmFwIj5MaW5rIHRvIGFsbCBsb2dzOjwvc3Bhbj48L3A+PHAgZGlyPSJs
dHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0
b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7
Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0
OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3Jh
dGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw
Ij4KPC9zcGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFy
Z2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6
MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6
dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFy
aWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxp
bmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPjxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0
Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5v
cmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0aWZhY3Qv
IiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lu
cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv
YXJ0aWZhY3QvPC9hPjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdo
dDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0i
Zm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3Jv
dW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1h
bDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFs
aWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD48cCBkaXI9
Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJv
dHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlh
bDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWln
aHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNv
cmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdy
YXAiPjxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0
ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9j
aGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvdGVzdFJlcG9ydC9qdW5pdC8lMjhyb290JTI5LzAw
MV9pbml0aWFsaXplX2VuZ2luZS90ZXN0X2luaXRpYWxpemVfZW5naW5lLyIgdGFyZ2V0PSJf
YmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3Jn
L2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MTk0L3Rlc3RSZXBvcnQv
anVuaXQvKHJvb3QpLzAwMV9pbml0aWFsaXplX2VuZ2luZS90ZXN0X2luaXRpYWxpemVfZW5n
aW5lLzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4z
ODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQt
c2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1j
b2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9u
dC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpi
YXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+PC9iPjxiIHN0eWxl
PSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1p
bnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAg
ZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdp
bi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6
QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQt
d2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQt
ZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnBy
ZS13cmFwIj48YSBjbGFzcz0ibV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1m
cmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0
ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0
cy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2lu
ZS5weS9sYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zh
cl9sb2cvbWVzc2FnZXMvKnZpZXcqLyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2Vu
ZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hh
bmdlLXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy91cGdy
YWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3JhZGUtZnJv
bS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2luZS5weS9s
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cv
bWVzc2FnZXMvKnZpZXcqLzwvYT4KPC9zcGFuPjwvcD4KPHAgZGlyPSJsdHIiIHN0eWxlPSJs
aW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3Bh
biBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAw
MDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0
eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3Zl
cnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4oUmVsZXZhbnQp
IGVycm9yIHNuaXBwZXQgZnJvbSB0aGUgbG9nOiA8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz
dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw
dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y
OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7
Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246
bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0
O2Vycm9yJmd0Ozwvc3Bhbj48L3A+CjwvYj5Gcm9tIG1lc3NhZ2VzIGxvZwo8YiBzdHlsZT0i
Zm9udC13ZWlnaHQ6bm9ybWFsIiBpZD0ibV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NGRvY3MtaW50
ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0zYTZlNGUxZDlhNjYiPgoKTm92
IDI4IDEyOjMyOjEzIGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu
Z2luZSBzeXN0ZW1kOiBTdGFydGVkIG9WaXJ0IEVuZ2luZS4KTm92IDI4IDEyOjMyOjEzIGxh
Z28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBS
ZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1
aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogQ29uZmlndXJhdGlvbiBmaWxlIC91c3IvbGli
L3N5c3RlbWQvc3lzdGVtL2VidGFibGVzLnNlcnZpY2UgaXMgbWFya2VkIGV4ZWN1dGFibGUu
IFBsZWFzZSByZW1vdmUgZXhlY3V0YWJsZSBwZXJtaXNzaW9uIGJpdHMuIFByb2NlZWRpbmcg
YW55d2F5LgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0
ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFN0YXJ0aW5nIG9WaXJ0IEVuZ2luZSBEYXRhIFdh
cmVob3VzZS4uLgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFN0YXJ0ZWQgb1ZpcnQgRW5naW5lIERhdGEg
V2FyZWhvdXNlLgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFJlbG9hZGluZy4KTm92IDI4IDEyOjMyOjEz
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1k
OiBDb25maWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vZWJ0YWJsZXMu
c2VydmljZSBpcyBtYXJrZWQgZXhlY3V0YWJsZS4gUGxlYXNlIHJlbW92ZSBleGVjdXRhYmxl
IHBlcm1pc3Npb24gYml0cy4gUHJvY2VlZGluZyBhbnl3YXkuCk5vdiAyOCAxMjozMjoxMyBs
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDog
U3RhcnRpbmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpOb3YgMjggMTI6MzI6MTQgbGFnby11
cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8t
cHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92IDI4IDEyOjMy
OjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly
dC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9vdmlydC1pbWFnZWlvLXByb3h5Iiwg
bGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFk
ZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5
OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3MsIGNvbmZpZykKTm92IDI4IDEyOjMy
OjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly
dC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdl
cy9vdmlydF9pbWFnZWlvX3Byb3h5L2ltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFp
bgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0
ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGltYWdlX3NlcnZlci5zdGFydChjb25m
aWcpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1h
c3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0aG9u
Mi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS9zZXJ2ZXIucHkiLCBsaW5l
IDQ1LCBpbiBzdGFydApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1ZXN0
SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp
dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0
L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQxOSwgaW4gX19pbml0X18KTm92
IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu
Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNlcnZlcl9iaW5kKCkKTm92IDI4IDEy
OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dpcmVm
L3NpbXBsZV9zZXJ2ZXIucHkiLCBsaW5lIDQ4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjggMTI6
MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92
aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBTZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4
IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu
ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9CYXNl
SFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjggMTI6MzI6
MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0
LWltYWdlaW8tcHJveHk6IFNvY2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyX2JpbmQoc2Vs
ZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFz
dGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhv
bjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQKTm92IDI4
IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu
ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tldC5iaW5kKHNlbGYuc2VydmVyX2Fk
ZHJlc3MpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRl
LW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9w
eXRob24yLjcvc29ja2V0LnB5IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDI4IDEyOjMyOjE0
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1p
bWFnZWlvLXByb3h5OiByZXR1cm4gZ2V0YXR0cihzZWxmLl9zb2NrLG5hbWUpKCphcmdzKQpO
b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIt
ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBB
ZGRyZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJv
bS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1w
cm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVkLCBjb2RlPWV4aXRlZCwgc3RhdHVz
PTEvRkFJTFVSRQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJbWFn
ZUlPIFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1wcm94eS5z
ZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdy
YWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0LWlt
YWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1h
Z2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBvdmVyLCBzY2hlZHVsaW5nIHJlc3Rh
cnQuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1h
c3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRpbmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpO
b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIt
ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2Fs
bCBsYXN0KToKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp
dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9v
dmlydC1pbWFnZWlvLXByb3h5IiwgbGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4
IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu
ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3Ms
IGNvbmZpZykKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp
dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9w
eXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5L2ltYWdlX3Byb3h5
LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGlt
YWdlX3NlcnZlci5zdGFydChjb25maWcpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUt
ZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTog
RmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19w
cm94eS9zZXJ2ZXIucHkiLCBsaW5lIDQ1LCBpbiBzdGFydApOb3YgMjggMTI6MzI6MTQgbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdl
aW8tcHJveHk6IFdTR0lSZXF1ZXN0SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5l
IDQxOSwgaW4gX19pbml0X18KTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJl
bGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNl
cnZlcl9iaW5kKCkKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut
c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xp
YjY0L3B5dGhvbjIuNy93c2dpcmVmL3NpbXBsZV9zZXJ2ZXIucHkiLCBsaW5lIDQ4LCBpbiBz
ZXJ2ZXJfYmluZApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBTZXJ2ZXIuc2Vy
dmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh
c2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNy
L2xpYjY0L3B5dGhvbjIuNy9CYXNlSFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2
ZXJfYmluZApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0
ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFNvY2tldFNlcnZlci5UQ1BT
ZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1m
cm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG
aWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwg
aW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh
c2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tl
dC5iaW5kKHNlbGYuc2VydmVyX2FkZHJlc3MpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94
eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc29ja2V0LnB5IiwgbGluZSAyMjQsIGlu
IG1ldGgKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUt
bWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiByZXR1cm4gZ2V0YXR0cihzZWxm
Ll9zb2NrLG5hbWUpKCphcmdzKQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20t
cmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tl
dC5lcnJvcjogW0Vycm5vIDk4XSBBZGRyZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjoz
MjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lz
dGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVk
LCBjb2RlPWV4aXRlZCwgc3RhdHVzPTEvRkFJTFVSRQpOb3YgMjggMTI6MzI6MTQgbGFnby11
cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxl
ZCB0byBzdGFydCBvVmlydCBJbWFnZUlPIFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11
cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQg
b3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3Yg
MjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5n
aW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAy
OCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdp
bmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBv
dmVyLCBzY2hlZHVsaW5nIHJlc3RhcnQuCjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWln
aHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9
ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dy
b3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3Jt
YWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1h
bGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0Oy9lcnJvciZndDs8L3Nw
YW4+PC9wPjwvYj4KCgoKCgoKCgoKCgoKCgoKPC9wcmU+CiAgICAgICAgICAgIDwvZGl2Pgog
ICAgICAgICAgICBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fXzxicj4KICAgICAgICAgICAgRGV2ZWwgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAg
ICA8YSBocmVmPSJtYWlsdG86RGV2ZWxAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayIKICAg
ICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPkRldmVsQG92aXJ0Lm9yZzwvYT48
YnI+CiAgICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h
bi9saXN0aW5mby9kZXZlbCIKICAgICAgICAgICAgICByZWw9Im5vcmVmZXJyZXIiIHRhcmdl
dD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmlydC5v
cmcvbWFpbG1hbi9saXN0aW5mby9kZXZlbDwvYT48L2Jsb2NrcXVvdGU+CiAgICAgICAgPC9k
aXY+CiAgICAgIDwvZGl2PgogICAgPC9ibG9ja3F1b3RlPgogICAgPHA+PGJyPgogICAgPC9w
PgogIDwvYm9keT4KPC9odG1sPgo=
--------------98CB106BD8B4230B34117EA6--
5:03 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com> wrote:
this is the plugin info from steup log but I don't see anything
more than
we have seen except a timeout.
https://pastebin.com/QVtNRNWV
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4194/artifact/exported-artifacts/upgrade-from-
release-suit-master-el7/test_logs/upgrade-from-release-
suite-master/post-001_initialize_engine.py/lago-
upgrade-from-release-suite-master-engine/_var_log/ovirt-
engine/setup/ovirt-engine-setup-20171128123116-mmjen3.log
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
> *Hi,
>
> *
>
> *We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still
only see errors in the messages log.
> *
>
> *I'm writing the reported patch but I don't think it has anything to do with
this issue.
>
> Link and headline of suspected patches: restapi: Enable update to no default network
provider of cluster - https://gerrit.ovirt.org/#/c/84814/
<https://gerrit.ovirt.org/#/c/84814/>
>
> Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
>
> Link to all logs:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
> *
>
>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
> (Relevant) error snippet from the log: <error>
> *From messages log
>
> *
>
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration
file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove
executable permission bits. Proceeding anyway.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
Engine Data Warehouse...
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt
Engine Data Warehouse.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration
file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove
executable permission bits. Proceeding anyway.
> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> </error>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
--
Didi
7:56 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
This is a multi-part message in MIME format.
--------------D2B187A49A90F623211FB3B4
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
we had a failure on 002_bootstrap.verify_add_hosts but the error is on
imageio
I looked at the host log that Nir added and I can only see that the
address is in use which seems to be the same issue we have in initialize
engine.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifa...
I cannot see anything in host-deploy.
Didi, would we be able to see anything here?
Thanks,
Dafna
On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com
<mailto:dron@redhat.com>> wrote:
this is the plugin info from steup log but I don't see anything
more than we have seen except a timeout.
https://pastebin.com/QVtNRNWV
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
> Do we have more info from Didi's debug plugin now?
>
> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com
> <mailto:dron@redhat.com>> wrote:
>
> *
>
> Hi,
>
> **
>
> We have failed cq with ovirt-imageio failing to start on
> upgrade suite*. *I can still only see errors**in the messages
> log.
>
> **
>
> I'm writing the reported patch but I don't think it has
> anything to do with this issue.
>
> Link and headline of suspected patches:
>
> restapi: Enable update to no default network provider of
> cluster - https://gerrit.ovirt.org/#/c/84814/
> <https://gerrit.ovirt.org/#/c/84814/>
>
> Link to Job:
>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
>
>
> Link to all logs:
>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
>
> **
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>
>
> (Relevant) error snippet from the log:
>
> <error>
>
> *From messages log
> *Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine
> systemd: Started oVirt Engine. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Reloading. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Configuration file /usr/lib/systemd/system/ebtables.service
> is marked executable. Please remove executable permission
> bits. Proceeding anyway. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Starting oVirt Engine Data Warehouse... Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Started oVirt Engine Data Warehouse. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Reloading. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Configuration file /usr/lib/systemd/system/ebtables.service
> is marked executable. Please remove executable permission
> bits. Proceeding anyway. Nov 28 12:32:13
> lago-upgrade-from-release-suite-master-engine systemd:
> Starting oVirt ImageIO Proxy... Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: Traceback (most recent call last): Nov
> 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy",
> line 85, in <module> Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
> line 21, in main Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: image_server.start(config) Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
> line 45, in start Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/SocketServer.py", line 430, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
> line 224, in meth Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: socket.error: [Errno 98] Address already
> in use Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd:
> ovirt-imageio-proxy.service: main process exited,
> code=exited, status=1/FAILURE Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd: Failed
> to start oVirt ImageIO Proxy. Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd: Unit
> ovirt-imageio-proxy.service entered failed state. Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd:
> ovirt-imageio-proxy.service holdoff time over, scheduling
> restart. Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd:
> Starting oVirt ImageIO Proxy... Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: Traceback (most recent call last): Nov
> 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy",
> line 85, in <module> Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: status = image_proxy.main(args, config)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
> line 21, in main Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: image_server.start(config) Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
> line 45, in start Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File
> "/usr/lib64/python2.7/SocketServer.py", line 430, in
> server_bind Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: self.socket.bind(self.server_address)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
> line 224, in meth Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
> ovirt-imageio-proxy: socket.error: [Errno 98] Address already
> in use Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd:
> ovirt-imageio-proxy.service: main process exited,
> code=exited, status=1/FAILURE Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd: Failed
> to start oVirt ImageIO Proxy. Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd: Unit
> ovirt-imageio-proxy.service entered failed state. Nov 28
> 12:32:14 lago-upgrade-from-release-suite-master-engine
> systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14
> lago-upgrade-from-release-suite-master-engine systemd:
> ovirt-imageio-proxy.service holdoff time over, scheduling
> restart.
>
> </error>
>
> *
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/devel
> <http://lists.ovirt.org/mailman/listinfo/devel>
>
--
Didi
--------------D2B187A49A90F623211FB3B4
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl
eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp
dGUtcHJlZml4Ij53ZSBoYWQgYSBmYWlsdXJlIG9uIDxzcGFuCiAgICAgICAgc3R5bGU9ImNv
bG9yOiByZ2IoNTEsIDUxLCA1MSk7IGZvbnQtc2l6ZTogMTRweDsgZm9udC1zdHlsZToKICAg
ICAgICBub3JtYWw7IGZvbnQtdmFyaWFudC1saWdhdHVyZXM6IG5vcm1hbDsgZm9udC12YXJp
YW50LWNhcHM6CiAgICAgICAgbm9ybWFsOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0
LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6CiAgICAgICAgMHB4OyB0ZXh0LXRyYW5zZm9y
bTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOgogICAgICAgIDBw
eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6CiAgICAgICAgaW5pdGlhbDsgdGV4dC1kZWNvcmF0aW9uLWNvbG9yOiBpbml0aWFsOyBk
aXNwbGF5OiBpbmxpbmUgIQogICAgICAgIGltcG9ydGFudDsgZmxvYXQ6IG5vbmU7Ij4wMDJf
Ym9vdHN0cmFwLnZlcmlmeV9hZGRfaG9zdHMgYnV0IHRoZQogICAgICAgIGVycm9yIGlzIG9u
IGltYWdlaW8gPGJyPgogICAgICAgIDxicj4KICAgICAgICBJIGxvb2tlZCBhdCB0aGUgaG9z
dCBsb2cgdGhhdCBOaXIgYWRkZWQgYW5kIEkgY2FuIG9ubHkgc2VlIHRoYXQKICAgICAgICB0
aGUgYWRkcmVzcyBpcyBpbiB1c2Ugd2hpY2ggc2VlbXMgdG8gYmUgdGhlIHNhbWUgaXNzdWUg
d2UgaGF2ZQogICAgICAgIGluIGluaXRpYWxpemUgZW5naW5lLiA8YnI+CiAgICAgIDwvc3Bh
bj48YnI+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9q
ZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIv
NDIwNS9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMtc3VpdC1tYXN0ZXItZWw3
L3Rlc3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJfYm9vdHN0cmFwLnB5L2xh
Z28tYmFzaWMtc3VpdGUtbWFzdGVyLWhvc3QtMC9fdmFyX2xvZy9vdmlydC1pbWFnZWlvLWRh
ZW1vbi8iPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5n
ZS1xdWV1ZS10ZXN0ZXIvNDIwNS9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMt
c3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJf
Ym9vdHN0cmFwLnB5L2xhZ28tYmFzaWMtc3VpdGUtbWFzdGVyLWhvc3QtMC9fdmFyX2xvZy9v
dmlydC1pbWFnZWlvLWRhZW1vbi88L2E+PGJyPgogICAgICA8YnI+CiAgICAgIEkgY2Fubm90
IHNlZSBhbnl0aGluZyBpbiBob3N0LWRlcGxveS4gPGJyPgogICAgICBEaWRpLCB3b3VsZCB3
ZSBiZSBhYmxlIHRvIHNlZSBhbnl0aGluZyBoZXJlPyA8YnI+CiAgICAgIDxicj4KICAgICAg
VGhhbmtzLCA8YnI+CiAgICAgIERhZm5hPGJyPgogICAgICA8YnI+CiAgICAgIDxicj4KICAg
ICAgT24gMTEvMjkvMjAxNyAxMTowMyBBTSwgWWVkaWR5YWggQmFyIERhdmlkIHdyb3RlOjxi
cj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKY2l0ZT0ibWlkOkNB
SFJ3WVh1bUdkLUU1eDZrZ21GRzdtT2lMcFZnUlNrPXVLZ0hwTU5pM242Q0o4X2Vrd0BtYWls
LmdtYWlsLmNvbSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgIDxkaXYgY2xhc3M9
ImdtYWlsX2V4dHJhIj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBX
ZWQsIE5vdiAyOSwgMjAxNyBhdCAxOjAwIFBNLAogICAgICAgICAgICBEYWZuYSBSb24gPHNw
YW4gZGlyPSJsdHIiPiZsdDs8YQogICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOmRyb25A
cmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICBtb3otZG8tbm90
LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAg
ICB3cm90ZTo8YnI+CiAgICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90
ZSIgc3R5bGU9Im1hcmdpbjowIDAgMAogICAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6
MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgPGRpdiB0
ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgICAgICAgICAgICAgIDxkaXYg
Y2xhc3M9Im1fMTIzMjE1NjgyODI3NDgwMjU2N21vei1jaXRlLXByZWZpeCI+dGhpcwogICAg
ICAgICAgICAgICAgICBpcyB0aGUgcGx1Z2luIGluZm8gZnJvbSBzdGV1cCBsb2cgYnV0IEkg
ZG9uJ3Qgc2VlCiAgICAgICAgICAgICAgICAgIGFueXRoaW5nIG1vcmUgdGhhbiB3ZSBoYXZl
IHNlZW4gZXhjZXB0IGEgdGltZW91dC4gPGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAg
ICAgICAgICAgICAgICAgIDxhIGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1Njdtb3otdHh0
LWxpbmstZnJlZXRleHQiCiAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cHM6Ly9wYXN0
ZWJpbi5jb20vUVZ0TlJOV1YiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICAg
IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9wYXN0ZWJpbi5jb20vUVZ0TlJOV1Y8
L2E+PGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgIDxhIGNs
YXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1Njdtb3otdHh0LWxpbmstZnJlZXRleHQiCmhyZWY9
Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1
ZS10ZXN0ZXIvNDE5NC9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvdXBncmFkZS1mcm9t
LXJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby11cGdy
YWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJfbG9nL292aXJ0LWVu
Z2luZS9zZXR1cC9vdmlydC1lbmdpbmUtc2V0dXAtMjAxNzExMjgxMjMxMTYtbW1qZW4zLmxv
ZyIKICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5k
PSJ0cnVlIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+b3ZpcnQtbWFzdGVy
X2NoYW5nZS1xdWV1ZS08d2JyPnRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLTx3YnI+
YXJ0aWZhY3RzL3VwZ3JhZGUtZnJvbS08d2JyPnJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rl
c3RfPHdicj5sb2dzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyL3Bv
c3QtMDAxXzx3YnI+aW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby08d2JyPnVwZ3JhZGUtZnJv
bS1yZWxlYXNlLXN1aXRlLTx3YnI+bWFzdGVyLWVuZ2luZS9fdmFyX2xvZy9vdmlydC08d2Jy
PmVuZ2luZS9zZXR1cC9vdmlydC1lbmdpbmUtPHdicj5zZXR1cC0yMDE3MTEyODEyMzExNi1t
bWplbjMuPHdicj5sb2c8L2E+PGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAg
ICAgICAgICAgIERpZGksIGlzIHRoZXJlIGFueXdoZXJlIGVsc2UgSSBzaG91bGQgbG9vaz8g
PGJyPgogICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAg
ICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAg
IDwvZGl2PgogICAgICAgICAgICA8ZGl2PlNhZGx5LCBhcyBhbHJlYWR5IHJlcGxpZWQsIG5v
dCB5ZXQuIEhvcGVmdWxseSBuZXh0CiAgICAgICAgICAgICAgdGltZS4uLjxicj4KICAgICAg
ICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXY+wqA8L2Rpdj4KICAgICAgICAgICAgPGJs
b2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAg
ICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6
MWV4Ij4KICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZG
RkYiPgogICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3
bW96LWNpdGUtcHJlZml4Ij4KICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAgICAg
ICAgICAgICA8ZGl2IGNsYXNzPSJoNSI+IDxicj4KICAgICAgICAgICAgICAgICAgICAgIDxi
cj4KICAgICAgICAgICAgICAgICAgICAgIE9uIDExLzI5LzIwMTcgMTA6MTggQU0sIE5pciBT
b2ZmZXIgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg
ICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAg
PGRpdj4KICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iaDUiPgogICAgICAgICAgICAg
ICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgICAgICAgICAg
PGRpdiBkaXI9Imx0ciI+RG8gd2UgaGF2ZSBtb3JlIGluZm8gZnJvbSBEaWRpJ3MKICAgICAg
ICAgICAgICAgICAgICAgICAgZGVidWcgcGx1Z2luIG5vdz88YnI+CiAgICAgICAgICAgICAg
ICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21h
aWxfcXVvdGUiPgogICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPk9u
IFdlZCwgTm92IDI5LCAyMDE3IGF0IDEyOjA3CiAgICAgICAgICAgICAgICAgICAgICAgICAg
ICBQTSBEYWZuYSBSb24gJmx0OzxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhy
ZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iCiAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0
LmNvbTwvYT4mZ3Q7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+CiAg
ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAg
ICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIgogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweAogICAg
ICAgICAgICAgICAgICAgICAgICAgICAgI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xv
cj0iI0ZGRkZGRiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwcmUgc3R5bGU9
ImNvbG9yOnJnYigwLDAsMCk7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50LWxpZ2F0
dXJlczpub3JtYWw7Zm9udC12YXJpYW50LWNhcHM6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1h
bDtsZXR0ZXItc3BhY2luZzpub3JtYWw7dGV4dC1hbGlnbjpzdGFydDt0ZXh0LWluZGVudDow
cHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3b3JkLXNwYWNpbmc6MHB4O3RleHQtZGVjb3JhdGlv
bi1zdHlsZTppbml0aWFsO3RleHQtZGVjb3JhdGlvbi1jb2xvcjppbml0aWFsO3dvcmQtd3Jh
cDpicmVhay13b3JkO3doaXRlLXNwYWNlOnByZS13cmFwIj48YiBzdHlsZT0iZm9udC13ZWln
aHQ6bm9ybWFsIiBpZD0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3bV8tNzk2Mzg5Mjc2MzM4MTgy
NDI4NGRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0zYTZlNGUx
ZDlhNjYiPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9w
OjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2Zv
bnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3Bh
cmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5v
cm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0
ZS1zcGFjZTpwcmUtd3JhcCI+SGksIAo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0i
bGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNw
YW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAw
MDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1z
dHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2
ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48
L3A+PC9iPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXzEyMzIxNTY4Mjgy
NzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTli
N2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJs
aW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3Bh
biBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAw
MDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0
eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3Zl
cnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5XZSBoYXZlIGZh
aWxlZCBjcSB3aXRoIG92aXJ0LWltYWdlaW8gZmFpbGluZyB0byBzdGFydCBvbiB1cGdyYWRl
IHN1aXRlPGI+LiA8L2I+SSBjYW4gc3RpbGwgb25seSBzZWUgZXJyb3JzPGI+IDwvYj5pbiB0
aGUgbWVzc2FnZXMgbG9nLiAKPC9zcGFuPjwvcD48L2I+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0
Om5vcm1hbCIgaWQ9Im1fMTIzMjE1NjgyODI3NDgwMjU2N21fLTc5NjM4OTI3NjMzODE4MjQy
ODRkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtMDczNi1jN2ZjLTI5YzAtM2E2ZTRlMWQ5
YTY2Ij48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDow
cHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250
LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJl
bnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3Jt
YWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUt
c3BhY2U6cHJlLXdyYXAiPkknbSB3cml0aW5nIHRoZSByZXBvcnRlZCBwYXRjaCBidXQgSSBk
b24ndCB0aGluayBpdCBoYXMgYW55dGhpbmcgdG8gZG8gd2l0aCB0aGlzIGlzc3VlLiAKPC9z
cGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRv
cDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtm
b250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw
YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpu
b3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hp
dGUtc3BhY2U6cHJlLXdyYXAiPgo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGlu
ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4g
c3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7
YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHls
ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0
aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+TGluayBhbmQgaGVh
ZGxpbmUgb2Ygc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48L3A+cmVzdGFwaTogRW5hYmxl
IHVwZGF0ZSB0byBubyBkZWZhdWx0IG5ldHdvcmsgcHJvdmlkZXIgb2YgY2x1c3RlciAtIDxh
IGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96
LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9j
Lzg0ODE0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6
Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy88d2JyPjg0ODE0LzwvYT4KPHAgZGlyPSJsdHIiIHN0
eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0
Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6
IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtm
b250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpu
b25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9z
cGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRv
cDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtm
b250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw
YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpu
b3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hp
dGUtc3BhY2U6cHJlLXdyYXAiPkxpbmsgdG8gSm9iOjwvc3Bhbj48L3A+PGEgY2xhc3M9Im1f
MTIzMjE1NjgyODI3NDgwMjU2N21fLTc5NjM4OTI3NjMzODE4MjQyODRtb3otdHh0LWxpbmst
ZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFz
dGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDE5NC8iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8t
bm90LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmly
dC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLTx3YnI+dGVzdGVyLzQxOTQvPC9hPgoKPHAgZGlyPSJs
dHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0
b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7
Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0
OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3Jh
dGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw
Ij5MaW5rIHRvIGFsbCBsb2dzOjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5l
LWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBz
dHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDti
YWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxl
Om5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRp
Y2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD48
cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFy
Z2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWls
eTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9u
dC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4
dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6
cHJlLXdyYXAiPjxhIGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYz
MzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5v
dmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0
aWZhY3QvIiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8v
amVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+b3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS08
d2JyPnRlc3Rlci80MTk0L2FydGlmYWN0LzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz
dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw
dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y
OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7
Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246
bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwv
c3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10
b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7
Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5z
cGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6
bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3do
aXRlLXNwYWNlOnByZS13cmFwIj48YSBjbGFzcz0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3bV8t
Nzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDov
L2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rl
ci80MTk0L3Rlc3RSZXBvcnQvanVuaXQvJTI4cm9vdCUyOS8wMDFfaW5pdGlhbGl6ZV9lbmdp
bmUvdGVzdF9pbml0aWFsaXplX2VuZ2luZS8iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90
LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmlydC1t
YXN0ZXJfY2hhbmdlLXF1ZXVlLTx3YnI+dGVzdGVyLzQxOTQvdGVzdFJlcG9ydC9qdW5pdC8o
PHdicj5yb290KS8wMDFfaW5pdGlhbGl6ZV9lbmdpbmUvPHdicj50ZXN0X2luaXRpYWxpemVf
ZW5naW5lLzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6
MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZv
bnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3Vu
ZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7
Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGln
bjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+PC9iPjxiIHN0
eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03
OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdm
Yy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDox
LjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9u
dC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5k
LWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtm
b250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWdu
OmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj48YSBjbGFzcz0ibV8xMjMyMTU2ODI4
Mjc0ODAyNTY3bV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1mcmVldGV4dCIg
aHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdl
LXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy91cGdyYWRl
LWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3JhZGUtZnJvbS1y
ZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2luZS5weS9sYWdv
LXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cvbWVz
c2FnZXMvKnZpZXcqLyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+
aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi88d2JyPm92aXJ0LW1hc3Rlcl9jaGFuZ2Ut
cXVldWUtPHdicj50ZXN0ZXIvNDE5NC9hcnRpZmFjdC9leHBvcnRlZC08d2JyPmFydGlmYWN0
cy91cGdyYWRlLWZyb20tPHdicj5yZWxlYXNlLXN1aXQtbWFzdGVyLWVsNy90ZXN0Xzx3YnI+
bG9ncy91cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci9wb3N0LTAwMV88
d2JyPmluaXRpYWxpemVfZW5naW5lLnB5L2xhZ28tPHdicj51cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS08d2JyPm1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cvPHdicj5tZXNzYWdlcy8qdmll
dyovPC9hPgo8L3NwYW4+PC9wPgo8cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEu
Mzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250
LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQt
Y29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2Zv
bnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246
YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPihSZWxldmFudCkgZXJyb3Igc25pcHBl
dCBmcm9tIHRoZSBsb2c6IDwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhl
aWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNr
Z3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5v
cm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2Fs
LWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4mbHQ7ZXJyb3ImZ3Q7PC9z
cGFuPjwvcD4KPC9iPkZyb20gbWVzc2FnZXMgbG9nCjxiIHN0eWxlPSJmb250LXdlaWdodDpu
b3JtYWwiIGlkPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0
ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2
NiI+CgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1
aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRlZCBvVmlydCBFbmdpbmUuCk5vdiAy
OCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy
LWVuZ2luZSBzeXN0ZW1kOiBSZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBDb25m
aWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vPHdicj5lYnRhYmxlcy5z
ZXJ2aWNlIGlzIG1hcmtlZCBleGVjdXRhYmxlLiBQbGVhc2UgcmVtb3ZlIGV4ZWN1dGFibGUg
cGVybWlzc2lvbiBiaXRzLiBQcm9jZWVkaW5nIGFueXdheS4KTm92IDI4IDEyOjMyOjEzIGxh
Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3Rl
bWQ6IFN0YXJ0aW5nIG9WaXJ0IEVuZ2luZSBEYXRhIFdhcmVob3VzZS4uLgpOb3YgMjggMTI6
MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdp
bmUgc3lzdGVtZDogU3RhcnRlZCBvVmlydCBFbmdpbmUgRGF0YSBXYXJlaG91c2UuCk5vdiAy
OCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy
LWVuZ2luZSBzeXN0ZW1kOiBSZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBDb25m
aWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vPHdicj5lYnRhYmxlcy5z
ZXJ2aWNlIGlzIG1hcmtlZCBleGVjdXRhYmxlLiBQbGVhc2UgcmVtb3ZlIGV4ZWN1dGFibGUg
cGVybWlzc2lvbiBiaXRzLiBQcm9jZWVkaW5nIGFueXdheS4KTm92IDI4IDEyOjMyOjEzIGxh
Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3Rl
bWQ6IFN0YXJ0aW5nIG9WaXJ0IEltYWdlSU8gUHJveHkuLi4KTm92IDI4IDEyOjMyOjE0IGxh
Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0
LWltYWdlaW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92
IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0
ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvYmluL292aXJ0LWlt
YWdlaW8tcHJveHkiPHdicj4sIGxpbmUgODUsIGluICZsdDttb2R1bGUmZ3Q7Ck5vdiAyOCAx
MjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVu
Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFy
Z3MsIGNvbmZpZykKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut
PHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91
c3IvbGliL3B5dGhvbjIuNy9zaXRlLTx3YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94
eS88d2JyPmltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjggMTI6MzI6
MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg
b3ZpcnQtaW1hZ2Vpby1wcm94eTogaW1hZ2Vfc2VydmVyLnN0YXJ0KGNvbmZpZykKTm92IDI4
IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXIt
ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9z
aXRlLTx3YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS88d2JyPnNlcnZlci5weSIs
IGxpbmUgNDUsIGluIHN0YXJ0Ck5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y
ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBX
U0dJUmVxdWVzdEhhbmRsZXIpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y
ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG
aWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPlNvY2tldFNlcnZlci5weSIsIGxpbmUg
NDE5LCBpbiBfX2luaXRfXwpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVs
ZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogc2Vs
Zi5zZXJ2ZXJfYmluZCgpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxl
YXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxl
ICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dpcmVmLzx3YnI+c2ltcGxlX3NlcnZlci5weSIs
IGxpbmUgNDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUt
ZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiBIVFRQU2VydmVyLnNlcnZlcl9iaW5kKHNlbGYpCk5vdiAyOCAxMjozMjoxNCBsYWdv
LXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1p
bWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPkJhc2VIVFRQ
U2VydmVyLnB5IiwgbGluZSAxMDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBs
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly
dC1pbWFnZWlvLXByb3h5OiBTb2NrZXRTZXJ2ZXIuVENQU2VydmVyLnNlcnZlcl88d2JyPmJp
bmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi
cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3Iv
bGliNjQvcHl0aG9uMi43Lzx3YnI+U29ja2V0U2VydmVyLnB5IiwgbGluZSA0MzAsIGluIHNl
cnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3
YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tl
dC5iaW5kKHNlbGYuc2VydmVyXzx3YnI+YWRkcmVzcykKTm92IDI4IDEyOjMyOjE0IGxhZ28t
dXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWlt
YWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NvY2tldC48d2JyPnB5
IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9t
LXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6
IHJldHVybiBnZXRhdHRyKHNlbGYuX3NvY2ssbmFtZSkoKjx3YnI+YXJncykKTm92IDI4IDEy
OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n
aW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBBZGRy
ZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y
ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBvdmlydC1pbWFnZWlv
LXByb3h5LnNlcnZpY2U6IG1haW4gcHJvY2VzcyBleGl0ZWQsIGNvZGU9ZXhpdGVkLCBzdGF0
dXM9MS9GQUlMVVJFCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl
LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBGYWlsZWQgdG8gc3RhcnQgb1Zp
cnQgSW1hZ2VJTyBQcm94eS4KTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJl
bGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1h
Z2Vpby1wcm94eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjggMTI6MzI6
MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg
c3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGZhaWxlZC4KTm92IDI4IDEy
OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n
aW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBob2xkb2ZmIHRpbWUg
b3Zlciwgc2NoZWR1bGluZyByZXN0YXJ0LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRl
LWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRp
bmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRl
LWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1w
cm94eTogVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgpOb3YgMjggMTI6MzI6
MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg
b3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9iaW4vb3ZpcnQtaW1hZ2Vpby1wcm94
eSI8d2JyPiwgbGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4IDEyOjMyOjE0IGxh
Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0
LWltYWdlaW8tcHJveHk6IHN0YXR1cyA9IGltYWdlX3Byb3h5Lm1haW4oYXJncywgY29uZmln
KQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRl
LW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0
aG9uMi43L3NpdGUtPHdicj5wYWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5Lzx3YnI+aW1h
Z2VfcHJveHkucHkiLCBsaW5lIDIxLCBpbiBtYWluCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVw
Z3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFn
ZWlvLXByb3h5OiBpbWFnZV9zZXJ2ZXIuc3RhcnQoY29uZmlnKQpOb3YgMjggMTI6MzI6MTQg
bGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3Zp
cnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtPHdicj5w
YWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5Lzx3YnI+c2VydmVyLnB5IiwgbGluZSA0NSwg
aW4gc3RhcnQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi
cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1ZXN0
SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi
cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3Iv
bGliNjQvcHl0aG9uMi43Lzx3YnI+U29ja2V0U2VydmVyLnB5IiwgbGluZSA0MTksIGluIF9f
aW5pdF9fCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+
c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNlcnZlcl9i
aW5kKCkKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5z
dWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGli
NjQvcHl0aG9uMi43L3dzZ2lyZWYvPHdicj5zaW1wbGVfc2VydmVyLnB5IiwgbGluZSA0OCwg
aW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh
c2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBT
ZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1m
cm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJv
eHk6IEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43Lzx3YnI+QmFzZUhUVFBTZXJ2ZXIucHki
LCBsaW5lIDEwOCwgaW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFk
ZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8t
cHJveHk6IFNvY2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyXzx3YnI+YmluZChzZWxmKQpO
b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1h
c3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9weXRo
b24yLjcvPHdicj5Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQK
Tm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1t
YXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNlbGYuc29ja2V0LmJpbmQoc2Vs
Zi5zZXJ2ZXJfPHdicj5hZGRyZXNzKQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94
eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc29ja2V0Ljx3YnI+cHkiLCBsaW5lIDIy
NCwgaW4gbWV0aApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08
d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogcmV0dXJuIGdl
dGF0dHIoc2VsZi5fc29jayxuYW1lKSgqPHdicj5hcmdzKQpOb3YgMjggMTI6MzI6MTQgbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQt
aW1hZ2Vpby1wcm94eTogc29ja2V0LmVycm9yOiBbRXJybm8gOThdIEFkZHJlc3MgYWxyZWFk
eSBpbiB1c2UKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi
cj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2Vy
dmljZTogbWFpbiBwcm9jZXNzIGV4aXRlZCwgY29kZT1leGl0ZWQsIHN0YXR1cz0xL0ZBSUxV
UkUKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0
ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJbWFnZUlP
IFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2Jy
PnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogVW5pdCBvdmlydC1pbWFnZWlvLXByb3h5
LnNlcnZpY2UgZW50ZXJlZCBmYWlsZWQgc3RhdGUuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVw
Z3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBv
dmlydC1pbWFnZWlvLXByb3h5LnNlcnZpY2UgZmFpbGVkLgpOb3YgMjggMTI6MzI6MTQgbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVt
ZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBvdmVyLCBzY2hl
ZHVsaW5nIHJlc3RhcnQuCjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODtt
YXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6
ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xv
cjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12
YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNl
bGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0Oy9lcnJvciZndDs8L3NwYW4+PC9wPjwv
Yj4KCgoKCgoKCgoKCgoKCgoKPC9wcmU+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIF9fX19fX19fX19fX19fX19fX19f
X19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICBEZXZlbCBtYWlsaW5nIGxpc3Q8YnI+CiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8YSBocmVmPSJtYWlsdG86RGV2ZWxAb3ZpcnQub3JnIgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVl
Ij5EZXZlbEBvdmlydC5vcmc8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
PGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cDovL2xpc3RzLm92
aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RldmVsIgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5v
dmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL2RldmVsPC9hPjwvYmxvY2txdW90ZT4K
ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICA8
L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAg
ICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAg
ICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICA8L2Rpdj4K
ICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8
YnI+CiAgICAgICAgICA8YnIgY2xlYXI9ImFsbCI+CiAgICAgICAgICA8YnI+CiAgICAgICAg
ICAtLSA8YnI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEt
c21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPkRpZGk8YnI+CiAgICAgICAgICA8L2Rpdj4K
ICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8cD48
YnI+CiAgICA8L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg==
--------------D2B187A49A90F623211FB3B4--
8:57 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron(a)redhat.com> wrote:
we had a failure on 002_bootstrap.verify_add_hosts but the error is
on
imageio
I looked at the host log that Nir added and I can only see that the
address is in use which seems to be the same issue we have in initialize
engine.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4205/artifact/exported-artifacts/basic-suit-master-
el7/test_logs/basic-suite-master/post-002_bootstrap.py/
lago-basic-suite-master-host-0/_var_log/ovirt-imageio-daemon/
I cannot see anything in host-deploy.
Didi, would we be able to see anything here?
Sorry, seems like my plugin is not enough. Will have a look.
Thanks,
Dafna
On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com> wrote:
> this is the plugin info from steup log but I don't see anything more than
> we have seen except a timeout.
>
> https://pastebin.com/QVtNRNWV
>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/4194/artifact/exported-artifacts/upgrade-from-release-
> suit-master-el7/test_logs/upgrade-from-release-suite-
> master/post-001_initialize_engine.py/lago-upgrade-from-
> release-suite-master-engine/_var_log/ovirt-engine/setup/
> ovirt-engine-setup-20171128123116-mmjen3.log
>
> Didi, is there anywhere else I should look?
>
Sadly, as already replied, not yet. Hopefully next time...
>
>
> On 11/29/2017 10:18 AM, Nir Soffer wrote:
>
> Do we have more info from Didi's debug plugin now?
>
> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
>
>> *Hi,
>>
>> *
>>
>> *We have failed cq with ovirt-imageio failing to start on upgrade suite. I can
still only see errors in the messages log.
>> *
>>
>> *I'm writing the reported patch but I don't think it has anything to do
with this issue.
>>
>> Link and headline of suspected patches: restapi: Enable update to no default
network provider of cluster - https://gerrit.ovirt.org/#/c/84814/
<https://gerrit.ovirt.org/#/c/84814/>
>>
>> Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
>>
>> Link to all logs:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
>> *
>>
>>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>
>> (Relevant) error snippet from the log: <error>
>> *From messages log
>>
>> *
>>
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started
oVirt Engine.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Reloading.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please
remove executable permission bits. Proceeding anyway.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting
oVirt Engine Data Warehouse...
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started
oVirt Engine Data Warehouse.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Reloading.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please
remove executable permission bits. Proceeding anyway.
>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting
oVirt ImageIO Proxy...
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting
oVirt ImageIO Proxy...
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>> </error>*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
--
Didi
--
Didi
9:03 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <didi(a)redhat.com> wrote:
On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron(a)redhat.com>
wrote:
>
> we had a failure on 002_bootstrap.verify_add_hosts but the error is on
> imageio
>
> I looked at the host log that Nir added and I can only see that the
> address is in use which seems to be the same issue we have in initialize
> engine.
>
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifa...
>
> I cannot see anything in host-deploy.
> Didi, would we be able to see anything here?
Sorry, seems like my plugin is not enough. Will have a look.
Now merged an updated plugin, should hopefully pass changequeue soon.
Let's see what happens next time a service fails.
Search engine-setup/host-deploy logs for 'tcp connections'.
Best regards,
>
>
> Thanks,
> Dafna
>
>
>
> On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
>
> On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>
>> this is the plugin info from steup log but I don't see anything more than
>> we have seen except a timeout.
>>
>> https://pastebin.com/QVtNRNWV
>>
>>
>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>
>> Didi, is there anywhere else I should look?
>
>
> Sadly, as already replied, not yet. Hopefully next time...
>
>>
>>
>>
>> On 11/29/2017 10:18 AM, Nir Soffer wrote:
>>
>> Do we have more info from Didi's debug plugin now?
>>
>> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> We have failed cq with ovirt-imageio failing to start on upgrade suite.
>>> I can still only see errors in the messages log.
>>>
>>> I'm writing the reported patch but I don't think it has anything to
do
>>> with this issue.
>>>
>>> Link and headline of suspected patches:
>>>
>>> restapi: Enable update to no default network provider of cluster -
>>> https://gerrit.ovirt.org/#/c/84814/
>>>
>>> Link to Job:
>>>
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
>>>
>>> Link to all logs:
>>>
>>>
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
>>>
>>>
>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
>>>
>>>
>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>>
>>> (Relevant) error snippet from the log:
>>>
>>> <error>
>>>
>>> From messages log
>>>
>>>
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Started oVirt Engine.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Reloading.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>> executable. Please remove executable permission bits. Proceeding anyway.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Starting oVirt Engine Data Warehouse...
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Started oVirt Engine Data Warehouse.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Reloading.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>> executable. Please remove executable permission bits. Proceeding anyway.
>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>> Starting oVirt ImageIO Proxy...
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85,
in
>>> <module>
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>> 21, in main
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: image_server.start(config)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45,
>>> in start
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py",
line 419,
>>> in __init__
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: self.server_bind()
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>> line 48, in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py",
line
>>> 108, in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py",
line 430,
>>> in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line
224, in
>>> meth
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>> status=1/FAILURE
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> Failed to start oVirt ImageIO Proxy.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> Unit ovirt-imageio-proxy.service entered failed state.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service failed.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> Starting oVirt ImageIO Proxy...
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85,
in
>>> <module>
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>> 21, in main
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: image_server.start(config)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45,
>>> in start
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py",
line 419,
>>> in __init__
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: self.server_bind()
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>> line 48, in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py",
line
>>> 108, in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py",
line 430,
>>> in server_bind
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line
224, in
>>> meth
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>> status=1/FAILURE
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> Failed to start oVirt ImageIO Proxy.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> Unit ovirt-imageio-proxy.service entered failed state.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service failed.
>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>
>>> </error>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
>
> --
> Didi
>
>
--
Didi
--
Didi
9:38 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
On 11/30/2017 03:03 PM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David
<didi(a)redhat.com> wrote:
> On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron(a)redhat.com> wrote:
>> we had a failure on 002_bootstrap.verify_add_hosts but the error is on
>> imageio
>>
>> I looked at the host log that Nir added and I can only see that the
>> address is in use which seems to be the same issue we have in initialize
>> engine.
>>
>>
>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifa...
>>
>> I cannot see anything in host-deploy.
>> Didi, would we be able to see anything here?
>
> Sorry, seems like my plugin is not enough. Will have a look.
Now merged an updated plugin, should hopefully pass changequeue soon.
Let's see what happens next time a service fails.
Search engine-setup/host-deploy logs for 'tcp connections'.
Best regards,
thank you :)
will update once we have another failure.
>>
>> Thanks,
>> Dafna
>>
>>
>>
>> On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
>>
>> On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>> this is the plugin info from steup log but I don't see anything more
than
>>> we have seen except a timeout.
>>>
>>> https://pastebin.com/QVtNRNWV
>>>
>>>
>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>>
>>> Didi, is there anywhere else I should look?
>>
>> Sadly, as already replied, not yet. Hopefully next time...
>>
>>>
>>>
>>> On 11/29/2017 10:18 AM, Nir Soffer wrote:
>>>
>>> Do we have more info from Didi's debug plugin now?
>>>
>>> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com> wrote:
>>>> Hi,
>>>>
>>>> We have failed cq with ovirt-imageio failing to start on upgrade suite.
>>>> I can still only see errors in the messages log.
>>>>
>>>> I'm writing the reported patch but I don't think it has anything
to do
>>>> with this issue.
>>>>
>>>> Link and headline of suspected patches:
>>>>
>>>> restapi: Enable update to no default network provider of cluster -
>>>> https://gerrit.ovirt.org/#/c/84814/
>>>>
>>>> Link to Job:
>>>>
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
>>>>
>>>> Link to all logs:
>>>>
>>>>
>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
>>>>
>>>>
>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe...
>>>>
>>>>
>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa...
>>>>
>>>> (Relevant) error snippet from the log:
>>>>
>>>> <error>
>>>>
>>>> From messages log
>>>>
>>>>
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Started oVirt Engine.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Reloading.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>>> executable. Please remove executable permission bits. Proceeding anyway.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Starting oVirt Engine Data Warehouse...
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Started oVirt Engine Data Warehouse.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Reloading.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>>> executable. Please remove executable permission bits. Proceeding anyway.
>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Starting oVirt ImageIO Proxy...
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line
85, in
>>>> <module>
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>> 21, in main
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: image_server.start(config)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>> in start
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419,
>>>> in __init__
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: self.server_bind()
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>> line 48, in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>> 108, in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430,
>>>> in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
line 224, in
>>>> meth
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>> status=1/FAILURE
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Failed to start oVirt ImageIO Proxy.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service failed.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Starting oVirt ImageIO Proxy...
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line
85, in
>>>> <module>
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>> 21, in main
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: image_server.start(config)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>> in start
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419,
>>>> in __init__
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: self.server_bind()
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>> line 48, in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>> 108, in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430,
>>>> in server_bind
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
line 224, in
>>>> meth
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>> status=1/FAILURE
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Failed to start oVirt ImageIO Proxy.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service failed.
>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>>
>>>> </error>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>> --
>> Didi
>>
>>
>
>
> --
> Didi
2579
days inactive
2580
days old
8 comments
3 participants
participants (3)
-
Dafna Ron -
Nir Soffer -
Yedidyah Bar David