[ OST Failure Report ] [ oVirt master ] [ 23-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
5:29 p.m.
This is a multi-part message in MIME format.
--------------F9CBFA00796816161239008C
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
We have a failing on test 001_initialize_engine.test_initialize_engine.
This is failing with error Failed to start service 'ovirt-imageio-proxy
**
*Link and headline ofto suspected patches: *
***
build: Make resulting RPMs architecture-specific -
https://gerrit.ovirt.org/#/c/84534/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055*
*
*
*Link to all logs:*
*
*
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
**
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/*
*
(Relevant) error snippet from the log:
<error>
*from lago log:
Failed to start service 'ovirt-imageio-proxy
messages logs:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting Session 8
of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
**
*</error>*
**
--------------F9CBFA00796816161239008C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We have a failing on test
001_initialize_engine.test_initialize_engine. <br>
</p>
<p>This is failing with error Failed to start service
'ovirt-imageio-proxy</p>
<p><br>
</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-e975-9a0d-3b01-8de053cdfef7">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
and headline ofto suspected patches:
</span></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-e975-9a0d-3b01-8de053cdfef7">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">build:
Make resulting RPMs architecture-specific - <a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/84534/">https://gerrit.ov...
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job:</span></p>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-e975-9a0d-3b01-8de053cdfef7"><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs:</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
</b></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-e975-9a0d-3b01-8de053cdfef7">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
<br>
</b>from lago log: <br>
</p>
<pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap:
break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style:
initial; text-decoration-color: initial;">Failed to start service
'ovirt-imageio-proxy
messages logs:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting Session 8
of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.</pre>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-e975-9a0d-3b01-8de053cdfef7">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b><br class="Apple-interchange-newline">
</p>
</body>
</html>
--------------F9CBFA00796816161239008C--
5:39 p.m.
Daniel/Nir?
On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com> wrote:
Hi,
We have a failing on test 001_initialize_engine.test_initialize_engine.
This is failing with error Failed to start service 'ovirt-imageio-proxy
*Link and headline ofto suspected patches: *
* build: Make resulting RPMs architecture-specific -
https://gerrit.ovirt.org/#/c/84534/ <https://gerrit.ovirt.org/#/c/84534/>
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
*Link to all logs:*
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4055/artifact/
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
* (Relevant) error snippet from the log: <error> *from lago log:
Failed to start service 'ovirt-imageio-proxy
messages logs:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting Session 8
of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
*</error>*
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
8:16 p.m.
This is a multi-part message in MIME format.
--------------D11D052340527DBBD99C6D52
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
there were two different patches reported as failing cq today with the
ovirt-imageio-proxy service failing to start.
Here is the latest failure:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4130/artifact
On 11/23/2017 03:39 PM, Allon Mureinik wrote:
Daniel/Nir?
On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com
<mailto:dron@redhat.com>> wrote:
Hi,
We have a failing on test
001_initialize_engine.test_initialize_engine.
This is failing with error Failed to start service
'ovirt-imageio-proxy
**
*Link and headline ofto suspected patches: *
***
build: Make resulting RPMs architecture-specific -
https://gerrit.ovirt.org/#/c/84534/
<https://gerrit.ovirt.org/#/c/84534/>
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
*
*
*Link to all logs:*
*
*
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
**
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
*
(Relevant) error snippet from the log:
<error>
*from lago log:
Failed to start service 'ovirt-imageio-proxy
messages logs:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting
Session 8 of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
**
*</error>*
**
_______________________________________________
Infra mailing list
Infra(a)ovirt.org <mailto:Infra@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/infra
<http://lists.ovirt.org/mailman/listinfo/infra>
--------------D11D052340527DBBD99C6D52
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl
eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp
dGUtcHJlZml4Ij50aGVyZSB3ZXJlIHR3byBkaWZmZXJlbnQgcGF0Y2hlcwogICAgICByZXBv
cnRlZCBhcyBmYWlsaW5nIGNxIHRvZGF5IHdpdGggdGhlIG92aXJ0LWltYWdlaW8tcHJveHkg
c2VydmljZQogICAgICBmYWlsaW5nIHRvIHN0YXJ0LiA8YnI+CiAgICAgIDxicj4KICAgICAg
SGVyZSBpcyB0aGUgbGF0ZXN0IGZhaWx1cmU6CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJl
ZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVy
X2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDEzMC9hcnRpZmFjdCI+aHR0cDovL2plbmtpbnMub3Zp
cnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MTMwL2FydGlm
YWN0PC9hPjxicj4KICAgICAgPGJyPgogICAgICA8YnI+CiAgICAgIDxicj4KICAgICAgT24g
MTEvMjMvMjAxNyAwMzozOSBQTSwgQWxsb24gTXVyZWluaWsgd3JvdGU6PGJyPgogICAgPC9k
aXY+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIgpjaXRlPSJtaWQ6Q0FEZ1hmM3ljSFdf
MHNDYmFOU0o2eVZlVEJTdVYtZFhoRTRXLTdlV083LXF1VFk0eS1nQG1haWwuZ21haWwuY29t
Ij4KICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZGVm
YXVsdCIKICAgICAgICAgIHN0eWxlPSJmb250LWZhbWlseTptb25vc3BhY2UsbW9ub3NwYWNl
Ij5EYW5pZWwvTmlyPzwvZGl2PgogICAgICA8L2Rpdj4KICAgICAgPGRpdiBjbGFzcz0iZ21h
aWxfZXh0cmEiPjxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1
LCBOb3YgMjMsIDIwMTcgYXQgNToyOSBQTSwgRGFmbmEKICAgICAgICAgIFJvbiA8c3BhbiBk
aXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iCiAgICAgICAg
ICAgICAgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+ZHJvbkByZWRo
YXQuY29tPC9hPiZndDs8L3NwYW4+CiAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICA8
YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAg
ICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6
MWV4Ij4KICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZG
Ij4KICAgICAgICAgICAgICA8cD5IaSwgPGJyPgogICAgICAgICAgICAgIDwvcD4KICAgICAg
ICAgICAgICA8cD5XZSBoYXZlIGEgZmFpbGluZyBvbiB0ZXN0IDAwMV9pbml0aWFsaXplX2Vu
Z2luZS50ZXN0Xzx3YnI+aW5pdGlhbGl6ZV9lbmdpbmUuCiAgICAgICAgICAgICAgICA8YnI+
CiAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgIDxwPlRoaXMgaXMgZmFpbGluZyB3
aXRoIGVycm9yIEZhaWxlZCB0byBzdGFydCBzZXJ2aWNlCiAgICAgICAgICAgICAgICAnb3Zp
cnQtaW1hZ2Vpby1wcm94eTwvcD4KICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAg
ICAgPC9wPgogICAgICAgICAgICAgIDxwPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwi
CmlkPSJtXy0xNjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2Ex
LWU5NzUtOWEwZC0zYjAxLThkZTA1M2NkZmVmNyI+CiAgICAgICAgICAgICAgICA8L2I+PC9w
PgogICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAgICAgc3R5bGU9Imxp
bmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxiCiAg
ICAgICAgICAgICAgICAgIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0xNjUz
MzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0z
YjAxLThkZTA1M2NkZmVmNyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFt
aWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtm
b250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0
ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFj
ZTpwcmUtd3JhcCI+TGluayBhbmQgaGVhZGxpbmUgb2Z0byBzdXNwZWN0ZWQgcGF0Y2hlczog
Cjwvc3Bhbj48L2I+PC9wPgogICAgICAgICAgICAgIDxiIHN0eWxlPSJmb250LXdlaWdodDpu
b3JtYWwiCmlkPSJtXy0xNjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4
NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2NkZmVmNyI+CiAgICAgICAgICAgICAgPC9i
PjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0xNjUzMzkwMDAwMDU2NDEy
MjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2Nk
ZmVmNyI+CiAgICAgICAgICAgICAgICA8cCBkaXI9Imx0ciIKICAgICAgICAgICAgICAgICAg
c3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTow
cHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xv
cjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAw
O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9u
Om5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPmJ1
aWxkOiBNYWtlIHJlc3VsdGluZyBSUE1zIGFyY2hpdGVjdHVyZS1zcGVjaWZpYyAtIDxhIGNs
YXNzPSJtXy0xNjUzMzkwMDAwMDU2NDEyMjMxbW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVm
PSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzg0NTM0LyIgdGFyZ2V0PSJfYmxhbmsi
IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy88
d2JyPjg0NTM0LzwvYT4KPC9zcGFuPjwvcD4KICAgICAgICAgICAgICAgIDxicj4KICAgICAg
ICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAgICAgICBzdHlsZT0ibGluZS1o
ZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFj
a2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpu
b3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNh
bC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+TGluayB0byBKb2I6PC9z
cGFuPjwvcD4KICAgICAgICAgICAgICAgIDxhIGNsYXNzPSJtXy0xNjUzMzkwMDAwMDU2NDEy
MjMxbW96LXR4dC1saW5rLWZyZWV0ZXh0IgpocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5v
cmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQwNTUiCiAgICAgICAg
ICAgICAgICAgIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6
Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVl
LTx3YnI+dGVzdGVyLzQwNTU8L2E+PC9iPgogICAgICAgICAgICAgIDxwPjxiIHN0eWxlPSJm
b250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0xNjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRl
cm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2NkZmVmNyI+PGJyPgog
ICAgICAgICAgICAgICAgPC9iPjwvcD4KICAgICAgICAgICAgICA8cCBkaXI9Imx0ciIKICAg
ICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21h
cmdpbi1ib3R0b206MHB0Ij48YgogICAgICAgICAgICAgICAgICBzdHlsZT0iZm9udC13ZWln
aHQ6bm9ybWFsIgppZD0ibV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJuYWwtZ3Vp
ZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91
bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFs
O2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxp
Z246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPkxpbmsgdG8gYWxsIGxvZ3M6PC9z
cGFuPjwvYj48L3A+CiAgICAgICAgICAgICAgPGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1h
bCIKaWQ9Im1fLTE2NTMzOTAwMDAwNTY0MTIyMzFkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3
YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3Ij4KICAgICAgICAgICAgICAgIDxwIGRp
cj0ibHRyIgogICAgICAgICAgICAgICAgICBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJn
aW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox
MXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0
cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJp
YW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGlu
ZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgPC9i
PgogICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAgICAgc3R5bGU9Imxp
bmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFu
IHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAw
O2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5
bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVy
dGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPjxhIGNsYXNzPSJt
Xy0xNjUzMzkwMDAwMDU2NDEyMjMxbW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRw
Oi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVz
dGVyLzQwNTUvYXJ0aWZhY3QvIiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0
cnVlIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+b3ZpcnQtbWFzdGVyX2No
YW5nZS1xdWV1ZS08d2JyPnRlc3Rlci80MDU1L2FydGlmYWN0LzwvYT48L3NwYW4+PC9wPgog
ICAgICAgICAgICAgIDxwPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0x
NjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEw
ZC0zYjAxLThkZTA1M2NkZmVmNyI+CiAgICAgICAgICAgICAgICA8L2I+PC9wPgogICAgICAg
ICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAgICAgc3R5bGU9ImxpbmUtaGVpZ2h0
OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxiCiAgICAgICAgICAg
ICAgICAgIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0xNjUzMzkwMDAwMDU2
NDEyMjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1
M2NkZmVmNyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFs
O2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdo
dDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29y
YXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3Jh
cCI+PGEgY2xhc3M9Im1fLTE2NTMzOTAwMDAwNTY0MTIyMzFtb3otdHh0LWxpbmstZnJlZXRl
eHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2No
YW5nZS1xdWV1ZS10ZXN0ZXIvNDA1NS9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvdXBn
cmFkZS1mcm9tLXJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRlLWZy
b20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUucHkv
bGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJfbG9n
L21lc3NhZ2VzLyp2aWV3Ki8iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRy
dWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmlydC1tYXN0ZXJfY2hh
bmdlLXF1ZXVlLTx3YnI+dGVzdGVyLzQwNTUvYXJ0aWZhY3QvZXhwb3J0ZWQtPHdicj5hcnRp
ZmFjdHMvdXBncmFkZS1mcm9tLTx3YnI+cmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF88
d2JyPmxvZ3MvdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXIvcG9zdC0w
MDFfPHdicj5pbml0aWFsaXplX2VuZ2luZS5weS9sYWdvLTx3YnI+dXBncmFkZS1mcm9tLXJl
bGVhc2Utc3VpdGUtPHdicj5tYXN0ZXItZW5naW5lL192YXJfbG9nLzx3YnI+bWVzc2FnZXMv
KnZpZXcqLzwvYT48L3NwYW4+PC9iPjwvcD4KICAgICAgICAgICAgICA8YiBzdHlsZT0iZm9u
dC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJu
YWwtZ3VpZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPgogICAgICAg
ICAgICAgICAgPHAgZGlyPSJsdHIiCiAgICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhl
aWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNr
Z3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5v
cm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2Fs
LWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD4KICAg
ICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAg
ICAgICAgICAgICBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJn
aW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5
OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250
LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0
LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpw
cmUtd3JhcCI+KFJlbGV2YW50KSBlcnJvciBzbmlwcGV0IGZyb20gdGhlIGxvZzogPC9zcGFu
PjwvcD4KICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAgICAgICBz
dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw
dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y
OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7
Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246
bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0
O2Vycm9yJmd0Ozwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAg
ICAgPC9iPmZyb20gbGFnbyBsb2c6IDxicj4KICAgICAgICAgICAgICA8cHJlIHN0eWxlPSJi
b3gtc2l6aW5nOmJvcmRlci1ib3g7d2hpdGUtc3BhY2U6cHJlLXdyYXA7d29yZC13cmFwOmJy
ZWFrLXdvcmQ7bWFyZ2luOjBweDtjb2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQtc2l6ZToxNHB4
O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudC1saWdhdHVyZXM6bm9ybWFsO2ZvbnQt
dmFyaWFudC1jYXBzOm5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6
bm9ybWFsO3RleHQtYWxpZ246c3RhcnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3Jt
Om5vbmU7d29yZC1zcGFjaW5nOjBweDt0ZXh0LWRlY29yYXRpb24tc3R5bGU6aW5pdGlhbDt0
ZXh0LWRlY29yYXRpb24tY29sb3I6aW5pdGlhbCI+RmFpbGVkIHRvIHN0YXJ0IHNlcnZpY2Ug
J292aXJ0LWltYWdlaW8tcHJveHkKCm1lc3NhZ2VzIGxvZ3M6IAoKTm92IDIzIDA3OjMwOjQ3
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5
c3RlbWQ6IFN0YXJ0aW5nIFNlc3Npb24gOCBvZiB1c2VyIHJvb3QuCk5vdiAyMyAwNzozMDo0
NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBUcmFjZWJhY2sgKG1vc3QgcmVjZW50IGNhbGwgbGFzdCk6
Ck5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUt
bWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9vdmly
dC1pbWFnZWlvLXByb3h5Ijx3YnI+LCBsaW5lIDg1LCBpbiAmbHQ7bW9kdWxlJmd0OwpOb3Yg
MjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rl
ci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogc3RhdHVzID0gaW1hZ2VfcHJveHkubWFp
bihhcmdzLCBjb25maWcpCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxl
YXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxl
ICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS08d2JyPnBhY2thZ2VzL292aXJ0X2ltYWdlaW9f
cHJveHkvPHdicj5pbWFnZV9wcm94eS5weSIsIGxpbmUgMjEsIGluIG1haW4KTm92IDIzIDA3
OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n
aW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGltYWdlX3NlcnZlci5zdGFydChjb25maWcpCk5v
diAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFz
dGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9weXRob24y
Ljcvc2l0ZS08d2JyPnBhY2thZ2VzL292aXJ0X2ltYWdlaW9fcHJveHkvPHdicj5zZXJ2ZXIu
cHkiLCBsaW5lIDQ1LCBpbiBzdGFydApOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94
eTogV1NHSVJlcXVlc3RIYW5kbGVyKQpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94
eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvPHdicj5Tb2NrZXRTZXJ2ZXIucHkiLCBs
aW5lIDQxOSwgaW4gX19pbml0X18KTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9t
LXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6
IHNlbGYuc2VydmVyX2JpbmQoKQpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20t
cmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTog
RmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvd3NnaXJlZi88d2JyPnNpbXBsZV9zZXJ2ZXIu
cHkiLCBsaW5lIDQ4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdy
YWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vp
by1wcm94eTogSFRUUFNlcnZlci5zZXJ2ZXJfYmluZChzZWxmKQpOb3YgMjMgMDc6MzA6NDcg
bGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3Zp
cnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvPHdicj5CYXNl
SFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6MzA6
NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg
b3ZpcnQtaW1hZ2Vpby1wcm94eTogU29ja2V0U2VydmVyLlRDUFNlcnZlci5zZXJ2ZXJfPHdi
cj5iaW5kKHNlbGYpCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl
LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIv
dXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPlNvY2tldFNlcnZlci5weSIsIGxpbmUgNDMwLCBp
biBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogc2VsZi5z
b2NrZXQuYmluZChzZWxmLnNlcnZlcl88d2JyPmFkZHJlc3MpCk5vdiAyMyAwNzozMDo0NyBs
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly
dC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zb2NrZXQuPHdi
cj5weSIsIGxpbmUgMjI0LCBpbiBtZXRoCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUt
ZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiByZXR1cm4gZ2V0YXR0cihzZWxmLl9zb2NrLG5hbWUpKCo8d2JyPmFyZ3MpCk5vdiAy
MyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy
LWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzb2NrZXQuZXJyb3I6IFtFcnJubyA5OF0g
QWRkcmVzcyBhbHJlYWR5IGluIHVzZQpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1h
Z2Vpby1wcm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVkLCBjb2RlPWV4aXRlZCwg
c3RhdHVzPTEvRkFJTFVSRQpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVs
ZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogRmFpbGVkIHRvIHN0YXJ0
IG9WaXJ0IEltYWdlSU8gUHJveHkuCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJv
bS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBVbml0IG92aXJ0
LWltYWdlaW8tcHJveHkuc2VydmljZSBlbnRlcmVkIGZhaWxlZCBzdGF0ZS4KTm92IDIzIDA3
OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n
aW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAy
MyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy
LWVuZ2luZSBzeXN0ZW1kOiBvdmlydC1pbWFnZWlvLXByb3h5LnNlcnZpY2UgaG9sZG9mZiB0
aW1lIG92ZXIsIHNjaGVkdWxpbmcgcmVzdGFydC4KTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFN0
YXJ0aW5nIG9WaXJ0IEltYWdlSU8gUHJveHkuLi4KTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdl
aW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92IDIzIDA3
OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n
aW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvYmluL292aXJ0LWltYWdlaW8t
cHJveHkiPHdicj4sIGxpbmUgODUsIGluICZsdDttb2R1bGUmZ3Q7Ck5vdiAyMyAwNzozMDo0
OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3MsIGNv
bmZpZykKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5z
dWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGli
L3B5dGhvbjIuNy9zaXRlLTx3YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS88d2Jy
PmltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjMgMDc6MzA6NDggbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQt
aW1hZ2Vpby1wcm94eTogaW1hZ2Vfc2VydmVyLnN0YXJ0KGNvbmZpZykKTm92IDIzIDA3OjMw
OjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5l
IG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLTx3
YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS88d2JyPnNlcnZlci5weSIsIGxpbmUg
NDUsIGluIHN0YXJ0Ck5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl
LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBXU0dJUmVx
dWVzdEhhbmRsZXIpCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl
LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIv
dXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPlNvY2tldFNlcnZlci5weSIsIGxpbmUgNDE5LCBp
biBfX2luaXRfXwpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08
d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogc2VsZi5zZXJ2
ZXJfYmluZCgpCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3
YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNy
L2xpYjY0L3B5dGhvbjIuNy93c2dpcmVmLzx3YnI+c2ltcGxlX3NlcnZlci5weSIsIGxpbmUg
NDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1y
ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBI
VFRQU2VydmVyLnNlcnZlcl9iaW5kKHNlbGYpCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3Jh
ZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlv
LXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPkJhc2VIVFRQU2VydmVy
LnB5IiwgbGluZSAxMDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVw
Z3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFn
ZWlvLXByb3h5OiBTb2NrZXRTZXJ2ZXIuVENQU2VydmVyLnNlcnZlcl88d2JyPmJpbmQoc2Vs
ZikKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0
ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliNjQv
cHl0aG9uMi43Lzx3YnI+U29ja2V0U2VydmVyLnB5IiwgbGluZSA0MzAsIGluIHNlcnZlcl9i
aW5kCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3Vp
dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tldC5iaW5k
KHNlbGYuc2VydmVyXzx3YnI+YWRkcmVzcykKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFk
ZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8t
cHJveHk6IEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NvY2tldC48d2JyPnB5IiwgbGlu
ZSAyMjQsIGluIG1ldGgKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh
c2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHJldHVy
biBnZXRhdHRyKHNlbGYuX3NvY2ssbmFtZSkoKjx3YnI+YXJncykKTm92IDIzIDA3OjMwOjQ4
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92
aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBBZGRyZXNzIGFs
cmVhZHkgaW4gdXNlCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl
LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBvdmlydC1pbWFnZWlvLXByb3h5
LnNlcnZpY2U6IG1haW4gcHJvY2VzcyBleGl0ZWQsIGNvZGU9ZXhpdGVkLCBzdGF0dXM9MS9G
QUlMVVJFCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+
c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBGYWlsZWQgdG8gc3RhcnQgb1ZpcnQgSW1h
Z2VJTyBQcm94eS4KTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut
PHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1w
cm94eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjMgMDc6MzA6NDggbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVt
ZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGZhaWxlZC4KTm92IDIzIDA3OjMwOjQ4
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5
c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBob2xkb2ZmIHRpbWUgb3Zlciwg
c2NoZWR1bGluZyByZXN0YXJ0LgpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20t
cmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogc3RhcnQgcmVxdWVz
dCByZXBlYXRlZCB0b28gcXVpY2tseSBmb3Igb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNl
Ck5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUt
bWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBGYWlsZWQgdG8gc3RhcnQgb1ZpcnQgSW1hZ2VJTyBQ
cm94eS4KTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1wcm94eS5z
ZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdy
YWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3Zp
cnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGZhaWxlZC48L3ByZT4KICAgICAgICAgICAgICA8
cD48YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMTY1MzM5MDAwMDA1NjQx
MjIzMWRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNj
ZGZlZjciPgogICAgICAgICAgICAgICAgPC9iPjwvcD4KICAgICAgICAgICAgICA8cCBkaXI9
Imx0ciIKICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10
b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48YgogICAgICAgICAgICAgICAgICBzdHlsZT0i
Zm9udC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50
ZXJuYWwtZ3VpZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPjxzcGFu
IHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAw
O2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5
bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVy
dGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPiZsdDsvZXJyb3Im
Z3Q7PC9zcGFuPjwvYj48L3A+CiAgICAgICAgICAgICAgPGIgc3R5bGU9ImZvbnQtd2VpZ2h0
Om5vcm1hbCIKaWQ9Im1fLTE2NTMzOTAwMDAwNTY0MTIyMzFkb2NzLWludGVybmFsLWd1aWQt
NTg1OWI3YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3Ij4KICAgICAgICAgICAgICA8
L2I+PGJyCiAgICAgICAgICAgICAgICBjbGFzcz0ibV8tMTY1MzM5MDAwMDA1NjQxMjIzMUFw
cGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAg
ICAgPGJyPgogICAgICAgICAgICBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX188d2Jy
Pl9fX19fX19fX19fX19fX19fPGJyPgogICAgICAgICAgICBJbmZyYSBtYWlsaW5nIGxpc3Q8
YnI+CiAgICAgICAgICAgIDxhIGhyZWY9Im1haWx0bzpJbmZyYUBvdmlydC5vcmciIG1vei1k
by1ub3Qtc2VuZD0idHJ1ZSI+SW5mcmFAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAg
PGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2luZnJh
IgogICAgICAgICAgICAgIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiIG1vei1k
by1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy88d2JyPm1haWxtYW4v
bGlzdGluZm8vaW5mcmE8L2E+PGJyPgogICAgICAgICAgICA8YnI+CiAgICAgICAgICA8L2Js
b2NrcXVvdGU+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJyPgogICAgICA8L2Rpdj4KICAg
IDwvYmxvY2txdW90ZT4KICAgIDxwPjxicj4KICAgIDwvcD4KICA8L2JvZHk+CjwvaHRtbD4K
--------------D11D052340527DBBD99C6D52--
4:53 p.m.
We still see this issue on the upgrade suite from latest release to master
[1].
I don't see any evidence in "/var/log/messages" [2] that
"ovirt-imageio-proxy" was started twice.
[1]
http://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-...
[2]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com> wrote:
there were two different patches reported as failing cq today with
the
ovirt-imageio-proxy service failing to start.
Here is the latest failure: http://jenkins.ovirt.org/job/
ovirt-master_change-queue-tester/4130/artifact
On 11/23/2017 03:39 PM, Allon Mureinik wrote:
Daniel/Nir?
On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com> wrote:
> Hi,
>
> We have a failing on test 001_initialize_engine.test_initialize_engine.
>
> This is failing with error Failed to start service 'ovirt-imageio-proxy
>
>
> *Link and headline ofto suspected patches: *
>
>
> * build: Make resulting RPMs architecture-specific -
> https://gerrit.ovirt.org/#/c/84534/ <https://gerrit.ovirt.org/#/c/84534/>
> Link to Job:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
>
>
> *Link to all logs:*
>
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/4055/artifact/
>
>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
>
>
>
> * (Relevant) error snippet from the log: <error> *from lago log:
>
> Failed to start service 'ovirt-imageio-proxy
>
> messages logs:
>
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting
Session 8 of user root.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
status = image_proxy.main(args, config)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
21, in main
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
in start
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
return getattr(self._sock,name)(*args)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>
> *</error>*
>
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--
*GAL bEN HAIM*
RHV DEVOPS
5:33 p.m.
On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim(a)redhat.com> wrote:
We still see this issue on the upgrade suite from latest release to
master
[1].
I don't see any evidence in "/var/log/messages" [2] that
"ovirt-imageio-proxy" was started twice.
Since it's not a registered port and a high port, could it be used by
something else (what are the odds though ?
Is it consistent?
Y.
[1] http://jenkins.ovirt.org/blue/rest/organizations/
jenkins/pipelines/ovirt-master_change-queue-tester/
runs/4153/nodes/123/steps/241/log/?start=0
[2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/
ovirt-master_change-queue-tester/4153/artifact/exported-
artifacts/upgrade-from-release-suit-master-el7/test_
logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-
upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com> wrote:
> there were two different patches reported as failing cq today with the
> ovirt-imageio-proxy service failing to start.
>
> Here is the latest failure: http://jenkins.ovirt.org/job/o
> virt-master_change-queue-tester/4130/artifact
>
>
>
>
> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
>
> Daniel/Nir?
>
> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com> wrote:
>
>> Hi,
>>
>> We have a failing on test 001_initialize_engine.test_initialize_engine.
>>
>> This is failing with error Failed to start service 'ovirt-imageio-proxy
>>
>>
>> *Link and headline ofto suspected patches: *
>>
>>
>> * build: Make resulting RPMs architecture-specific -
>> https://gerrit.ovirt.org/#/c/84534/ <https://gerrit.ovirt.org/#/c/84534/>
>> Link to Job:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
>>
>>
>> *Link to all logs:*
>>
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>> r/4055/artifact/
>>
>>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
>>
>>
>>
>> * (Relevant) error snippet from the log: <error> *from lago log:
>>
>> Failed to start service 'ovirt-imageio-proxy
>>
>> messages logs:
>>
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting
Session 8 of user root.
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting
oVirt ImageIO Proxy...
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start
request repeated too quickly for ovirt-imageio-proxy.service
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to
start oVirt ImageIO Proxy.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>
>> *</error>*
>>
>>
>> _______________________________________________
>> Infra mailing list
>> Infra(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
--
*GAL bEN HAIM*
RHV DEVOPS
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
7:10 p.m.
The failure is not consistent.
On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim(a)redhat.com> wrote:
> We still see this issue on the upgrade suite from latest release to
> master [1].
> I don't see any evidence in "/var/log/messages" [2] that
> "ovirt-imageio-proxy" was started twice.
>
Since it's not a registered port and a high port, could it be used by
something else (what are the odds though ?
Is it consistent?
Y.
>
> [1] http://jenkins.ovirt.org/blue/rest/organizations/jenkins
> /pipelines/ovirt-master_change-queue-tester/runs/4153/
> nodes/123/steps/241/log/?start=0
>
> [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovir
> t-master_change-queue-tester/4153/artifact/exported-artifac
> ts/upgrade-from-release-suit-master-el7/test_logs/upgrade-
> from-release-suite-master/post-001_initialize_engine.py/
> lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>
> On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com> wrote:
>
>> there were two different patches reported as failing cq today with the
>> ovirt-imageio-proxy service failing to start.
>>
>> Here is the latest failure: http://jenkins.ovirt.org/job/o
>> virt-master_change-queue-tester/4130/artifact
>>
>>
>>
>>
>> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
>>
>> Daniel/Nir?
>>
>> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>
>>> Hi,
>>>
>>> We have a failing on test 001_initialize_engine.test_initialize_engine.
>>>
>>>
>>> This is failing with error Failed to start service 'ovirt-imageio-proxy
>>>
>>>
>>> *Link and headline ofto suspected patches: *
>>>
>>>
>>> * build: Make resulting RPMs architecture-specific -
>>> https://gerrit.ovirt.org/#/c/84534/
<https://gerrit.ovirt.org/#/c/84534/>
>>> Link to Job:
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
>>>
>>>
>>> *Link to all logs:*
>>>
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>>> r/4055/artifact/
>>>
>>>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
>>>
>>>
>>>
>>> * (Relevant) error snippet from the log: <error> *from lago log:
>>>
>>> Failed to start service 'ovirt-imageio-proxy
>>>
>>> messages logs:
>>>
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Starting Session 8 of user root.
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed
to start oVirt ImageIO Proxy.
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt ImageIO Proxy...
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed
to start oVirt ImageIO Proxy.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start
request repeated too quickly for ovirt-imageio-proxy.service
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed
to start oVirt ImageIO Proxy.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>>
>>> *</error>*
>>>
>>>
>>> _______________________________________________
>>> Infra mailing list
>>> Infra(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>
>>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
> --
> *GAL bEN HAIM*
> RHV DEVOPS
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
--
*GAL bEN HAIM*
RHV DEVOPS
7:24 p.m.
I think we need to check and report which process is listening on a port
when starting a server on that port fail.
Didi, do you think we can integrate this in the deploy code, or this
should be implemented in each server?
Maybe when deployment fails, the deploy code can report all the
listening sockets and the processes bound to these sockets?
Nir
On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim(a)redhat.com> wrote:
The failure is not consistent.
On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>
> On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim(a)redhat.com>
> wrote:
>
>> We still see this issue on the upgrade suite from latest release to
>> master [1].
>> I don't see any evidence in "/var/log/messages" [2] that
>> "ovirt-imageio-proxy" was started twice.
>>
>
> Since it's not a registered port and a high port, could it be used by
> something else (what are the odds though ?
> Is it consistent?
> Y.
>
>
>>
>> [1]
>>
http://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-...
>>
>> [2]
>>
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
>>
>>
>> On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>
>>> there were two different patches reported as failing cq today with the
>>> ovirt-imageio-proxy service failing to start.
>>>
>>> Here is the latest failure:
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4130/artifact
>>>
>>>
>>>
>>>
>>> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
>>>
>>> Daniel/Nir?
>>>
>>> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have a failing on test
>>>> 001_initialize_engine.test_initialize_engine.
>>>>
>>>> This is failing with error Failed to start service
'ovirt-imageio-proxy
>>>>
>>>>
>>>> *Link and headline ofto suspected patches: *
>>>>
>>>>
>>>> * build: Make resulting RPMs architecture-specific -
>>>> https://gerrit.ovirt.org/#/c/84534/
<https://gerrit.ovirt.org/#/c/84534/>
>>>> Link to Job:
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
>>>>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
>>>>
>>>>
>>>> *Link to all logs:*
>>>>
>>>>
>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
>>>>
>>>>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>>>
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
>>>>
>>>>
>>>>
>>>> * (Relevant) error snippet from the log: <error> *from lago log:
>>>>
>>>> Failed to start service 'ovirt-imageio-proxy
>>>>
>>>> messages logs:
>>>>
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Starting Session 8 of user root.
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Failed to start oVirt ImageIO Proxy.
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Unit ovirt-imageio-proxy.service entered failed state.
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
Starting oVirt ImageIO Proxy...
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
Failed to start oVirt ImageIO Proxy.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
Unit ovirt-imageio-proxy.service entered failed state.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
start request repeated too quickly for ovirt-imageio-proxy.service
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
Failed to start oVirt ImageIO Proxy.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
Unit ovirt-imageio-proxy.service entered failed state.
>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
>>>>
>>>> *</error>*
>>>>
>>>>
>>>> _______________________________________________
>>>> Infra mailing list
>>>> Infra(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>>
>> --
>> *GAL bEN HAIM*
>> RHV DEVOPS
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
--
*GAL bEN HAIM*
RHV DEVOPS
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:38 a.m.
On Sun, Nov 26, 2017 at 7:24 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
I think we need to check and report which process is listening on a
port
when starting a server on that port fail.
How do you know that a server was "started on that port", and that
if failed specifically because it failed to bind?
There is no standardized (Unix) way to mark that a service wants to
listen on a specific port, or that it failed because a specific port
was bound by some other process.
There are various classical *inetd* daemons, and modern systemd.socket,
that listen *instead* of some service. Then they can manage the port
resources and perhaps do something intelligent about them.
Didi, do you think we can integrate this in the deploy code, or this
should be implemented in each server?
It should be quite easy to patch otopi's services.state to run something
if start fails, e.g. 'ss -anp' or whatever you want.
It should even be not-too-hard to do this in a self-contained plugin,
so can be part of otopi-debug-plugins.
If we decide that something needs to be implemented by each server,
perhaps "something" should be to be controlled by a systemd.socket unit.
Didn't try, though, to see what this actually buys us.
Maybe when deployment fails, the deploy code can report all the
listening sockets and the processes bound to these sockets?
Pushed now:
https://gerrit.ovirt.org/84699 core: Name TRANSACTION_INIT
https://gerrit.ovirt.org/84700 plugins: debug: Add debug_failure
https://gerrit.ovirt.org/84701 automation: Test failure
Will merge soon, if all goes well.
Feel free to open BZ for other things discussed above, if relevant.
Nir
On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim(a)redhat.com> wrote:
>
> The failure is not consistent.
>
> On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>
>>
>>
>> On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim(a)redhat.com>
>> wrote:
>>>
>>> We still see this issue on the upgrade suite from latest release to
>>> master [1].
>>> I don't see any evidence in "/var/log/messages" [2] that
>>> "ovirt-imageio-proxy" was started twice.
>>
>>
>> Since it's not a registered port and a high port, could it be used by
>> something else (what are the odds though ?
>> Is it consistent?
>> Y.
>>
>>>
>>>
>>> [1]
>>>
http://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-...
>>>
>>> [2]
>>>
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
>>>
>>> On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>>>
>>>> there were two different patches reported as failing cq today with the
>>>> ovirt-imageio-proxy service failing to start.
>>>>
>>>> Here is the latest failure:
>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4130/artifact
>>>>
>>>>
>>>>
>>>>
>>>> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
>>>>
>>>> Daniel/Nir?
>>>>
>>>> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com>
wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have a failing on test
>>>>> 001_initialize_engine.test_initialize_engine.
>>>>>
>>>>> This is failing with error Failed to start service
>>>>> 'ovirt-imageio-proxy
>>>>>
>>>>>
>>>>> Link and headline ofto suspected patches:
>>>>>
>>>>> build: Make resulting RPMs architecture-specific -
>>>>> https://gerrit.ovirt.org/#/c/84534/
>>>>>
>>>>>
>>>>> Link to Job:
>>>>>
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
>>>>>
>>>>>
>>>>> Link to all logs:
>>>>>
>>>>>
>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
>>>>>
>>>>>
>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
>>>>>
>>>>>
>>>>> (Relevant) error snippet from the log:
>>>>>
>>>>> <error>
>>>>>
>>>>>
>>>>> from lago log:
>>>>>
>>>>> Failed to start service 'ovirt-imageio-proxy
>>>>>
>>>>> messages logs:
>>>>>
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Starting Session 8 of user root.
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy",
line 85, in
>>>>> <module>
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>>> 21, in main
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>>> in start
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419,
>>>>> in __init__
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>>> line 48, in server_bind
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>>> 108, in server_bind
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430,
>>>>> in server_bind
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
line 224, in
>>>>> meth
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>> status=1/FAILURE
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Failed to start oVirt ImageIO Proxy.
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service failed.
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Starting oVirt ImageIO Proxy...
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy",
line 85, in
>>>>> <module>
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>>> 21, in main
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>>> in start
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419,
>>>>> in __init__
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>>> line 48, in server_bind
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>>> 108, in server_bind
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430,
>>>>> in server_bind
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py",
line 224, in
>>>>> meth
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>> status=1/FAILURE
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Failed to start oVirt ImageIO Proxy.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service failed.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> start request repeated too quickly for ovirt-imageio-proxy.service
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Failed to start oVirt ImageIO Proxy.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>> ovirt-imageio-proxy.service failed.
>>>>>
>>>>> </error>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Infra mailing list
>>>>> Infra(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>>
>>>
>>>
>>> --
>>> GAL bEN HAIM
>>> RHV DEVOPS
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
>
> --
> GAL bEN HAIM
> RHV DEVOPS
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
--
Didi
2:57 p.m.
On Mon, Nov 27, 2017 at 10:38 AM, Yedidyah Bar David <didi(a)redhat.com>
wrote:
On Sun, Nov 26, 2017 at 7:24 PM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> I think we need to check and report which process is listening on a port
> when starting a server on that port fail.
How do you know that a server was "started on that port", and that
if failed specifically because it failed to bind?
There is no standardized (Unix) way to mark that a service wants to
listen on a specific port, or that it failed because a specific port
was bound by some other process.
There are various classical *inetd* daemons, and modern systemd.socket,
that listen *instead* of some service. Then they can manage the port
resources and perhaps do something intelligent about them.
>
> Didi, do you think we can integrate this in the deploy code, or this
> should be implemented in each server?
It should be quite easy to patch otopi's services.state to run something
if start fails, e.g. 'ss -anp' or whatever you want.
It should even be not-too-hard to do this in a self-contained plugin,
so can be part of otopi-debug-plugins.
If we decide that something needs to be implemented by each server,
perhaps "something" should be to be controlled by a systemd.socket unit.
Didn't try, though, to see what this actually buys us.
>
> Maybe when deployment fails, the deploy code can report all the
> listening sockets and the processes bound to these sockets?
Pushed now:
https://gerrit.ovirt.org/84699 core: Name TRANSACTION_INIT
https://gerrit.ovirt.org/84700 plugins: debug: Add debug_failure
https://gerrit.ovirt.org/84701 automation: Test failure
Will merge soon, if all goes well.
Merged them.
Pushed to OST:
https://gerrit.ovirt.org/84710
Dafna - thanks for opening the bug on ovirt-imageio, but I am not
sure anyone can do much about it without more info, such as might
be provided by above patches. When I suggested below to open BZ
I meant on otopi or host-deploy to provide more debugging info,
not for imageio - obviously no harm in opening it, and it's good
to have it even if only for reference.
Feel free to open BZ for other things discussed above, if relevant.
>
> Nir
>
> On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim(a)redhat.com>
wrote:
>>
>> The failure is not consistent.
>>
>> On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>>
>>>
>>>
>>> On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim(a)redhat.com>
>>> wrote:
>>>>
>>>> We still see this issue on the upgrade suite from latest release to
>>>> master [1].
>>>> I don't see any evidence in "/var/log/messages" [2] that
>>>> "ovirt-imageio-proxy" was started twice.
>>>
>>>
>>> Since it's not a registered port and a high port, could it be used by
>>> something else (what are the odds though ?
>>> Is it consistent?
>>> Y.
>>>
>>>>
>>>>
>>>> [1]
>>>> http://jenkins.ovirt.org/blue/rest/organizations/jenkins/
pipelines/ovirt-master_change-queue-tester/runs/4153/nodes/
123/steps/241/log/?start=0
>>>>
>>>> [2]
>>>> http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/
ovirt-master_change-queue-tester/4153/artifact/exported-
artifacts/upgrade-from-release-suit-master-el7/test_
logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-
upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>>>
>>>> On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com>
wrote:
>>>>>
>>>>> there were two different patches reported as failing cq today with
the
>>>>> ovirt-imageio-proxy service failing to start.
>>>>>
>>>>> Here is the latest failure:
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4130/artifact
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
>>>>>
>>>>> Daniel/Nir?
>>>>>
>>>>> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron(a)redhat.com>
wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have a failing on test
>>>>>> 001_initialize_engine.test_initialize_engine.
>>>>>>
>>>>>> This is failing with error Failed to start service
>>>>>> 'ovirt-imageio-proxy
>>>>>>
>>>>>>
>>>>>> Link and headline ofto suspected patches:
>>>>>>
>>>>>> build: Make resulting RPMs architecture-specific -
>>>>>> https://gerrit.ovirt.org/#/c/84534/
>>>>>>
>>>>>>
>>>>>> Link to Job:
>>>>>>
>>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
>>>>>>
>>>>>>
>>>>>> Link to all logs:
>>>>>>
>>>>>>
>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4055/artifact/
>>>>>>
>>>>>>
>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/4055/artifact/exported-artifacts/upgrade-from-
release-suit-master-el7/test_logs/upgrade-from-release-
suite-master/post-001_initialize_engine.py/lago-
upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>>>>>
>>>>>>
>>>>>> (Relevant) error snippet from the log:
>>>>>>
>>>>>> <error>
>>>>>>
>>>>>>
>>>>>> from lago log:
>>>>>>
>>>>>> Failed to start service 'ovirt-imageio-proxy
>>>>>>
>>>>>> messages logs:
>>>>>>
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Starting Session 8 of user root.
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85,
in
>>>>>> <module>
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line
>>>>>> 21, in main
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45,
>>>>>> in start
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
line 419,
>>>>>> in __init__
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/
simple_server.py",
>>>>>> line 48, in server_bind
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py",
line
>>>>>> 108, in server_bind
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
line 430,
>>>>>> in server_bind
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line
224, in
>>>>>> meth
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in
use
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>>> status=1/FAILURE
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Failed to start oVirt ImageIO Proxy.
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service failed.
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling
restart.
>>>>>> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Starting oVirt ImageIO Proxy...
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85,
in
>>>>>> <module>
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
line
>>>>>> 21, in main
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
line 45,
>>>>>> in start
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
line 419,
>>>>>> in __init__
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/
simple_server.py",
>>>>>> line 48, in server_bind
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py",
line
>>>>>> 108, in server_bind
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
line 430,
>>>>>> in server_bind
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line
224, in
>>>>>> meth
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
>>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in
use
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>>> status=1/FAILURE
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Failed to start oVirt ImageIO Proxy.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service failed.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling
restart.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> start request repeated too quickly for
ovirt-imageio-proxy.service
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Failed to start oVirt ImageIO Proxy.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>>> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd:
>>>>>> ovirt-imageio-proxy.service failed.
>>>>>>
>>>>>> </error>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Infra mailing list
>>>>>> Infra(a)ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> GAL bEN HAIM
>>>> RHV DEVOPS
>>>>
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>>
>>
>>
>>
>> --
>> GAL bEN HAIM
>> RHV DEVOPS
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
--
Didi
--
Didi
5:37 p.m.
Didi - we just saw a report of a similar failure on engine-4.1's OST.
COuld you please backport these patches there too?
On Mon, Nov 27, 2017 at 2:57 PM, Yedidyah Bar David <didi(a)redhat.com> wrote:
On Mon, Nov 27, 2017 at 10:38 AM, Yedidyah Bar David
<didi(a)redhat.com>
wrote:
> On Sun, Nov 26, 2017 at 7:24 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > I think we need to check and report which process is listening on a
> port
> > when starting a server on that port fail.
>
> How do you know that a server was "started on that port", and that
> if failed specifically because it failed to bind?
>
> There is no standardized (Unix) way to mark that a service wants to
> listen on a specific port, or that it failed because a specific port
> was bound by some other process.
>
> There are various classical *inetd* daemons, and modern systemd.socket,
> that listen *instead* of some service. Then they can manage the port
> resources and perhaps do something intelligent about them.
>
> >
> > Didi, do you think we can integrate this in the deploy code, or this
> > should be implemented in each server?
>
> It should be quite easy to patch otopi's services.state to run something
> if start fails, e.g. 'ss -anp' or whatever you want.
>
> It should even be not-too-hard to do this in a self-contained plugin,
> so can be part of otopi-debug-plugins.
>
> If we decide that something needs to be implemented by each server,
> perhaps "something" should be to be controlled by a systemd.socket unit.
> Didn't try, though, to see what this actually buys us.
>
> >
> > Maybe when deployment fails, the deploy code can report all the
> > listening sockets and the processes bound to these sockets?
>
> Pushed now:
>
> https://gerrit.ovirt.org/84699 core: Name TRANSACTION_INIT
> https://gerrit.ovirt.org/84700 plugins: debug: Add debug_failure
> https://gerrit.ovirt.org/84701 automation: Test failure
>
> Will merge soon, if all goes well.
>
Merged them.
Pushed to OST:
https://gerrit.ovirt.org/84710
Dafna - thanks for opening the bug on ovirt-imageio, but I am not
sure anyone can do much about it without more info, such as might
be provided by above patches. When I suggested below to open BZ
I meant on otopi or host-deploy to provide more debugging info,
not for imageio - obviously no harm in opening it, and it's good
to have it even if only for reference.
>
> Feel free to open BZ for other things discussed above, if relevant.
>
> >
> > Nir
> >
> > On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim(a)redhat.com>
> wrote:
> >>
> >> The failure is not consistent.
> >>
> >> On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
> >>>
> >>>
> >>>
> >>> On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim
<gbenhaim(a)redhat.com>
> >>> wrote:
> >>>>
> >>>> We still see this issue on the upgrade suite from latest release to
> >>>> master [1].
> >>>> I don't see any evidence in "/var/log/messages" [2]
that
> >>>> "ovirt-imageio-proxy" was started twice.
> >>>
> >>>
> >>> Since it's not a registered port and a high port, could it be used
by
> >>> something else (what are the odds though ?
> >>> Is it consistent?
> >>> Y.
> >>>
> >>>>
> >>>>
> >>>> [1]
> >>>> http://jenkins.ovirt.org/blue/rest/organizations/jenkins/pip
> elines/ovirt-master_change-queue-tester/runs/4153/nodes/123/
> steps/241/log/?start=0
> >>>>
> >>>> [2]
> >>>> http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovir
> t-master_change-queue-tester/4153/artifact/exported-artifac
> ts/upgrade-from-release-suit-master-el7/test_logs/upgrade-
> from-release-suite-master/post-001_initialize_engine.py/
> lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
> >>>>
> >>>> On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron(a)redhat.com>
wrote:
> >>>>>
> >>>>> there were two different patches reported as failing cq today
with
> the
> >>>>> ovirt-imageio-proxy service failing to start.
> >>>>>
> >>>>> Here is the latest failure:
> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/4130/artifact
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 11/23/2017 03:39 PM, Allon Mureinik wrote:
> >>>>>
> >>>>> Daniel/Nir?
> >>>>>
> >>>>> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron
<dron(a)redhat.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> We have a failing on test
> >>>>>> 001_initialize_engine.test_initialize_engine.
> >>>>>>
> >>>>>> This is failing with error Failed to start service
> >>>>>> 'ovirt-imageio-proxy
> >>>>>>
> >>>>>>
> >>>>>> Link and headline ofto suspected patches:
> >>>>>>
> >>>>>> build: Make resulting RPMs architecture-specific -
> >>>>>> https://gerrit.ovirt.org/#/c/84534/
> >>>>>>
> >>>>>>
> >>>>>> Link to Job:
> >>>>>>
> >>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
> >>>>>>
> >>>>>>
> >>>>>> Link to all logs:
> >>>>>>
> >>>>>>
> >>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/4055/artifact/
> >>>>>>
> >>>>>>
> >>>>>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/4055/artifact/exported-artifacts/upgrade-from-release-
> suit-master-el7/test_logs/upgrade-from-release-suite-
> master/post-001_initialize_engine.py/lago-upgrade-from-
> release-suite-master-engine/_var_log/messages/*view*/
> >>>>>>
> >>>>>>
> >>>>>> (Relevant) error snippet from the log:
> >>>>>>
> >>>>>> <error>
> >>>>>>
> >>>>>>
> >>>>>> from lago log:
> >>>>>>
> >>>>>> Failed to start service 'ovirt-imageio-proxy
> >>>>>>
> >>>>>> messages logs:
> >>>>>>
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Starting Session 8 of user root.
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line
> 85, in
> >>>>>> <module>
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: status = image_proxy.main(args,
config)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
> >>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
> line
> >>>>>> 21, in main
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: image_server.start(config)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
> >>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
> line 45,
> >>>>>> in start
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
> line 419,
> >>>>>> in __init__
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: self.server_bind()
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/
> simple_server.py",
> >>>>>> line 48, in server_bind
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py",
> line
> >>>>>> 108, in server_bind
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
> line 430,
> >>>>>> in server_bind
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line
> 224, in
> >>>>>> meth
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address
already in
> use
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service: main process exited,
code=exited,
> >>>>>> status=1/FAILURE
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Failed to start oVirt ImageIO Proxy.
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Unit ovirt-imageio-proxy.service entered failed state.
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service failed.
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling
restart.
> >>>>>> Nov 23 07:30:47
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Starting oVirt ImageIO Proxy...
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line
> 85, in
> >>>>>> <module>
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: status = image_proxy.main(args,
config)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
> >>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py",
> line
> >>>>>> 21, in main
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: image_server.start(config)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
> >>>>>>
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py",
> line 45,
> >>>>>> in start
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
> line 419,
> >>>>>> in __init__
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: self.server_bind()
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/
> simple_server.py",
> >>>>>> line 48, in server_bind
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py",
> line
> >>>>>> 108, in server_bind
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py",
> line 430,
> >>>>>> in server_bind
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line
> 224, in
> >>>>>> meth
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> >>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address
already in
> use
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service: main process exited,
code=exited,
> >>>>>> status=1/FAILURE
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Failed to start oVirt ImageIO Proxy.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Unit ovirt-imageio-proxy.service entered failed state.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service failed.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling
restart.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> start request repeated too quickly for
ovirt-imageio-proxy.service
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Failed to start oVirt ImageIO Proxy.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> Unit ovirt-imageio-proxy.service entered failed state.
> >>>>>> Nov 23 07:30:48
lago-upgrade-from-release-suite-master-engine
> systemd:
> >>>>>> ovirt-imageio-proxy.service failed.
> >>>>>>
> >>>>>> </error>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Infra mailing list
> >>>>>> Infra(a)ovirt.org
> >>>>>> http://lists.ovirt.org/mailman/listinfo/infra
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Devel mailing list
> >>>>> Devel(a)ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/devel
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> GAL bEN HAIM
> >>>> RHV DEVOPS
> >>>>
> >>>> _______________________________________________
> >>>> Devel mailing list
> >>>> Devel(a)ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/devel
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> GAL bEN HAIM
> >> RHV DEVOPS
> >> _______________________________________________
> >> Devel mailing list
> >> Devel(a)ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
> --
> Didi
>
--
Didi
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2:11 p.m.
This is a multi-part message in MIME format.
--------------510D2C3E2A3E3638E7FA4B2B
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
I opened a bug for this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1517764
On 11/26/2017 05:24 PM, Nir Soffer wrote:
I think we need to check and report which process is listening on a
port
when starting a server on that port fail.
Didi, do you think we can integrate this in the deploy code, or this
should be implemented in each server?
Maybe when deployment fails, the deploy code can report all the
listening sockets and the processes bound to these sockets?
Nir
On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim(a)redhat.com
<mailto:gbenhaim@redhat.com>> wrote:
The failure is not consistent.
On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul(a)redhat.com
<mailto:ykaul@redhat.com>> wrote:
On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim
<gbenhaim(a)redhat.com <mailto:gbenhaim@redhat.com>> wrote:
We still see this issue on the upgrade suite from latest
release to master [1].
I don't see any evidence in "/var/log/messages" [2] that
"ovirt-imageio-proxy" was started twice.
Since it's not a registered port and a high port, could it be
used by something else (what are the odds though ?
Is it consistent?
Y.
[1]
http://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-...
[2]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron
<dron(a)redhat.com <mailto:dron@redhat.com>> wrote:
there were two different patches reported as failing
cq today with the ovirt-imageio-proxy service failing
to start.
Here is the latest failure:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4130/artifact
On 11/23/2017 03:39 PM, Allon Mureinik wrote:
> Daniel/Nir?
>
> On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron
> <dron(a)redhat.com <mailto:dron@redhat.com>> wrote:
>
> Hi,
>
> We have a failing on test
> 001_initialize_engine.test_initialize_engine.
>
> This is failing with error Failed to start
> service 'ovirt-imageio-proxy
>
>
> **
>
> *Link and headline ofto suspected patches: *
>
> ***
>
> build: Make resulting RPMs architecture-specific
> - https://gerrit.ovirt.org/#/c/84534/
>
>
> Link to Job:
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055*
>
>
> *
> *
>
> *Link to all logs:*
>
> *
>
> *
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
>
> **
>
>
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/*
>
> *
>
>
> (Relevant) error snippet from the log:
>
> <error>
>
>
> *from lago log:
>
> Failed to start service 'ovirt-imageio-proxy
>
> messages logs:
>
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: Starting Session 8 of user root.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: Failed to start oVirt ImageIO Proxy.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: Unit ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service failed.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
systemd: Starting oVirt ImageIO Proxy...
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: Traceback (most recent call last):
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
<module>
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: status = image_proxy.main(args, config)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: image_server.start(config)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: WSGIRequestHandler)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in
__init__
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.server_bind()
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line
48, in server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: HTTPServer.server_bind(self)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in
server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in
server_bind
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: self.socket.bind(self.server_address)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: Failed to start oVirt ImageIO Proxy.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: Unit ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service failed.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: start request repeated too quickly for ovirt-imageio-proxy.service
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: Failed to start oVirt ImageIO Proxy.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: Unit ovirt-imageio-proxy.service entered failed state.
> Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine
systemd: ovirt-imageio-proxy.service failed.
>
> **
>
> *</error>*
>
> **
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org <mailto:Infra@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel
--
*GAL bEN HAIM*
RHV DEVOPS
_______________________________________________
Devel mailing list
Devel(a)ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel
--
*GAL bEN HAIM*
RHV DEVOPS
_______________________________________________
Devel mailing list
Devel(a)ovirt.org <mailto:Devel@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--------------510D2C3E2A3E3638E7FA4B2B
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl
eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp
dGUtcHJlZml4Ij5JIG9wZW5lZCBhIGJ1ZyBmb3IgdGhpcyBpc3N1ZTogPGJyPgogICAgICA8
YnI+CiAgICAgIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHBz
Oi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTUxNzc2NCI+aHR0cHM6
Ly9idWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD0xNTE3NzY0PC9hPjxicj4K
ICAgICAgPGJyPgogICAgICA8YnI+CiAgICAgIE9uIDExLzI2LzIwMTcgMDU6MjQgUE0sIE5p
ciBTb2ZmZXIgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJj
aXRlIgpjaXRlPSJtaWQ6Q0FNUmJ5eXNVQjYyT251aSszTXRTRGNKUkFtMlNvSE5kX2pwMz1Z
eTloV2dabkNBVHVnQG1haWwuZ21haWwuY29tIj4KICAgICAgPGRpdiBkaXI9Imx0ciI+SSB0
aGluayB3ZSBuZWVkIHRvIGNoZWNrIGFuZCByZXBvcnQgd2hpY2jCoCBwcm9jZXNzCiAgICAg
ICAgaXMgbGlzdGVuaW5nIG9uIGEgcG9ydAogICAgICAgIDxkaXY+d2hlbiBzdGFydGluZyBh
IHNlcnZlciBvbiB0aGF0IHBvcnQgZmFpbC4KICAgICAgICAgIDxkaXY+CiAgICAgICAgICAg
IDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgPGRpdj5EaWRpLCBk
byB5b3UgdGhpbmsgd2UgY2FuIGludGVncmF0ZSB0aGlzIGluIHRoZSBkZXBsb3kKICAgICAg
ICAgICAgICBjb2RlLCBvciB0aGlzPC9kaXY+CiAgICAgICAgICAgIDxkaXY+c2hvdWxkIGJl
IGltcGxlbWVudGVkIGluIGVhY2ggc2VydmVyPzwvZGl2PgogICAgICAgICAgICA8ZGl2Pjxi
cj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXY+TWF5YmUgd2hlbiBkZXBs
b3ltZW50IGZhaWxzLCB0aGUgZGVwbG95IGNvZGUgY2FuIHJlcG9ydAogICAgICAgICAgICAg
IGFsbCB0aGU8L2Rpdj4KICAgICAgICAgICAgPGRpdj5saXN0ZW5pbmcgc29ja2V0cyBhbmQg
dGhlIHByb2Nlc3NlcyBib3VuZCB0byB0aGVzZQogICAgICAgICAgICAgIHNvY2tldHM/PC9k
aXY+CiAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAg
ICAgPGRpdj5OaXI8L2Rpdj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgICA8ZGl2IGNs
YXNzPSJnbWFpbF9xdW90ZSI+CiAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+T24gU3Vu
LCBOb3YgMjYsIDIwMTcgYXQgNzoxMSBQTSBHYWwgQmVuCiAgICAgICAgICAgICAgICBIYWlt
ICZsdDs8YSBocmVmPSJtYWlsdG86Z2JlbmhhaW1AcmVkaGF0LmNvbSIKICAgICAgICAgICAg
ICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5nYmVuaGFpbUByZWRoYXQuY29tPC9hPiZn
dDsKICAgICAgICAgICAgICAgIHdyb3RlOjxicj4KICAgICAgICAgICAgICA8L2Rpdj4KICAg
ICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJn
aW46MCAwIDAKICAgICAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29s
aWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj5U
aGUgZmFpbHVyZSBpcyBub3QgY29uc2lzdGVudC48L2Rpdj4KICAgICAgICAgICAgICAgIDxk
aXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+CiAgICAgICAgICAgICAgICAgIDxkaXYgY2xh
c3M9ImdtYWlsX3F1b3RlIj5PbiBTdW4sIE5vdiAyNiwgMjAxNyBhdCA1OjMzCiAgICAgICAg
ICAgICAgICAgICAgUE0sIFlhbml2IEthdWwgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YQogICAg
ICAgICAgICAgICAgICAgICAgICBocmVmPSJtYWlsdG86eWthdWxAcmVkaGF0LmNvbSIgdGFy
Z2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0i
dHJ1ZSI+eWthdWxAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgICAgICAg
ICAgIHdyb3RlOjxicj4KICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0i
Z21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAgICAgICAg
IC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAg
ICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj48YnI+CiAgICAgICAgICAgICAg
ICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+CiAgICAgICAgICAgICAg
ICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxzcGFuPk9uIFN1biwgTm92
IDI2LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAyMDE3IGF0IDQ6NTMgUE0sIEdh
bCBCZW4gSGFpbSA8c3BhbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpcj0i
bHRyIj4mbHQ7PGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Im1h
aWx0bzpnYmVuaGFpbUByZWRoYXQuY29tIgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+Z2JlbmhhaW1A
cmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICB3cm90ZTo8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3Rl
IGNsYXNzPSJnbWFpbF9xdW90ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBz
dHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4CiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj5XZSBzdGlsbCBzZWUg
dGhpcyBpc3N1ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb24gdGhlIHVw
Z3JhZGUgc3VpdGUgZnJvbSBsYXRlc3QKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIHJlbGVhc2UgdG8gbWFzdGVyIFsxXS4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIDxkaXY+SSBkb24ndCBzZWUgYW55IGV2aWRlbmNlIGluCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICIvdmFyL2xvZy9tZXNzYWdlcyIgWzJdIHRoYXQKICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIm92aXJ0LWltYWdlaW8tcHJveHki
IHdhcyBzdGFydGVkCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR3aWNl
LjwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvc3Bhbj4KICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDxkaXY+U2luY2UgaXQncyBub3QgYSByZWdpc3RlcmVkIHBvcnQg
YW5kIGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaGlnaCBwb3J0LCBjb3VsZCBp
dCBiZSB1c2VkIGJ5IHNvbWV0aGluZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBl
bHNlICh3aGF0IGFyZSB0aGUgb2RkcyB0aG91Z2ggPzwvZGl2PgogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgPGRpdj5JcyBpdCBjb25zaXN0ZW50PzwvZGl2PgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPHNwYW4gY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZIT0Vu
WmIiPjxmb250CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9IiM4ODg4
ODgiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+WS48L2Rpdj4KICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9mb250Pjwvc3Bhbj4KICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxk
aXYgY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZoNSI+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPGRpdj7CoDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIKICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2MKICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgPGRpdj5bMV3CoDxhCmhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9i
bHVlL3Jlc3Qvb3JnYW5pemF0aW9ucy9qZW5raW5zL3BpcGVsaW5lcy9vdmlydC1tYXN0ZXJf
Y2hhbmdlLXF1ZXVlLXRlc3Rlci9ydW5zLzQxNTMvbm9kZXMvMTIzL3N0ZXBzLzI0MS9sb2cv
P3N0YXJ0PTAiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXJn
ZXQ9Il9ibGFuayIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1v
ei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2JsdWUvcmVz
dC9vcmdhbml6YXRpb25zL2plbmtpbnMvcGlwZWxpbmVzL292aXJ0LW1hc3Rlcl9jaGFuZ2Ut
cXVldWUtdGVzdGVyL3J1bnMvNDE1My9ub2Rlcy8xMjMvc3RlcHMvMjQxL2xvZy8/c3RhcnQ9
MDwvYT48L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj48
YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PlsyXSA8YQpocmVmPSJodHRwOi8v
amVua2lucy5vdmlydC5vcmcvdmlldy9DaGFuZ2UlMjBxdWV1ZSUyMGpvYnMvam9iL292aXJ0
LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxNTMvYXJ0aWZhY3QvZXhwb3J0ZWQtYXJ0
aWZhY3RzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXQtbWFzdGVyLWVsNy90ZXN0X2xvZ3Mv
dXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyL3Bvc3QtMDAxX2luaXRpYWxpemVf
ZW5naW5lLnB5L2xhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu
ZS9fdmFyX2xvZy9tZXNzYWdlcy8qdmlldyovIgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5z
Lm92aXJ0Lm9yZy92aWV3L0NoYW5nZSUyMHF1ZXVlJTIwam9icy9qb2Ivb3ZpcnQtbWFzdGVy
X2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDE1My9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMv
dXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRl
LWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUu
cHkvbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJf
bG9nL21lc3NhZ2VzLyp2aWV3Ki88L2E+wqDCoDwvZGl2PgogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDxkaXYKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4
MTEwMTU3aDUiPk9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IEZyaSwgTm92IDI0LCAyMDE3IGF0IDg6MTYgUE0sCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgIERhZm5hIFJvbiA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhCmhy
ZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90
LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiCiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAuOGV4O2JvcmRlci1sZWZ0OjFw
eCAjY2NjCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzb2xpZDtw
YWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxk
aXYKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0i
bV8tMzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdoNSI+CiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAw
MDAwIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYmdj
b2xvcj0iI0ZGRkZGRiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8ZGl2CmNsYXNzPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0
MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1OTM2MjM2MDIzbW96
LWNpdGUtcHJlZml4Ij50aGVyZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICB3ZXJlIHR3byBkaWZmZXJlbnQKICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgcGF0Y2hlcyByZXBvcnRlZCBhcwogICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmYWlsaW5nIGNxIHRv
ZGF5IHdpdGgKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgdGhlIG92aXJ0LWltYWdlaW8tcHJveHkKICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgc2VydmljZSBmYWlsaW5nIHRvCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0YXJ0LiA8YnI+CiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSGVyZSBpcyB0aGUgbGF0
ZXN0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZh
aWx1cmU6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDxhCmNsYXNzPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1
N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1OTM2MjM2MDIzbW96LXR4dC1saW5r
LWZyZWV0ZXh0IgpocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1h
c3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxMzAvYXJ0aWZhY3QiCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRv
LW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1h
c3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxMzAvYXJ0aWZhY3Q8L2E+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdgpjbGFzcz0ibV8t
MzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3NTUw
MjY2NzQxMGg1Ij48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICBPbiAxMS8yMy8yMDE3IDAzOjM5CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBQTSwgQWxsb24gTXVy
ZWluaWsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIHdyb3RlOjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDxkaXYKY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2
MDQxMDk4MTEwMTU3bV8yMTI3Mjc1NzU1MDI2Njc0MTBoNSI+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUKICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR5cGU9ImNpdGUi
PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
PGRpdiBkaXI9Imx0ciI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDxkaXYKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0iZm9udC1mYW1pbHk6bW9ub3NwYWNlLG1v
bm9zcGFjZSI+RGFuaWVsL05pcj88L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdgogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iZ21haWxfZXh0cmEi
Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgPGRpdgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIGNsYXNzPSJnbWFpbF9xdW90ZSI+T24KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUaHUsIE5vdiAyMywKICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAy
MDE3IGF0IDU6MjkgUE0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgRGFmbmEgUm9uIDxzcGFuCiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkaXI9Imx0ciI+Jmx0Ozxh
CmhyZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8t
bm90LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdyb3Rl
Ojxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8YmxvY2txdW90ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9ImdtYWlsX3F1b3RlIgpzdHlsZT0ibWFy
Z2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0
OjFleCI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICB0ZXh0PSIjMDAwMDAwIgpiZ2NvbG9yPSIjRkZGRkZGIj4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIDxwPkhpLCA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8cD5XZSBoYXZlIGEKICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZhaWxpbmcg
b24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIHRlc3QKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDAwMV9pbml0aWFsaXplX2VuZ2luZS50ZXN0X2luaXRpYWxpemVf
ZW5naW5lLgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+VGhpcyBpcwogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmFpbGluZyB3
aXRoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICBlcnJvciBGYWlsZWQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIHRvIHN0YXJ0CiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZXJ2aWNlCiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnb3Zp
cnQtaW1hZ2Vpby1wcm94eTwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxiCiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMzQ3MTgwNzI5MDk5MTc0MzM0
Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3NTUwMjY2NzQxMG1fMjcwMDMxMjYw
NTkzNjIzNjAyM21fLTE2NTMzOTAwMDAwNTY0MTIyMzFkb2NzLWludGVybmFsLWd1aWQtNTg1
OWI3YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3Ij4KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYj48L3A+CiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
cCBkaXI9Imx0ciIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0
O21hcmdpbi1ib3R0b206MHB0Ij48YgpzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIgppZD0i
bV8tMzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3
NTUwMjY2NzQxMG1fMjcwMDMxMjYwNTkzNjIzNjAyM21fLTE2NTMzOTAwMDAwNTY0MTIyMzFk
b2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3
Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6
IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtm
b250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpu
b25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5MaW5r
IGFuZCBoZWFkbGluZSBvZnRvIHN1c3BlY3RlZCBwYXRjaGVzOiAKPC9zcGFuPjwvYj48L3A+
CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICA8YgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIKaWQ9Im1fLTM0NzE4MDcy
OTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4MTEwMTU3bV8yMTI3Mjc1NzU1MDI2Njc0MTBt
XzI3MDAzMTI2MDU5MzYyMzYwMjNtXy0xNjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRlcm5h
bC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2NkZmVmNyI+CiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2I+
PGIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0zNDcxODA3MjkwOTkx
NzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAw
MzEyNjA1OTM2MjM2MDIzbV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJuYWwtZ3Vp
ZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHAgZGlyPSJs
dHIiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4t
Ym90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFy
aWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdl
aWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRl
Y29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUt
d3JhcCI+YnVpbGQ6IE1ha2UgcmVzdWx0aW5nIFJQTXMgYXJjaGl0ZWN0dXJlLXNwZWNpZmlj
IC0gPGEgY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4MTEw
MTU3bV8yMTI3Mjc1NzU1MDI2Njc0MTBtXzI3MDAzMTI2MDU5MzYyMzYwMjNtXy0xNjUzMzkw
MDAwMDU2NDEyMjMxbW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJp
dC5vdmlydC5vcmcvIy9jLzg0NTM0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2Vu
ZD0idHJ1ZSI+aHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy84NDUzNC88L2E+Cjwvc3Bh
bj48L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8cCBkaXI9Imx0ciIKICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdo
dDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0i
Zm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3Jv
dW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1h
bDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFs
aWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5MaW5rIHRvIEpvYjo8L3NwYW4+
PC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgPGEKY2xhc3M9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQx
MDk4MTEwMTU3bV8yMTI3Mjc1NzU1MDI2Njc0MTBtXzI3MDAzMTI2MDU5MzYyMzYwMjNtXy0x
NjUzMzkwMDAwMDU2NDEyMjMxbW96LXR4dC1saW5rLWZyZWV0ZXh0IgpocmVmPSJodHRwOi8v
amVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVy
LzQwNTUiCnRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9q
ZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIv
NDA1NTwvYT48L2I+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA8cD48YgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIK
aWQ9Im1fLTM0NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4MTEwMTU3bV8yMTI3
Mjc1NzU1MDI2Njc0MTBtXzI3MDAzMTI2MDU5MzYyMzYwMjNtXy0xNjUzMzkwMDAwMDU2NDEy
MjMxZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2Nk
ZmVmNyI+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPC9iPjwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3R5bGU9Imxp
bmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxiCnN0
eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2bV8t
NzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1OTM2
MjM2MDIzbV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5Yjdh
MS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPjxzcGFuIHN0eWxlPSJmb250LXNpemU6
MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6
dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFy
aWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxp
bmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPkxpbmsgdG8gYWxsIGxvZ3M6PC9zcGFuPjwvYj48
L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8YgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIKaWQ9Im1fLTM0NzE4
MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4MTEwMTU3bV8yMTI3Mjc1NzU1MDI2Njc0
MTBtXzI3MDAzMTI2MDU5MzYyMzYwMjNtXy0xNjUzMzkwMDAwMDU2NDEyMjMxZG9jcy1pbnRl
cm5hbC1ndWlkLTU4NTliN2ExLWU5NzUtOWEwZC0zYjAxLThkZTA1M2NkZmVmNyI+CiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
cCBkaXI9Imx0ciIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0
O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1m
YW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50
O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFs
O3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNw
YWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYj4KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgog
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRv
bTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtj
b2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6
NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0
aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAi
PjxhIGNsYXNzPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1
N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1OTM2MjM2MDIzbV8tMTY1MzM5MDAw
MDA1NjQxMjIzMW1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMu
b3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MDU1L2Fy
dGlmYWN0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDov
L2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rl
ci80MDU1L2FydGlmYWN0LzwvYT48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGIKICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJm
b250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2bV8tNzQyOTY3
NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1OTM2MjM2MDIz
bV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS1lOTc1
LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9iPjwvcD4KICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRy
IgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJv
dHRvbTowcHQiPjxiCnN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0zNDcxODA3
MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEw
bV8yNzAwMzEyNjA1OTM2MjM2MDIzbV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJu
YWwtZ3VpZC01ODU5YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPjxzcGFuIHN0
eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2Jh
Y2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6
bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGlj
YWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPjxhIGNsYXNzPSJtXy0z
NDcxODA3MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAy
NjY3NDEwbV8yNzAwMzEyNjA1OTM2MjM2MDIzbV8tMTY1MzM5MDAwMDA1NjQxMjIzMW1vei10
eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9v
dmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MDU1L2FydGlmYWN0L2V4cG9ydGVk
LWFydGlmYWN0cy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9s
b2dzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFs
aXplX2VuZ2luZS5weS9sYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1l
bmdpbmUvX3Zhcl9sb2cvbWVzc2FnZXMvKnZpZXcqLyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1k
by1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1t
YXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MDU1L2FydGlmYWN0L2V4cG9ydGVkLWFydGlm
YWN0cy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3Vw
Z3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2Vu
Z2luZS5weS9sYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUv
X3Zhcl9sb2cvbWVzc2FnZXMvKnZpZXcqLzwvYT48L3NwYW4+PC9iPjwvcD4KICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiCiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMzQ3MTgwNzI5MDk5MTc0MzM0
Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3NTUwMjY2NzQxMG1fMjcwMDMxMjYw
NTkzNjIzNjAyM21fLTE2NTMzOTAwMDAwNTY0MTIyMzFkb2NzLWludGVybmFsLWd1aWQtNTg1
OWI3YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3Ij4KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRyIgog
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRv
bTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtj
b2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6
NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0
aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAi
Pgo8L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgPHAgZGlyPSJsdHIiCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0ibGlu
ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4g
c3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7
YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHls
ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0
aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+KFJlbGV2YW50KSBl
cnJvciBzbmlwcGV0IGZyb20gdGhlIGxvZzogPC9zcGFuPjwvcD4KICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGRpcj0ibHRy
IgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJv
dHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlh
bDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWln
aHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNv
cmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdy
YXAiPiZsdDtlcnJvciZndDs8L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9iPmZyb20gbGFn
bwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgbG9nOiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8cHJlIHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7
d2hpdGUtc3BhY2U6cHJlLXdyYXA7d29yZC13cmFwOmJyZWFrLXdvcmQ7bWFyZ2luOjBweDtj
b2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQtc2l6ZToxNHB4O2ZvbnQtc3R5bGU6bm9ybWFsO2Zv
bnQtdmFyaWFudC1saWdhdHVyZXM6bm9ybWFsO2ZvbnQtdmFyaWFudC1jYXBzOm5vcm1hbDtm
b250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO3RleHQtYWxpZ246c3Rh
cnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d29yZC1zcGFjaW5nOjBw
eDt0ZXh0LWRlY29yYXRpb24tc3R5bGU6aW5pdGlhbDt0ZXh0LWRlY29yYXRpb24tY29sb3I6
aW5pdGlhbCI+RmFpbGVkIHRvIHN0YXJ0IHNlcnZpY2UgJ292aXJ0LWltYWdlaW8tcHJveHkK
Cm1lc3NhZ2VzIGxvZ3M6IAoKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9tLXJl
bGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBTdGFydGluZyBTZXNzaW9uIDgg
b2YgdXNlciByb290LgpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFRyYWNlYmFjayAo
bW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1m
cm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG
aWxlICIvdXNyL2Jpbi9vdmlydC1pbWFnZWlvLXByb3h5IiwgbGluZSA4NSwgaW4gJmx0O21v
ZHVsZSZndDsKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp
dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9w
cm94eS5tYWluKGFyZ3MsIGNvbmZpZykKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1m
cm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG
aWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydF9pbWFnZWlvX3By
b3h5L2ltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjMgMDc6MzA6NDcg
bGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWlt
YWdlaW8tcHJveHk6IGltYWdlX3NlcnZlci5zdGFydChjb25maWcpCk5vdiAyMyAwNzozMDo0
NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQt
aW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMv
b3ZpcnRfaW1hZ2Vpb19wcm94eS9zZXJ2ZXIucHkiLCBsaW5lIDQ1LCBpbiBzdGFydApOb3Yg
MjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5n
aW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1ZXN0SGFuZGxlcikKTm92IDIzIDA3
OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRT
ZXJ2ZXIucHkiLCBsaW5lIDQxOSwgaW4gX19pbml0X18KTm92IDIzIDA3OjMwOjQ3IGxhZ28t
dXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlv
LXByb3h5OiBzZWxmLnNlcnZlcl9iaW5kKCkKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFk
ZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5
OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dpcmVmL3NpbXBsZV9zZXJ2ZXIucHki
LCBsaW5lIDQ4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRl
LWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6
IEhUVFBTZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9CYXNlSFRUUFNlcnZlci5weSIsIGxp
bmUgMTA4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZy
b20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFNv
Y2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDIzIDA3OjMwOjQ3
IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1p
bWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIu
cHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiBzZWxmLnNvY2tldC5iaW5kKHNlbGYuc2VydmVyX2FkZHJlc3MpCk5vdiAyMyAwNzoz
MDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3Zp
cnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc29ja2V0LnB5
IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDIzIDA3OjMwOjQ3IGxhZ28tdXBncmFkZS1mcm9t
LXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiByZXR1
cm4gZ2V0YXR0cihzZWxmLl9zb2NrLG5hbWUpKCphcmdzKQpOb3YgMjMgMDc6MzA6NDcgbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdl
aW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBBZGRyZXNzIGFscmVhZHkgaW4g
dXNlCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1h
c3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlOiBtYWlu
IHByb2Nlc3MgZXhpdGVkLCBjb2RlPWV4aXRlZCwgc3RhdHVzPTEvRkFJTFVSRQpOb3YgMjMg
MDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5l
IHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJbWFnZUlPIFByb3h5LgpOb3YgMjMg
MDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5l
IHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGVudGVyZWQgZmFp
bGVkIHN0YXRlLgpOb3YgMjMgMDc6MzA6NDcgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z
dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2Vydmlj
ZSBmYWlsZWQuCk5vdiAyMyAwNzozMDo0NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1
aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNl
IGhvbGRvZmYgdGltZSBvdmVyLCBzY2hlZHVsaW5nIHJlc3RhcnQuCk5vdiAyMyAwNzozMDo0
NyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVt
ZDogU3RhcnRpbmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpOb3YgMjMgMDc6MzA6NDggbGFn
by11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdl
aW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92IDIzIDA3
OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9vdmlydC1pbWFnZWlvLXByb3h5
IiwgbGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBn
cmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy
b3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3MsIGNvbmZpZykKTm92IDIzIDA3
OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBv
dmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNr
YWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5L2ltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4g
bWFpbgpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1t
YXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGltYWdlX3NlcnZlci5zdGFydChj
b25maWcpCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRl
LW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0
aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS9zZXJ2ZXIucHkiLCBs
aW5lIDQ1LCBpbiBzdGFydApOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVs
ZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1
ZXN0SGFuZGxlcikKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut
c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xp
YjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQxOSwgaW4gX19pbml0X18K
Tm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVy
LWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNlcnZlcl9iaW5kKCkKTm92IDIz
IDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu
ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dp
cmVmL3NpbXBsZV9zZXJ2ZXIucHkiLCBsaW5lIDQ4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMg
MDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5l
IG92aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBTZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92
IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu
Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9C
YXNlSFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjMgMDc6
MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92
aXJ0LWltYWdlaW8tcHJveHk6IFNvY2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyX2JpbmQo
c2VsZikKTm92IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUt
bWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5
dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQKTm92
IDIzIDA3OjMwOjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu
Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tldC5iaW5kKHNlbGYuc2VydmVy
X2FkZHJlc3MpCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1
aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2
NC9weXRob24yLjcvc29ja2V0LnB5IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDIzIDA3OjMw
OjQ4IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly
dC1pbWFnZWlvLXByb3h5OiByZXR1cm4gZ2V0YXR0cihzZWxmLl9zb2NrLG5hbWUpKCphcmdz
KQpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0
ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4
XSBBZGRyZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUt
ZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vp
by1wcm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVkLCBjb2RlPWV4aXRlZCwgc3Rh
dHVzPTEvRkFJTFVSRQpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJ
bWFnZUlPIFByb3h5LgpOb3YgMjMgMDc6MzA6NDggbGFnby11cGdyYWRlLWZyb20tcmVsZWFz
ZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1wcm94
eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjMgMDc6MzA6NDggbGFnby11
cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0
LWltYWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVw
Z3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQt
aW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBvdmVyLCBzY2hlZHVsaW5nIHJl
c3RhcnQuCk5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRl
LW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogc3RhcnQgcmVxdWVzdCByZXBlYXRlZCB0b28gcXVp
Y2tseSBmb3Igb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlCk5vdiAyMyAwNzozMDo0OCBs
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDog
RmFpbGVkIHRvIHN0YXJ0IG9WaXJ0IEltYWdlSU8gUHJveHkuCk5vdiAyMyAwNzozMDo0OCBs
YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDog
VW5pdCBvdmlydC1pbWFnZWlvLXByb3h5LnNlcnZpY2UgZW50ZXJlZCBmYWlsZWQgc3RhdGUu
Ck5vdiAyMyAwNzozMDo0OCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rl
ci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGZhaWxlZC48
L3ByZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDxwPjxiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIgppZD0ibV8t
MzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3NTUw
MjY2NzQxMG1fMjcwMDMxMjYwNTkzNjIzNjAyM21fLTE2NTMzOTAwMDAwNTY0MTIyMzFkb2Nz
LWludGVybmFsLWd1aWQtNTg1OWI3YTEtZTk3NS05YTBkLTNiMDEtOGRlMDUzY2RmZWY3Ij4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIDwvYj48L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA8cCBkaXI9Imx0ciIKICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDox
LjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48YgpzdHlsZT0iZm9udC13
ZWlnaHQ6bm9ybWFsIgppZD0ibV8tMzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEw
OTgxMTAxNTdtXzIxMjcyNzU3NTUwMjY2NzQxMG1fMjcwMDMxMjYwNTkzNjIzNjAyM21fLTE2
NTMzOTAwMDAwNTY0MTIyMzFkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtZTk3NS05YTBk
LTNiMDEtOGRlMDUzY2RmZWY3Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1m
YW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50
O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFs
O3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNw
YWNlOnByZS13cmFwIj4mbHQ7L2Vycm9yJmd0Ozwvc3Bhbj48L2I+PC9wPgogICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGIKICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiCmlkPSJtXy0zNDcxODA3MjkwOTkxNzQzMzQ2
bV8tNzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEwbV8yNzAwMzEyNjA1
OTM2MjM2MDIzbV8tMTY1MzM5MDAwMDA1NjQxMjIzMWRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5
YjdhMS1lOTc1LTlhMGQtM2IwMS04ZGUwNTNjZGZlZjciPgogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9iPjxicgpjbGFzcz0i
bV8tMzQ3MTgwNzI5MDk5MTc0MzM0Nm1fLTc0Mjk2NzYwNDEwOTgxMTAxNTdtXzIxMjcyNzU3
NTUwMjY2NzQxMG1fMjcwMDMxMjYwNTkzNjIzNjAyM21fLTE2NTMzOTAwMDAwNTY0MTIyMzFB
cHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgpfX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEluZnJh
IG1haWxpbmcKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIGxpc3Q8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICA8YQogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOkluZnJhQG92
aXJ0Lm9yZyIKdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+SW5mcmFA
b3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIDxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQu
b3JnL21haWxtYW4vbGlzdGluZm8vaW5mcmEiCnJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJf
YmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9t
YWlsbWFuL2xpc3RpbmZvL2luZnJhPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+
CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
c3Bhbj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxi
cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRGV2ZWwgbWFp
bGluZyBsaXN0PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICA8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9
Im1haWx0bzpEZXZlbEBvdmlydC5vcmciCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5EZXZlbEBvdmly
dC5vcmc8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICA8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9
Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby9kZXZlbCIKICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZWw9Im5vcmVmZXJyZXIi
CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJf
YmxhbmsiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96
LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz
dGluZm8vZGV2ZWw8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgPC9zcGFuPjwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxz
cGFuCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9Im1fLTM0
NzE4MDcyOTA5OTE3NDMzNDZtXy03NDI5Njc2MDQxMDk4MTEwMTU3SE9FblpiIj48Zm9udAog
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9IiM4ODg4ODgi
Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGVh
cj0iYWxsIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+
PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAtLSA8YnI+CiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2CmNsYXNzPSJtXy0zNDcxODA3
MjkwOTkxNzQzMzQ2bV8tNzQyOTY3NjA0MTA5ODExMDE1N21fMjEyNzI3NTc1NTAyNjY3NDEw
Z21haWxfc2lnbmF0dXJlIgpkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj4KICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+
PGZvbnQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNv
bG9yPSIjMDAwMDAwIiBzaXplPSIyIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgZmFjZT0ib3ZlcnBhc3MsCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICBzYW5zLXNlcmlmIj48c3BhbgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0idGV4dC10cmFuc2Zv
cm06dXBwZXJjYXNlIj48Yj5HQUwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICBiRU4gSEFJTTwvYj48L3NwYW4+PC9mb250PgogICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+PHNwYW4Kc3R5bGU9ImNv
bG9yOnJnYigwLDAsMCk7Zm9udC1mYW1pbHk6b3ZlcnBhc3Msc2Fucy1zZXJpZjt0ZXh0LXRy
YW5zZm9ybTp1cHBlcmNhc2UiPjxmb250CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgc2l6ZT0iMiI+UkhWIERFVk9QUzwvZm9udD48L3NwYW4+
PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2
PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZm9udD48L3NwYW4+PC9kaXY+CiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+Cl9fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgRGV2ZWwgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRvOkRldmVsQG92aXJ0Lm9yZyIKICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+RGV2
ZWxAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Imh0dHA6Ly9s
aXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby9kZXZlbCIKICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayIK
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0
cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vZGV2ZWw8L2E+
PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAg
ICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2
PgogICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgPC9i
bG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAg
PGJyPgogICAgICAgICAgICAgICAgICA8YnIgY2xlYXI9ImFsbCI+CiAgICAgICAgICAgICAg
ICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAg
ICAgLS0gPGJyPgogICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJtXy0zNDcxODA3Mjkw
OTkxNzQzMzQ2Z21haWxfc2lnbmF0dXJlIgogICAgICAgICAgICAgICAgICAgIGRhdGEtc21h
cnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPgogICAgICAgICAgICAgICAgICAgIDxkaXYgZGly
PSJsdHIiPjxmb250IGNvbG9yPSIjMDAwMDAwIiBzaXplPSIyIgogICAgICAgICAgICAgICAg
ICAgICAgICBmYWNlPSJvdmVycGFzcywgc2Fucy1zZXJpZiI+PHNwYW4KICAgICAgICAgICAg
ICAgICAgICAgICAgICBzdHlsZT0idGV4dC10cmFuc2Zvcm06dXBwZXJjYXNlIj48Yj5HQUwg
YkVOCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBIQUlNPC9iPjwvc3Bhbj48L2ZvbnQ+
CiAgICAgICAgICAgICAgICAgICAgICA8ZGl2PjxzcGFuCnN0eWxlPSJjb2xvcjpyZ2IoMCww
LDApO2ZvbnQtZmFtaWx5Om92ZXJwYXNzLHNhbnMtc2VyaWY7dGV4dC10cmFuc2Zvcm06dXBw
ZXJjYXNlIj48Zm9udAogICAgICAgICAgICAgICAgICAgICAgICAgICAgc2l6ZT0iMiI+UkhW
IERFVk9QUzwvZm9udD48L3NwYW4+PGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9kaXY+
CiAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIDwvZGl2Pgog
ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICBfX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAgICAgICAgIERl
dmVsIG1haWxpbmcgbGlzdDxicj4KICAgICAgICAgICAgICAgIDxhIGhyZWY9Im1haWx0bzpE
ZXZlbEBvdmlydC5vcmciIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICBtb3ot
ZG8tbm90LXNlbmQ9InRydWUiPkRldmVsQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAgICAg
ICAgICA8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v
ZGV2ZWwiCiAgICAgICAgICAgICAgICAgIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxh
bmsiCiAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2xp
c3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RldmVsPC9hPjwvYmxvY2txdW90ZT4K
ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L2Rpdj4KICAg
ICAgPC9kaXY+CiAgICAgIDxicj4KICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtaW1lQXR0YWNo
bWVudEhlYWRlciI+PC9maWVsZHNldD4KICAgICAgPGJyPgogICAgICA8cHJlIHdyYXA9IiI+
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KRGV2ZWwg
bWFpbGluZyBsaXN0CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9
Im1haWx0bzpEZXZlbEBvdmlydC5vcmciPkRldmVsQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9
Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9t
YWlsbWFuL2xpc3RpbmZvL2RldmVsIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v
bGlzdGluZm8vZGV2ZWw8L2E+PC9wcmU+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8cD48YnI+
CiAgICA8L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg==
--------------510D2C3E2A3E3638E7FA4B2B--
5:55 p.m.
On Thu, Nov 23, 2017 at 5:30 PM Dafna Ron <dron(a)redhat.com> wrote:
Hi,
We have a failing on test 001_initialize_engine.test_initialize_engine.
This is failing with error Failed to start service 'ovirt-imageio-proxy
Can your reproduce it?
If fail with:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine
ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
This means the proxy is already running, and someone is trying
to start ti again.
Systemd should not allow this, but we have seen this in the daemon
few weeks ago.
The owner of the code trying to start the proxy should check this.
*Link and headline ofto suspected patches: *
* build: Make resulting RPMs architecture-specific -
https://gerrit.ovirt.org/#/c/84534/ <https://gerrit.ovirt.org/#/c/84534/>
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055>*
*Link to all logs:*
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifa...
* (Relevant) error snippet from the log: <error> *from lago log:
Failed to start service 'ovirt-imageio-proxy
messages logs:
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting Session 8
of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt
ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status
= image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21,
in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in
start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File
"/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return
getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy:
socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request
repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start
oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit
ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd:
ovirt-imageio-proxy.service failed.
*</error>*
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
5:58 p.m.
On 23.11.2017 16:29, Dafna Ron wrote:
Hi,
We have a failing on test 001_initialize_engine.test_initialize_engine.
This is failing with error Failed to start service 'ovirt-imageio-proxy
[...]
The patch hasn't changed any code, so I *think* I'm innocent here.
FWIW, I've noticed the following error in
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/consol...
...
basic-suit] RuntimeError: Failed to run reposync 3 times for repoid:
centos-extras-el7, aborting.
....
Not sure whether this indicates a real error condition.
[...]
--
Mit freundlichen Grüßen/Kind Regards
Viktor Mihajlovski
7:59 p.m.
There was an issue with the mirrors and I see the patch merged 7 hours
ago so its possible it was a late alert.
On 11/23/2017 03:58 PM, Viktor Mihajlovski wrote:
On 23.11.2017 16:29, Dafna Ron wrote:
> Hi,
>
> We have a failing on test 001_initialize_engine.test_initialize_engine.
>
> This is failing with error Failed to start service 'ovirt-imageio-proxy
>
>
[...]
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/
>
The patch hasn't changed any code, so I *think* I'm innocent here.
FWIW, I've noticed the following error in
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/consol...
...
basic-suit] RuntimeError: Failed to run reposync 3 times for repoid:
centos-extras-el7, aborting.
....
Not sure whether this indicates a real error condition.
[...]
2552
days inactive
2556
days old
13 comments
7 participants
participants (7)
-
Allon Mureinik -
Dafna Ron -
Gal Ben Haim -
Nir Soffer -
Viktor Mihajlovski -
Yaniv Kaul -
Yedidyah Bar David