[ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]
This is a multi-part message in MIME format. --------------C2726ECDFD4C62F2178F2F20 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit * Hi, ** We have failed cq with ovirt-imageio failing to start on upgrade suite*. *I can still only see errors**in the messages log. ** I'm writing the reported patch but I don't think it has anything to do with this issue. Link and headline of suspected patches: restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... (Relevant) error snippet from the log: <error> *From messages log *Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. </error> * --------------C2726ECDFD4C62F2178F2F20 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; word-wrap: break-word; white-space: pre-wrap;"><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Hi, </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p></b><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">We have failed cq with ovirt-imageio failing to start on upgrade suite<b>. </b>I can still only see errors<b> </b>in the messages log. </span></p></b><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I'm writing the reported patch but I don't think it has anything to do with this issue. </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link and headline of suspected patches: </span></p>restapi: Enable update to no default network provider of cluster - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84814/">https://gerrit.ovirt.org/#/c/84814/</a> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job:</span></p><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/</a> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> </span></p></b><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/</a> </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> </b>From messages log <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-0736-c7fc-29c0-3a6e4e1d9a66"> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p></b> </pre> </body> </html> --------------C2726ECDFD4C62F2178F2F20--
Do we have more info from Didi's debug plugin now? On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
*Hi,
*
*We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log. *
*I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches: restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ <https://gerrit.ovirt.org/#/c/84814/>
Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/> *
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/>
(Relevant) error snippet from the log: <error> *From messages log
*
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. </error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Nov 29, 2017 at 12:18 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Do we have more info from Didi's debug plugin now?
Not yet. Patch to install the debug plugins was merged this morning, build below failed yesterday evening.
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
*Hi,
*
*We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log. *
*I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches: restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ <https://gerrit.ovirt.org/#/c/84814/>
Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/> *
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/>
(Relevant) error snippet from the log: <error> *From messages log
*
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. </error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
This is a multi-part message in MIME format. --------------98CB106BD8B4230B34117EA6 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit this is the plugin info from steup log but I don't see anything more than we have seen except a timeout. https://pastebin.com/QVtNRNWV http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... Didi, is there anywhere else I should look? On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com <mailto:dron@redhat.com>> wrote:
*
Hi,
**
We have failed cq with ovirt-imageio failing to start on upgrade suite*. *I can still only see errors**in the messages log.
**
I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/%28root%29/001_initialize_engine/test_initialize_engine/>
**
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/...
(Relevant) error snippet from the log:
<error>
*From messages log *Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
*
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel
--------------98CB106BD8B4230B34117EA6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp dGUtcHJlZml4Ij50aGlzIGlzIHRoZSBwbHVnaW4gaW5mbyBmcm9tIHN0ZXVwIGxvZwogICAg ICBidXQgSSBkb24ndCBzZWUgYW55dGhpbmcgbW9yZSB0aGFuIHdlIGhhdmUgc2VlbiBleGNl cHQgYSB0aW1lb3V0LgogICAgICA8YnI+CiAgICAgIDxicj4KICAgICAgPGEgY2xhc3M9Im1v ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cHM6Ly9wYXN0ZWJpbi5jb20vUVZ0TlJO V1YiPmh0dHBzOi8vcGFzdGViaW4uY29tL1FWdE5STldWPC9hPjxicj4KICAgICAgPGJyPgo8 YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5v dmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0 aWZhY3QvZXhwb3J0ZWQtYXJ0aWZhY3RzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXQtbWFz dGVyLWVsNy90ZXN0X2xvZ3MvdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyL3Bv c3QtMDAxX2luaXRpYWxpemVfZW5naW5lLnB5L2xhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut c3VpdGUtbWFzdGVyLWVuZ2luZS9fdmFyX2xvZy9vdmlydC1lbmdpbmUvc2V0dXAvb3ZpcnQt ZW5naW5lLXNldHVwLTIwMTcxMTI4MTIzMTE2LW1tamVuMy5sb2ciPmh0dHA6Ly9qZW5raW5z Lm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDE5NC9h cnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdC1t YXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIv cG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby11cGdyYWRlLWZyb20tcmVsZWFz ZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJfbG9nL292aXJ0LWVuZ2luZS9zZXR1cC9vdmly dC1lbmdpbmUtc2V0dXAtMjAxNzExMjgxMjMxMTYtbW1qZW4zLmxvZzwvYT48YnI+CiAgICAg IDxicj4KICAgICAgRGlkaSwgaXMgdGhlcmUgYW55d2hlcmUgZWxzZSBJIHNob3VsZCBsb29r PyA8YnI+CiAgICAgIDxicj4KICAgICAgPGJyPgogICAgICBPbiAxMS8yOS8yMDE3IDEwOjE4 IEFNLCBOaXIgU29mZmVyIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUg dHlwZT0iY2l0ZSIKY2l0ZT0ibWlkOkNBTVJieXl2Z1RIc2k3czNQX2I9RmhEN2czSG1QTTlB UXM0cGRBcHRoaVNNcktOSzdfQUBtYWlsLmdtYWlsLmNvbSI+CiAgICAgIDxkaXYgZGlyPSJs dHIiPkRvIHdlIGhhdmUgbW9yZSBpbmZvIGZyb20gRGlkaSdzIGRlYnVnIHBsdWdpbiBub3c/ PGJyPgogICAgICAgIDxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAg ICAgICAgICA8ZGl2IGRpcj0ibHRyIj5PbiBXZWQsIE5vdiAyOSwgMjAxNyBhdCAxMjowNyBQ TSBEYWZuYSBSb24gJmx0OzxhCiAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOmRyb25AcmVk aGF0LmNvbSIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5kcm9uQHJlZGhhdC5jb208L2E+Jmd0 OwogICAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxi bG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMAogICAg ICAgICAgICAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDox ZXgiPgogICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYi PgogICAgICAgICAgICAgIDxwcmUgc3R5bGU9ImNvbG9yOnJnYigwLDAsMCk7Zm9udC1zdHls ZTpub3JtYWw7Zm9udC12YXJpYW50LWxpZ2F0dXJlczpub3JtYWw7Zm9udC12YXJpYW50LWNh cHM6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3JtYWw7dGV4 dC1hbGlnbjpzdGFydDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3b3Jk LXNwYWNpbmc6MHB4O3RleHQtZGVjb3JhdGlvbi1zdHlsZTppbml0aWFsO3RleHQtZGVjb3Jh dGlvbi1jb2xvcjppbml0aWFsO3dvcmQtd3JhcDpicmVhay13b3JkO3doaXRlLXNwYWNlOnBy ZS13cmFwIj48YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIiBpZD0ibV8tNzk2Mzg5Mjc2 MzM4MTgyNDI4NGRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0z YTZlNGUxZDlhNjYiPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJn aW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox MXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0 cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJp YW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGlu ZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+SGksIAo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7 Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246 bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwv c3Bhbj48L3A+PC9iPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXy03OTYz ODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0y OWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4 O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1z aXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNv bG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250 LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJh c2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5XZSBoYXZlIGZhaWxlZCBjcSB3aXRoIG92 aXJ0LWltYWdlaW8gZmFpbGluZyB0byBzdGFydCBvbiB1cGdyYWRlIHN1aXRlPGI+LiA8L2I+ SSBjYW4gc3RpbGwgb25seSBzZWUgZXJyb3JzPGI+IDwvYj5pbiB0aGUgbWVzc2FnZXMgbG9n LiAKPC9zcGFuPjwvcD48L2I+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIgaWQ9Im1f LTc5NjM4OTI3NjMzODE4MjQyODRkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtMDczNi1j N2ZjLTI5YzAtM2E2ZTRlMWQ5YTY2Ij48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0 OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91 bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFs O2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxp Z246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPkknbSB3cml0aW5nIHRoZSByZXBv cnRlZCBwYXRjaCBidXQgSSBkb24ndCB0aGluayBpdCBoYXMgYW55dGhpbmcgdG8gZG8gd2l0 aCB0aGlzIGlzc3VlLiAKPC9zcGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVp Z2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tn cm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9y bWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwt YWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPgo8L3NwYW4+PC9wPjxwIGRp cj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4t Ym90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFy aWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdl aWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRl Y29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUt d3JhcCI+TGluayBhbmQgaGVhZGxpbmUgb2Ygc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48 L3A+cmVzdGFwaTogRW5hYmxlIHVwZGF0ZSB0byBubyBkZWZhdWx0IG5ldHdvcmsgcHJvdmlk ZXIgb2YgY2x1c3RlciAtIDxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96LXR4 dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzg0 ODE0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9n ZXJyaXQub3ZpcnQub3JnLyMvYy84NDgxNC88L2E+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGlu ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7 YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHls ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0 aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+ PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21h cmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1p bHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2Zv bnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3Rl eHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNl OnByZS13cmFwIj5MaW5rIHRvIEpvYjo8L3NwYW4+PC9wPjxhIGNsYXNzPSJtXy03OTYzODky NzYzMzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lu cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv IiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lu cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv PC9hPgoKPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6 MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9u dC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFy ZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9y bWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRl LXNwYWNlOnByZS13cmFwIj5MaW5rIHRvIGFsbCBsb2dzOjwvc3Bhbj48L3A+PHAgZGlyPSJs dHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0 b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7 Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0 OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3Jh dGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw Ij4KPC9zcGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFy Z2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6 dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFy aWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxp bmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPjxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0 Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5v cmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0aWZhY3Qv IiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lu cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQv YXJ0aWZhY3QvPC9hPjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdo dDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3Jv dW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1h bDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFs aWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD48cCBkaXI9 Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJv dHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlh bDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWln aHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNv cmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdy YXAiPjxhIGNsYXNzPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0 ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9j aGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvdGVzdFJlcG9ydC9qdW5pdC8lMjhyb290JTI5LzAw MV9pbml0aWFsaXplX2VuZ2luZS90ZXN0X2luaXRpYWxpemVfZW5naW5lLyIgdGFyZ2V0PSJf YmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3Jn L2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MTk0L3Rlc3RSZXBvcnQv anVuaXQvKHJvb3QpLzAwMV9pbml0aWFsaXplX2VuZ2luZS90ZXN0X2luaXRpYWxpemVfZW5n aW5lLzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4z ODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1j b2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9u dC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpi YXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+PC9iPjxiIHN0eWxl PSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXy03OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1p bnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAg ZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdp bi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6 QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQt d2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQt ZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnBy ZS13cmFwIj48YSBjbGFzcz0ibV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1m cmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0 ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0 cy91cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3Jh ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2lu ZS5weS9sYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zh cl9sb2cvbWVzc2FnZXMvKnZpZXcqLyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2Vu ZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hh bmdlLXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy91cGdy YWRlLWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3JhZGUtZnJv bS1yZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2luZS5weS9s YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cv bWVzc2FnZXMvKnZpZXcqLzwvYT4KPC9zcGFuPjwvcD4KPHAgZGlyPSJsdHIiIHN0eWxlPSJs aW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAw MDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0 eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3Zl cnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4oUmVsZXZhbnQp IGVycm9yIHNuaXBwZXQgZnJvbSB0aGUgbG9nOiA8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7 Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246 bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0 O2Vycm9yJmd0Ozwvc3Bhbj48L3A+CjwvYj5Gcm9tIG1lc3NhZ2VzIGxvZwo8YiBzdHlsZT0i Zm9udC13ZWlnaHQ6bm9ybWFsIiBpZD0ibV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NGRvY3MtaW50 ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0zYTZlNGUxZDlhNjYiPgoKTm92 IDI4IDEyOjMyOjEzIGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu Z2luZSBzeXN0ZW1kOiBTdGFydGVkIG9WaXJ0IEVuZ2luZS4KTm92IDI4IDEyOjMyOjEzIGxh Z28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBS ZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1 aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogQ29uZmlndXJhdGlvbiBmaWxlIC91c3IvbGli L3N5c3RlbWQvc3lzdGVtL2VidGFibGVzLnNlcnZpY2UgaXMgbWFya2VkIGV4ZWN1dGFibGUu IFBsZWFzZSByZW1vdmUgZXhlY3V0YWJsZSBwZXJtaXNzaW9uIGJpdHMuIFByb2NlZWRpbmcg YW55d2F5LgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0 ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFN0YXJ0aW5nIG9WaXJ0IEVuZ2luZSBEYXRhIFdh cmVob3VzZS4uLgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFN0YXJ0ZWQgb1ZpcnQgRW5naW5lIERhdGEg V2FyZWhvdXNlLgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFJlbG9hZGluZy4KTm92IDI4IDEyOjMyOjEz IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1k OiBDb25maWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vZWJ0YWJsZXMu c2VydmljZSBpcyBtYXJrZWQgZXhlY3V0YWJsZS4gUGxlYXNlIHJlbW92ZSBleGVjdXRhYmxl IHBlcm1pc3Npb24gYml0cy4gUHJvY2VlZGluZyBhbnl3YXkuCk5vdiAyOCAxMjozMjoxMyBs YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDog U3RhcnRpbmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpOb3YgMjggMTI6MzI6MTQgbGFnby11 cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8t cHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92IDI4IDEyOjMy OjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly dC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9vdmlydC1pbWFnZWlvLXByb3h5Iiwg bGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFk ZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5 OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3MsIGNvbmZpZykKTm92IDI4IDEyOjMy OjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly dC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdl cy9vdmlydF9pbWFnZWlvX3Byb3h5L2ltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFp bgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0 ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGltYWdlX3NlcnZlci5zdGFydChjb25m aWcpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1h c3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0aG9u Mi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS9zZXJ2ZXIucHkiLCBsaW5l IDQ1LCBpbiBzdGFydApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFz ZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1ZXN0 SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0 L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQxOSwgaW4gX19pbml0X18KTm92 IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVu Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNlcnZlcl9iaW5kKCkKTm92IDI4IDEy OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBv dmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dpcmVm L3NpbXBsZV9zZXJ2ZXIucHkiLCBsaW5lIDQ4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjggMTI6 MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92 aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBTZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4 IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9CYXNl SFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2ZXJfYmluZApOb3YgMjggMTI6MzI6 MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0 LWltYWdlaW8tcHJveHk6IFNvY2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyX2JpbmQoc2Vs ZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFz dGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhv bjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQKTm92IDI4 IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tldC5iaW5kKHNlbGYuc2VydmVyX2Fk ZHJlc3MpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRl LW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9w eXRob24yLjcvc29ja2V0LnB5IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDI4IDEyOjMyOjE0 IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1p bWFnZWlvLXByb3h5OiByZXR1cm4gZ2V0YXR0cihzZWxmLl9zb2NrLG5hbWUpKCphcmdzKQpO b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIt ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBB ZGRyZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJv bS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1w cm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVkLCBjb2RlPWV4aXRlZCwgc3RhdHVz PTEvRkFJTFVSRQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJbWFn ZUlPIFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z dWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1hZ2Vpby1wcm94eS5z ZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdy YWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0LWlt YWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3Jh ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogb3ZpcnQtaW1h Z2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBvdmVyLCBzY2hlZHVsaW5nIHJlc3Rh cnQuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1h c3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRpbmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpO b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXIt ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2Fs bCBsYXN0KToKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2Jpbi9v dmlydC1pbWFnZWlvLXByb3h5IiwgbGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4 IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2lu ZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFyZ3Ms IGNvbmZpZykKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3Vp dGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYi9w eXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5L2ltYWdlX3Byb3h5 LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZy b20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IGlt YWdlX3NlcnZlci5zdGFydChjb25maWcpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUt ZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTog RmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19w cm94eS9zZXJ2ZXIucHkiLCBsaW5lIDQ1LCBpbiBzdGFydApOb3YgMjggMTI6MzI6MTQgbGFn by11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdl aW8tcHJveHk6IFdTR0lSZXF1ZXN0SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBn cmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy b3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5l IDQxOSwgaW4gX19pbml0X18KTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJl bGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNl cnZlcl9iaW5kKCkKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xp YjY0L3B5dGhvbjIuNy93c2dpcmVmL3NpbXBsZV9zZXJ2ZXIucHkiLCBsaW5lIDQ4LCBpbiBz ZXJ2ZXJfYmluZApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1z dWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBTZXJ2ZXIuc2Vy dmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh c2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxlICIvdXNy L2xpYjY0L3B5dGhvbjIuNy9CYXNlSFRUUFNlcnZlci5weSIsIGxpbmUgMTA4LCBpbiBzZXJ2 ZXJfYmluZApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0 ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFNvY2tldFNlcnZlci5UQ1BT ZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1m cm9tLXJlbGVhc2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG aWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwg aW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh c2Utc3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tl dC5iaW5kKHNlbGYuc2VydmVyX2FkZHJlc3MpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3Jh ZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94 eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc29ja2V0LnB5IiwgbGluZSAyMjQsIGlu IG1ldGgKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Utc3VpdGUt bWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiByZXR1cm4gZ2V0YXR0cihzZWxm Ll9zb2NrLG5hbWUpKCphcmdzKQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20t cmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tl dC5lcnJvcjogW0Vycm5vIDk4XSBBZGRyZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjoz MjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUgc3lz dGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlOiBtYWluIHByb2Nlc3MgZXhpdGVk LCBjb2RlPWV4aXRlZCwgc3RhdHVzPTEvRkFJTFVSRQpOb3YgMjggMTI6MzI6MTQgbGFnby11 cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxl ZCB0byBzdGFydCBvVmlydCBJbWFnZUlPIFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11 cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQg b3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3Yg MjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5n aW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBmYWlsZWQuCk5vdiAy OCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdp bmUgc3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBv dmVyLCBzY2hlZHVsaW5nIHJlc3RhcnQuCjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWln aHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dy b3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3Jt YWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1h bGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0Oy9lcnJvciZndDs8L3Nw YW4+PC9wPjwvYj4KCgoKCgoKCgoKCgoKCgoKPC9wcmU+CiAgICAgICAgICAgIDwvZGl2Pgog ICAgICAgICAgICBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXzxicj4KICAgICAgICAgICAgRGV2ZWwgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAg ICA8YSBocmVmPSJtYWlsdG86RGV2ZWxAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayIKICAg ICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPkRldmVsQG92aXJ0Lm9yZzwvYT48 YnI+CiAgICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h bi9saXN0aW5mby9kZXZlbCIKICAgICAgICAgICAgICByZWw9Im5vcmVmZXJyZXIiIHRhcmdl dD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmlydC5v cmcvbWFpbG1hbi9saXN0aW5mby9kZXZlbDwvYT48L2Jsb2NrcXVvdGU+CiAgICAgICAgPC9k aXY+CiAgICAgIDwvZGl2PgogICAgPC9ibG9ja3F1b3RlPgogICAgPHA+PGJyPgogICAgPC9w PgogIDwvYm9keT4KPC9odG1sPgo= --------------98CB106BD8B4230B34117EA6--
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron@redhat.com> wrote:
this is the plugin info from steup log but I don't see anything more than we have seen except a timeout.
http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/4194/artifact/exported-artifacts/upgrade-from- release-suit-master-el7/test_logs/upgrade-from-release- suite-master/post-001_initialize_engine.py/lago- upgrade-from-release-suite-master-engine/_var_log/ovirt- engine/setup/ovirt-engine-setup-20171128123116-mmjen3.log
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
*Hi,
*
*We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log. *
*I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches: restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ <https://gerrit.ovirt.org/#/c/84814/>
Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/%28root%29/001_initialize_engine/test_initialize_engine/> *
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/>
(Relevant) error snippet from the log: <error> *From messages log
*
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. </error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
This is a multi-part message in MIME format. --------------D2B187A49A90F623211FB3B4 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit we had a failure on 002_bootstrap.verify_add_hosts but the error is on imageio I looked at the host log that Nir added and I can only see that the address is in use which seems to be the same issue we have in initialize engine. http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifact/... I cannot see anything in host-deploy. Didi, would we be able to see anything here? Thanks, Dafna On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron@redhat.com <mailto:dron@redhat.com>> wrote:
this is the plugin info from steup log but I don't see anything more than we have seen except a timeout.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20171128123116-mmjen3.log>
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com <mailto:dron@redhat.com>> wrote:
*
Hi,
**
We have failed cq with ovirt-imageio failing to start on upgrade suite*. *I can still only see errors**in the messages log.
**
I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ <https://gerrit.ovirt.org/#/c/84814/>
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/%28root%29/001_initialize_engine/test_initialize_engine/>
**
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/>
(Relevant) error snippet from the log:
<error>
*From messages log *Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
*
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
-- Didi
--------------D2B187A49A90F623211FB3B4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGRpdiBjbGFzcz0ibW96LWNp dGUtcHJlZml4Ij53ZSBoYWQgYSBmYWlsdXJlIG9uIDxzcGFuCiAgICAgICAgc3R5bGU9ImNv bG9yOiByZ2IoNTEsIDUxLCA1MSk7IGZvbnQtc2l6ZTogMTRweDsgZm9udC1zdHlsZToKICAg ICAgICBub3JtYWw7IGZvbnQtdmFyaWFudC1saWdhdHVyZXM6IG5vcm1hbDsgZm9udC12YXJp YW50LWNhcHM6CiAgICAgICAgbm9ybWFsOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0 LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6CiAgICAgICAgMHB4OyB0ZXh0LXRyYW5zZm9y bTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOgogICAgICAgIDBw eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb24tc3R5 bGU6CiAgICAgICAgaW5pdGlhbDsgdGV4dC1kZWNvcmF0aW9uLWNvbG9yOiBpbml0aWFsOyBk aXNwbGF5OiBpbmxpbmUgIQogICAgICAgIGltcG9ydGFudDsgZmxvYXQ6IG5vbmU7Ij4wMDJf Ym9vdHN0cmFwLnZlcmlmeV9hZGRfaG9zdHMgYnV0IHRoZQogICAgICAgIGVycm9yIGlzIG9u IGltYWdlaW8gPGJyPgogICAgICAgIDxicj4KICAgICAgICBJIGxvb2tlZCBhdCB0aGUgaG9z dCBsb2cgdGhhdCBOaXIgYWRkZWQgYW5kIEkgY2FuIG9ubHkgc2VlIHRoYXQKICAgICAgICB0 aGUgYWRkcmVzcyBpcyBpbiB1c2Ugd2hpY2ggc2VlbXMgdG8gYmUgdGhlIHNhbWUgaXNzdWUg d2UgaGF2ZQogICAgICAgIGluIGluaXRpYWxpemUgZW5naW5lLiA8YnI+CiAgICAgIDwvc3Bh bj48YnI+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9q ZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIv NDIwNS9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMtc3VpdC1tYXN0ZXItZWw3 L3Rlc3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJfYm9vdHN0cmFwLnB5L2xh Z28tYmFzaWMtc3VpdGUtbWFzdGVyLWhvc3QtMC9fdmFyX2xvZy9vdmlydC1pbWFnZWlvLWRh ZW1vbi8iPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5n ZS1xdWV1ZS10ZXN0ZXIvNDIwNS9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMt c3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJf Ym9vdHN0cmFwLnB5L2xhZ28tYmFzaWMtc3VpdGUtbWFzdGVyLWhvc3QtMC9fdmFyX2xvZy9v dmlydC1pbWFnZWlvLWRhZW1vbi88L2E+PGJyPgogICAgICA8YnI+CiAgICAgIEkgY2Fubm90 IHNlZSBhbnl0aGluZyBpbiBob3N0LWRlcGxveS4gPGJyPgogICAgICBEaWRpLCB3b3VsZCB3 ZSBiZSBhYmxlIHRvIHNlZSBhbnl0aGluZyBoZXJlPyA8YnI+CiAgICAgIDxicj4KICAgICAg VGhhbmtzLCA8YnI+CiAgICAgIERhZm5hPGJyPgogICAgICA8YnI+CiAgICAgIDxicj4KICAg ICAgT24gMTEvMjkvMjAxNyAxMTowMyBBTSwgWWVkaWR5YWggQmFyIERhdmlkIHdyb3RlOjxi cj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKY2l0ZT0ibWlkOkNB SFJ3WVh1bUdkLUU1eDZrZ21GRzdtT2lMcFZnUlNrPXVLZ0hwTU5pM242Q0o4X2Vrd0BtYWls LmdtYWlsLmNvbSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgIDxkaXYgY2xhc3M9 ImdtYWlsX2V4dHJhIj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBX ZWQsIE5vdiAyOSwgMjAxNyBhdCAxOjAwIFBNLAogICAgICAgICAgICBEYWZuYSBSb24gPHNw YW4gZGlyPSJsdHIiPiZsdDs8YQogICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOmRyb25A cmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICBtb3otZG8tbm90 LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAg ICB3cm90ZTo8YnI+CiAgICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90 ZSIgc3R5bGU9Im1hcmdpbjowIDAgMAogICAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6 MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgPGRpdiB0 ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgICAgICAgICAgICAgIDxkaXYg Y2xhc3M9Im1fMTIzMjE1NjgyODI3NDgwMjU2N21vei1jaXRlLXByZWZpeCI+dGhpcwogICAg ICAgICAgICAgICAgICBpcyB0aGUgcGx1Z2luIGluZm8gZnJvbSBzdGV1cCBsb2cgYnV0IEkg ZG9uJ3Qgc2VlCiAgICAgICAgICAgICAgICAgIGFueXRoaW5nIG1vcmUgdGhhbiB3ZSBoYXZl IHNlZW4gZXhjZXB0IGEgdGltZW91dC4gPGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAg ICAgICAgICAgICAgICAgIDxhIGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1Njdtb3otdHh0 LWxpbmstZnJlZXRleHQiCiAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cHM6Ly9wYXN0 ZWJpbi5jb20vUVZ0TlJOV1YiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICAg IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9wYXN0ZWJpbi5jb20vUVZ0TlJOV1Y8 L2E+PGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgIDxhIGNs YXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1Njdtb3otdHh0LWxpbmstZnJlZXRleHQiCmhyZWY9 Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1 ZS10ZXN0ZXIvNDE5NC9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvdXBncmFkZS1mcm9t LXJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy91cGdyYWRlLWZyb20tcmVsZWFz ZS1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDFfaW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby11cGdy YWRlLWZyb20tcmVsZWFzZS1zdWl0ZS1tYXN0ZXItZW5naW5lL192YXJfbG9nL292aXJ0LWVu Z2luZS9zZXR1cC9vdmlydC1lbmdpbmUtc2V0dXAtMjAxNzExMjgxMjMxMTYtbW1qZW4zLmxv ZyIKICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5k PSJ0cnVlIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+b3ZpcnQtbWFzdGVy X2NoYW5nZS1xdWV1ZS08d2JyPnRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLTx3YnI+ YXJ0aWZhY3RzL3VwZ3JhZGUtZnJvbS08d2JyPnJlbGVhc2Utc3VpdC1tYXN0ZXItZWw3L3Rl c3RfPHdicj5sb2dzL3VwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyL3Bv c3QtMDAxXzx3YnI+aW5pdGlhbGl6ZV9lbmdpbmUucHkvbGFnby08d2JyPnVwZ3JhZGUtZnJv bS1yZWxlYXNlLXN1aXRlLTx3YnI+bWFzdGVyLWVuZ2luZS9fdmFyX2xvZy9vdmlydC08d2Jy PmVuZ2luZS9zZXR1cC9vdmlydC1lbmdpbmUtPHdicj5zZXR1cC0yMDE3MTEyODEyMzExNi1t bWplbjMuPHdicj5sb2c8L2E+PGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAg ICAgICAgICAgIERpZGksIGlzIHRoZXJlIGFueXdoZXJlIGVsc2UgSSBzaG91bGQgbG9vaz8g PGJyPgogICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAg IDwvZGl2PgogICAgICAgICAgICA8ZGl2PlNhZGx5LCBhcyBhbHJlYWR5IHJlcGxpZWQsIG5v dCB5ZXQuIEhvcGVmdWxseSBuZXh0CiAgICAgICAgICAgICAgdGltZS4uLjxicj4KICAgICAg ICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXY+wqA8L2Rpdj4KICAgICAgICAgICAgPGJs b2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAg ICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6 MWV4Ij4KICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZG RkYiPgogICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3 bW96LWNpdGUtcHJlZml4Ij4KICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJoNSI+IDxicj4KICAgICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICAgIE9uIDExLzI5LzIwMTcgMTA6MTggQU0sIE5pciBT b2ZmZXIgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg ICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAg PGRpdj4KICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iaDUiPgogICAgICAgICAgICAg ICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgICAgICAgICAg PGRpdiBkaXI9Imx0ciI+RG8gd2UgaGF2ZSBtb3JlIGluZm8gZnJvbSBEaWRpJ3MKICAgICAg ICAgICAgICAgICAgICAgICAgZGVidWcgcGx1Z2luIG5vdz88YnI+CiAgICAgICAgICAgICAg ICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21h aWxfcXVvdGUiPgogICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPk9u IFdlZCwgTm92IDI5LCAyMDE3IGF0IDEyOjA3CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICBQTSBEYWZuYSBSb24gJmx0OzxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhy ZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmRyb25AcmVkaGF0 LmNvbTwvYT4mZ3Q7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweAogICAg ICAgICAgICAgICAgICAgICAgICAgICAgI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xv cj0iI0ZGRkZGRiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwcmUgc3R5bGU9 ImNvbG9yOnJnYigwLDAsMCk7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50LWxpZ2F0 dXJlczpub3JtYWw7Zm9udC12YXJpYW50LWNhcHM6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1h bDtsZXR0ZXItc3BhY2luZzpub3JtYWw7dGV4dC1hbGlnbjpzdGFydDt0ZXh0LWluZGVudDow cHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3b3JkLXNwYWNpbmc6MHB4O3RleHQtZGVjb3JhdGlv bi1zdHlsZTppbml0aWFsO3RleHQtZGVjb3JhdGlvbi1jb2xvcjppbml0aWFsO3dvcmQtd3Jh cDpicmVhay13b3JkO3doaXRlLXNwYWNlOnByZS13cmFwIj48YiBzdHlsZT0iZm9udC13ZWln aHQ6bm9ybWFsIiBpZD0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3bV8tNzk2Mzg5Mjc2MzM4MTgy NDI4NGRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS0wNzM2LWM3ZmMtMjljMC0zYTZlNGUx ZDlhNjYiPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9w OjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2Zv bnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3Bh cmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5v cm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0 ZS1zcGFjZTpwcmUtd3JhcCI+SGksIAo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0i bGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAw MDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1z dHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2 ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48 L3A+PC9iPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXzEyMzIxNTY4Mjgy NzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTli N2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJs aW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAw MDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0 eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3Zl cnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5XZSBoYXZlIGZh aWxlZCBjcSB3aXRoIG92aXJ0LWltYWdlaW8gZmFpbGluZyB0byBzdGFydCBvbiB1cGdyYWRl IHN1aXRlPGI+LiA8L2I+SSBjYW4gc3RpbGwgb25seSBzZWUgZXJyb3JzPGI+IDwvYj5pbiB0 aGUgbWVzc2FnZXMgbG9nLiAKPC9zcGFuPjwvcD48L2I+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0 Om5vcm1hbCIgaWQ9Im1fMTIzMjE1NjgyODI3NDgwMjU2N21fLTc5NjM4OTI3NjMzODE4MjQy ODRkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtMDczNi1jN2ZjLTI5YzAtM2E2ZTRlMWQ5 YTY2Ij48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDow cHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250 LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJl bnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3Jt YWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUt c3BhY2U6cHJlLXdyYXAiPkknbSB3cml0aW5nIHRoZSByZXBvcnRlZCBwYXRjaCBidXQgSSBk b24ndCB0aGluayBpdCBoYXMgYW55dGhpbmcgdG8gZG8gd2l0aCB0aGlzIGlzc3VlLiAKPC9z cGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRv cDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtm b250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpu b3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hp dGUtc3BhY2U6cHJlLXdyYXAiPgo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGlu ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7 YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHls ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0 aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+TGluayBhbmQgaGVh ZGxpbmUgb2Ygc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48L3A+cmVzdGFwaTogRW5hYmxl IHVwZGF0ZSB0byBubyBkZWZhdWx0IG5ldHdvcmsgcHJvdmlkZXIgb2YgY2x1c3RlciAtIDxh IGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0bW96 LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9j Lzg0ODE0LyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6 Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy88d2JyPjg0ODE0LzwvYT4KPHAgZGlyPSJsdHIiIHN0 eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0 Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6 IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtm b250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpu b25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9z cGFuPjwvcD48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRv cDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtm b250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpu b3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hp dGUtc3BhY2U6cHJlLXdyYXAiPkxpbmsgdG8gSm9iOjwvc3Bhbj48L3A+PGEgY2xhc3M9Im1f MTIzMjE1NjgyODI3NDgwMjU2N21fLTc5NjM4OTI3NjMzODE4MjQyODRtb3otdHh0LWxpbmst ZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFz dGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDE5NC8iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8t bm90LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmly dC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLTx3YnI+dGVzdGVyLzQxOTQvPC9hPgoKPHAgZGlyPSJs dHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0 b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7 Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0 OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3Jh dGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw Ij5MaW5rIHRvIGFsbCBsb2dzOjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5l LWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBz dHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDti YWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxl Om5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRp Y2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4KPC9zcGFuPjwvcD48 cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFy Z2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWls eTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9u dC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4 dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6 cHJlLXdyYXAiPjxhIGNsYXNzPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYz MzgxODI0Mjg0bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5v dmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQxOTQvYXJ0 aWZhY3QvIiB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8v amVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+b3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS08 d2JyPnRlc3Rlci80MTk0L2FydGlmYWN0LzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7 Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246 bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwv c3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10 b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7 Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5z cGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6 bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3do aXRlLXNwYWNlOnByZS13cmFwIj48YSBjbGFzcz0ibV8xMjMyMTU2ODI4Mjc0ODAyNTY3bV8t Nzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDov L2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rl ci80MTk0L3Rlc3RSZXBvcnQvanVuaXQvJTI4cm9vdCUyOS8wMDFfaW5pdGlhbGl6ZV9lbmdp bmUvdGVzdF9pbml0aWFsaXplX2VuZ2luZS8iIHRhcmdldD0iX2JsYW5rIiBtb3otZG8tbm90 LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmlydC1t YXN0ZXJfY2hhbmdlLXF1ZXVlLTx3YnI+dGVzdGVyLzQxOTQvdGVzdFJlcG9ydC9qdW5pdC8o PHdicj5yb290KS8wMDFfaW5pdGlhbGl6ZV9lbmdpbmUvPHdicj50ZXN0X2luaXRpYWxpemVf ZW5naW5lLzwvYT48L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6 MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3Vu ZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7 Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGln bjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Cjwvc3Bhbj48L3A+PC9iPjxiIHN0 eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03 OTYzODkyNzYzMzgxODI0Mjg0ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdm Yy0yOWMwLTNhNmU0ZTFkOWE2NiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDox LjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9u dC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5k LWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtm b250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWdu OmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj48YSBjbGFzcz0ibV8xMjMyMTU2ODI4 Mjc0ODAyNTY3bV8tNzk2Mzg5Mjc2MzM4MTgyNDI4NG1vei10eHQtbGluay1mcmVldGV4dCIg aHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdl LXF1ZXVlLXRlc3Rlci80MTk0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy91cGdyYWRl LWZyb20tcmVsZWFzZS1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL3VwZ3JhZGUtZnJvbS1y ZWxlYXNlLXN1aXRlLW1hc3Rlci9wb3N0LTAwMV9pbml0aWFsaXplX2VuZ2luZS5weS9sYWdv LXVwZ3JhZGUtZnJvbS1yZWxlYXNlLXN1aXRlLW1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cvbWVz c2FnZXMvKnZpZXcqLyIgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+ aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi88d2JyPm92aXJ0LW1hc3Rlcl9jaGFuZ2Ut cXVldWUtPHdicj50ZXN0ZXIvNDE5NC9hcnRpZmFjdC9leHBvcnRlZC08d2JyPmFydGlmYWN0 cy91cGdyYWRlLWZyb20tPHdicj5yZWxlYXNlLXN1aXQtbWFzdGVyLWVsNy90ZXN0Xzx3YnI+ bG9ncy91cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci9wb3N0LTAwMV88 d2JyPmluaXRpYWxpemVfZW5naW5lLnB5L2xhZ28tPHdicj51cGdyYWRlLWZyb20tcmVsZWFz ZS1zdWl0ZS08d2JyPm1hc3Rlci1lbmdpbmUvX3Zhcl9sb2cvPHdicj5tZXNzYWdlcy8qdmll dyovPC9hPgo8L3NwYW4+PC9wPgo8cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEu Mzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQt Y29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2Zv bnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246 YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXAiPihSZWxldmFudCkgZXJyb3Igc25pcHBl dCBmcm9tIHRoZSBsb2c6IDwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhl aWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0Ij48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNr Z3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5v cm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2Fs LWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4mbHQ7ZXJyb3ImZ3Q7PC9z cGFuPjwvcD4KPC9iPkZyb20gbWVzc2FnZXMgbG9nCjxiIHN0eWxlPSJmb250LXdlaWdodDpu b3JtYWwiIGlkPSJtXzEyMzIxNTY4MjgyNzQ4MDI1NjdtXy03OTYzODkyNzYzMzgxODI0Mjg0 ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTA3MzYtYzdmYy0yOWMwLTNhNmU0ZTFkOWE2 NiI+CgpOb3YgMjggMTI6MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1 aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRlZCBvVmlydCBFbmdpbmUuCk5vdiAy OCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy LWVuZ2luZSBzeXN0ZW1kOiBSZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3Jh ZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBDb25m aWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vPHdicj5lYnRhYmxlcy5z ZXJ2aWNlIGlzIG1hcmtlZCBleGVjdXRhYmxlLiBQbGVhc2UgcmVtb3ZlIGV4ZWN1dGFibGUg cGVybWlzc2lvbiBiaXRzLiBQcm9jZWVkaW5nIGFueXdheS4KTm92IDI4IDEyOjMyOjEzIGxh Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3Rl bWQ6IFN0YXJ0aW5nIG9WaXJ0IEVuZ2luZSBEYXRhIFdhcmVob3VzZS4uLgpOb3YgMjggMTI6 MzI6MTMgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdp bmUgc3lzdGVtZDogU3RhcnRlZCBvVmlydCBFbmdpbmUgRGF0YSBXYXJlaG91c2UuCk5vdiAy OCAxMjozMjoxMyBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVy LWVuZ2luZSBzeXN0ZW1kOiBSZWxvYWRpbmcuCk5vdiAyOCAxMjozMjoxMyBsYWdvLXVwZ3Jh ZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBDb25m aWd1cmF0aW9uIGZpbGUgL3Vzci9saWIvc3lzdGVtZC9zeXN0ZW0vPHdicj5lYnRhYmxlcy5z ZXJ2aWNlIGlzIG1hcmtlZCBleGVjdXRhYmxlLiBQbGVhc2UgcmVtb3ZlIGV4ZWN1dGFibGUg cGVybWlzc2lvbiBiaXRzLiBQcm9jZWVkaW5nIGFueXdheS4KTm92IDI4IDEyOjMyOjEzIGxh Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3Rl bWQ6IFN0YXJ0aW5nIG9WaXJ0IEltYWdlSU8gUHJveHkuLi4KTm92IDI4IDEyOjMyOjE0IGxh Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0 LWltYWdlaW8tcHJveHk6IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKTm92 IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0 ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvYmluL292aXJ0LWlt YWdlaW8tcHJveHkiPHdicj4sIGxpbmUgODUsIGluICZsdDttb2R1bGUmZ3Q7Ck5vdiAyOCAx MjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVu Z2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzdGF0dXMgPSBpbWFnZV9wcm94eS5tYWluKGFy Z3MsIGNvbmZpZykKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2Ut PHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91 c3IvbGliL3B5dGhvbjIuNy9zaXRlLTx3YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94 eS88d2JyPmltYWdlX3Byb3h5LnB5IiwgbGluZSAyMSwgaW4gbWFpbgpOb3YgMjggMTI6MzI6 MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg b3ZpcnQtaW1hZ2Vpby1wcm94eTogaW1hZ2Vfc2VydmVyLnN0YXJ0KGNvbmZpZykKTm92IDI4 IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXIt ZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9z aXRlLTx3YnI+cGFja2FnZXMvb3ZpcnRfaW1hZ2Vpb19wcm94eS88d2JyPnNlcnZlci5weSIs IGxpbmUgNDUsIGluIHN0YXJ0Ck5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBX U0dJUmVxdWVzdEhhbmRsZXIpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBG aWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPlNvY2tldFNlcnZlci5weSIsIGxpbmUg NDE5LCBpbiBfX2luaXRfXwpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVs ZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogc2Vs Zi5zZXJ2ZXJfYmluZCgpCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxl YXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBGaWxl ICIvdXNyL2xpYjY0L3B5dGhvbjIuNy93c2dpcmVmLzx3YnI+c2ltcGxlX3NlcnZlci5weSIs IGxpbmUgNDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUt ZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXBy b3h5OiBIVFRQU2VydmVyLnNlcnZlcl9iaW5kKHNlbGYpCk5vdiAyOCAxMjozMjoxNCBsYWdv LXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1p bWFnZWlvLXByb3h5OiBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy88d2JyPkJhc2VIVFRQ U2VydmVyLnB5IiwgbGluZSAxMDgsIGluIHNlcnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBs YWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmly dC1pbWFnZWlvLXByb3h5OiBTb2NrZXRTZXJ2ZXIuVENQU2VydmVyLnNlcnZlcl88d2JyPmJp bmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3Iv bGliNjQvcHl0aG9uMi43Lzx3YnI+U29ja2V0U2VydmVyLnB5IiwgbGluZSA0MzAsIGluIHNl cnZlcl9iaW5kCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3 YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNvY2tl dC5iaW5kKHNlbGYuc2VydmVyXzx3YnI+YWRkcmVzcykKTm92IDI4IDEyOjMyOjE0IGxhZ28t dXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWlt YWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NvY2tldC48d2JyPnB5 IiwgbGluZSAyMjQsIGluIG1ldGgKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9t LXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6 IHJldHVybiBnZXRhdHRyKHNlbGYuX3NvY2ssbmFtZSkoKjx3YnI+YXJncykKTm92IDI4IDEy OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n aW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNvY2tldC5lcnJvcjogW0Vycm5vIDk4XSBBZGRy ZXNzIGFscmVhZHkgaW4gdXNlCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1y ZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBvdmlydC1pbWFnZWlv LXByb3h5LnNlcnZpY2U6IG1haW4gcHJvY2VzcyBleGl0ZWQsIGNvZGU9ZXhpdGVkLCBzdGF0 dXM9MS9GQUlMVVJFCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNl LTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBGYWlsZWQgdG8gc3RhcnQgb1Zp cnQgSW1hZ2VJTyBQcm94eS4KTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJl bGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IFVuaXQgb3ZpcnQtaW1h Z2Vpby1wcm94eS5zZXJ2aWNlIGVudGVyZWQgZmFpbGVkIHN0YXRlLgpOb3YgMjggMTI6MzI6 MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg c3lzdGVtZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGZhaWxlZC4KTm92IDI4IDEy OjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5n aW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2VydmljZSBob2xkb2ZmIHRpbWUg b3Zlciwgc2NoZWR1bGluZyByZXN0YXJ0LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRl LWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogU3RhcnRp bmcgb1ZpcnQgSW1hZ2VJTyBQcm94eS4uLgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRl LWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1w cm94eTogVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgpOb3YgMjggMTI6MzI6 MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUg b3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9iaW4vb3ZpcnQtaW1hZ2Vpby1wcm94 eSI8d2JyPiwgbGluZSA4NSwgaW4gJmx0O21vZHVsZSZndDsKTm92IDI4IDEyOjMyOjE0IGxh Z28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0 LWltYWdlaW8tcHJveHk6IHN0YXR1cyA9IGltYWdlX3Byb3h5Lm1haW4oYXJncywgY29uZmln KQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRl LW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0 aG9uMi43L3NpdGUtPHdicj5wYWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5Lzx3YnI+aW1h Z2VfcHJveHkucHkiLCBsaW5lIDIxLCBpbiBtYWluCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVw Z3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFn ZWlvLXByb3h5OiBpbWFnZV9zZXJ2ZXIuc3RhcnQoY29uZmlnKQpOb3YgMjggMTI6MzI6MTQg bGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3Zp cnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtPHdicj5w YWNrYWdlcy9vdmlydF9pbWFnZWlvX3Byb3h5Lzx3YnI+c2VydmVyLnB5IiwgbGluZSA0NSwg aW4gc3RhcnQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IFdTR0lSZXF1ZXN0 SGFuZGxlcikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi cj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3Iv bGliNjQvcHl0aG9uMi43Lzx3YnI+U29ja2V0U2VydmVyLnB5IiwgbGluZSA0MTksIGluIF9f aW5pdF9fCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVwZ3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+ c3VpdGUtbWFzdGVyLWVuZ2luZSBvdmlydC1pbWFnZWlvLXByb3h5OiBzZWxmLnNlcnZlcl9i aW5kKCkKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5z dWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEZpbGUgIi91c3IvbGli NjQvcHl0aG9uMi43L3dzZ2lyZWYvPHdicj5zaW1wbGVfc2VydmVyLnB5IiwgbGluZSA0OCwg aW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVh c2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IEhUVFBT ZXJ2ZXIuc2VydmVyX2JpbmQoc2VsZikKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1m cm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJv eHk6IEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43Lzx3YnI+QmFzZUhUVFBTZXJ2ZXIucHki LCBsaW5lIDEwOCwgaW4gc2VydmVyX2JpbmQKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFk ZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1tYXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8t cHJveHk6IFNvY2tldFNlcnZlci5UQ1BTZXJ2ZXIuc2VydmVyXzx3YnI+YmluZChzZWxmKQpO b3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1h c3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogRmlsZSAiL3Vzci9saWI2NC9weXRo b24yLjcvPHdicj5Tb2NrZXRTZXJ2ZXIucHkiLCBsaW5lIDQzMCwgaW4gc2VydmVyX2JpbmQK Tm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0ZS1t YXN0ZXItZW5naW5lIG92aXJ0LWltYWdlaW8tcHJveHk6IHNlbGYuc29ja2V0LmJpbmQoc2Vs Zi5zZXJ2ZXJfPHdicj5hZGRyZXNzKQpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZy b20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94 eTogRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc29ja2V0Ljx3YnI+cHkiLCBsaW5lIDIy NCwgaW4gbWV0aApOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08 d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQtaW1hZ2Vpby1wcm94eTogcmV0dXJuIGdl dGF0dHIoc2VsZi5fc29jayxuYW1lKSgqPHdicj5hcmdzKQpOb3YgMjggMTI6MzI6MTQgbGFn by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgb3ZpcnQt aW1hZ2Vpby1wcm94eTogc29ja2V0LmVycm9yOiBbRXJybm8gOThdIEFkZHJlc3MgYWxyZWFk eSBpbiB1c2UKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdi cj5zdWl0ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IG92aXJ0LWltYWdlaW8tcHJveHkuc2Vy dmljZTogbWFpbiBwcm9jZXNzIGV4aXRlZCwgY29kZT1leGl0ZWQsIHN0YXR1cz0xL0ZBSUxV UkUKTm92IDI4IDEyOjMyOjE0IGxhZ28tdXBncmFkZS1mcm9tLXJlbGVhc2UtPHdicj5zdWl0 ZS1tYXN0ZXItZW5naW5lIHN5c3RlbWQ6IEZhaWxlZCB0byBzdGFydCBvVmlydCBJbWFnZUlP IFByb3h5LgpOb3YgMjggMTI6MzI6MTQgbGFnby11cGdyYWRlLWZyb20tcmVsZWFzZS08d2Jy PnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVtZDogVW5pdCBvdmlydC1pbWFnZWlvLXByb3h5 LnNlcnZpY2UgZW50ZXJlZCBmYWlsZWQgc3RhdGUuCk5vdiAyOCAxMjozMjoxNCBsYWdvLXVw Z3JhZGUtZnJvbS1yZWxlYXNlLTx3YnI+c3VpdGUtbWFzdGVyLWVuZ2luZSBzeXN0ZW1kOiBv dmlydC1pbWFnZWlvLXByb3h5LnNlcnZpY2UgZmFpbGVkLgpOb3YgMjggMTI6MzI6MTQgbGFn by11cGdyYWRlLWZyb20tcmVsZWFzZS08d2JyPnN1aXRlLW1hc3Rlci1lbmdpbmUgc3lzdGVt ZDogb3ZpcnQtaW1hZ2Vpby1wcm94eS5zZXJ2aWNlIGhvbGRvZmYgdGltZSBvdmVyLCBzY2hl ZHVsaW5nIHJlc3RhcnQuCjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODtt YXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xv cjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12 YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNl bGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0Oy9lcnJvciZndDs8L3NwYW4+PC9wPjwv Yj4KCgoKCgoKCgoKCgoKCgoKPC9wcmU+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIF9fX19fX19fX19fX19fX19fX19f X19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICBEZXZlbCBtYWlsaW5nIGxpc3Q8YnI+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8YSBocmVmPSJtYWlsdG86RGV2ZWxAb3ZpcnQub3JnIgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIgbW96LWRvLW5vdC1zZW5kPSJ0cnVl Ij5EZXZlbEBvdmlydC5vcmc8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg PGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cDovL2xpc3RzLm92 aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RldmVsIgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5v dmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL2RldmVsPC9hPjwvYmxvY2txdW90ZT4K ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAg ICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8 YnI+CiAgICAgICAgICA8YnIgY2xlYXI9ImFsbCI+CiAgICAgICAgICA8YnI+CiAgICAgICAg ICAtLSA8YnI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEt c21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPkRpZGk8YnI+CiAgICAgICAgICA8L2Rpdj4K ICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8cD48 YnI+CiAgICA8L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg== --------------D2B187A49A90F623211FB3B4--
On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron@redhat.com> wrote:
we had a failure on 002_bootstrap.verify_add_hosts but the error is on imageio
I looked at the host log that Nir added and I can only see that the address is in use which seems to be the same issue we have in initialize engine.
http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/4205/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/ lago-basic-suite-master-host-0/_var_log/ovirt-imageio-daemon/
I cannot see anything in host-deploy. Didi, would we be able to see anything here?
Sorry, seems like my plugin is not enough. Will have a look.
Thanks, Dafna
On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron@redhat.com> wrote:
this is the plugin info from steup log but I don't see anything more than we have seen except a timeout.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/4194/artifact/exported-artifacts/upgrade-from-release- suit-master-el7/test_logs/upgrade-from-release-suite- master/post-001_initialize_engine.py/lago-upgrade-from- release-suite-master-engine/_var_log/ovirt-engine/setup/ ovirt-engine-setup-20171128123116-mmjen3.log
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
*Hi,
*
*We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log. *
*I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches: restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/ <https://gerrit.ovirt.org/#/c/84814/>
Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/>
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/%28root%29/001_initialize_engine/test_initialize_engine/> *
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/>
(Relevant) error snippet from the log: <error> *From messages log
*
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. </error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron@redhat.com> wrote:
we had a failure on 002_bootstrap.verify_add_hosts but the error is on imageio
I looked at the host log that Nir added and I can only see that the address is in use which seems to be the same issue we have in initialize engine.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifact/...
I cannot see anything in host-deploy. Didi, would we be able to see anything here?
Sorry, seems like my plugin is not enough. Will have a look.
Now merged an updated plugin, should hopefully pass changequeue soon. Let's see what happens next time a service fails. Search engine-setup/host-deploy logs for 'tcp connections'. Best regards,
Thanks, Dafna
On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron@redhat.com> wrote:
this is the plugin info from steup log but I don't see anything more than we have seen except a timeout.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/...
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
Hi,
We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log.
I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor...
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/...
(Relevant) error snippet from the log:
<error>
From messages log
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
-- Didi
On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron@redhat.com> wrote:
we had a failure on 002_bootstrap.verify_add_hosts but the error is on imageio
I looked at the host log that Nir added and I can only see that the address is in use which seems to be the same issue we have in initialize engine.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifact/...
I cannot see anything in host-deploy. Didi, would we be able to see anything here?
Sorry, seems like my plugin is not enough. Will have a look. Now merged an updated plugin, should hopefully pass changequeue soon. Let's see what happens next time a service fails. Search engine-setup/host-deploy logs for 'tcp connections'.
Best regards,
On 11/30/2017 03:03 PM, Yedidyah Bar David wrote: thank you :) will update once we have another failure.
Thanks, Dafna
On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron@redhat.com> wrote:
this is the plugin info from steup log but I don't see anything more than we have seen except a timeout.
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/...
Didi, is there anywhere else I should look?
Sadly, as already replied, not yet. Hopefully next time...
On 11/29/2017 10:18 AM, Nir Soffer wrote:
Do we have more info from Didi's debug plugin now?
On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron@redhat.com> wrote:
Hi,
We have failed cq with ovirt-imageio failing to start on upgrade suite. I can still only see errors in the messages log.
I'm writing the reported patch but I don't think it has anything to do with this issue.
Link and headline of suspected patches:
restapi: Enable update to no default network provider of cluster - https://gerrit.ovirt.org/#/c/84814/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRepor...
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/...
(Relevant) error snippet from the log:
<error>
From messages log
Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt Engine Data Warehouse... Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Started oVirt Engine Data Warehouse. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Reloading. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy... Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last): Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind() Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args) Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed. Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
</error>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
participants (3)
-
Dafna Ron -
Nir Soffer -
Yedidyah Bar David