I think we need to check and report which  process is listening on a port
when starting a server on that port fail.

Didi, do you think we can integrate this in the deploy code, or this
should be implemented in each server?

Maybe when deployment fails, the deploy code can report all the
listening sockets and the processes bound to these sockets?

Nir

On Sun, Nov 26, 2017 at 7:11 PM Gal Ben Haim <gbenhaim@redhat.com> wrote:
The failure is not consistent.

On Sun, Nov 26, 2017 at 5:33 PM, Yaniv Kaul <ykaul@redhat.com> wrote:


On Sun, Nov 26, 2017 at 4:53 PM, Gal Ben Haim <gbenhaim@redhat.com> wrote:
We still see this issue on the upgrade suite from latest release to master [1].
I don't see any evidence in "/var/log/messages" [2] that "ovirt-imageio-proxy" was started twice.

Since it's not a registered port and a high port, could it be used by something else (what are the odds though ?
Is it consistent?
Y.
 

On Fri, Nov 24, 2017 at 8:16 PM, Dafna Ron <dron@redhat.com> wrote:
there were two different patches reported as failing cq today with the ovirt-imageio-proxy service failing to start.

Here is the latest failure: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4130/artifact




On 11/23/2017 03:39 PM, Allon Mureinik wrote:
Daniel/Nir?

On Thu, Nov 23, 2017 at 5:29 PM, Dafna Ron <dron@redhat.com> wrote:

Hi,

We have a failing on test 001_initialize_engine.test_initialize_engine.

This is failing with error Failed to start service 'ovirt-imageio-proxy


Link and headline ofto suspected patches:

build: Make resulting RPMs architecture-specific - https://gerrit.ovirt.org/#/c/84534/


Link to Job:

http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055


Link to all logs:

http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/

http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4055/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/


(Relevant) error snippet from the log:

<error>


from lago log: 
Failed to start service 'ovirt-imageio-proxy

messages logs: 

Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting Session 8 of user root.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last):
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind()
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:47 lago-upgrade-from-release-suite-master-engine systemd: Starting oVirt ImageIO Proxy...
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: Traceback (most recent call last):
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in <module>
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: status = image_proxy.main(args, config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line 21, in main
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: image_server.start(config)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, in start
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: WSGIRequestHandler)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.server_bind()
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 48, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: HTTPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: self.socket.bind(self.server_address)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in meth
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service holdoff time over, scheduling restart.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: start request repeated too quickly for ovirt-imageio-proxy.service
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Failed to start oVirt ImageIO Proxy.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: Unit ovirt-imageio-proxy.service entered failed state.
Nov 23 07:30:48 lago-upgrade-from-release-suite-master-engine systemd: ovirt-imageio-proxy.service failed.

</error>



_______________________________________________
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra




_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel



--
GAL bEN HAIM
RHV DEVOPS

_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel




--
GAL bEN HAIM
RHV DEVOPS
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel