Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]

On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >> we had a failure on 002_bootstrap.verify_add_hosts but the error is on >> imageio >> >> I looked at the host log that Nir added and I can only see that the >> address is in use which seems to be the same issue we have in initialize >> engine. >> >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifa... >> >> I cannot see anything in host-deploy. >> Didi, would we be able to see anything here? > > Sorry, seems like my plugin is not enough. Will have a look. Now merged an updated plugin, should hopefully pass changequeue soon. Let's see what happens next time a service fails. Search engine-setup/host-deploy logs for 'tcp connections'. Best regards,

>> >> Thanks, >> Dafna >> >> >> >> On 11/29/2017 11:03 AM, Yedidyah Bar David wrote: >> >> On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>> this is the plugin info from steup log but I don't see anything more than >>> we have seen except a timeout. >>> >>> https://pastebin.com/QVtNRNWV >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa... >>> >>> Didi, is there anywhere else I should look? >> >> Sadly, as already replied, not yet. Hopefully next time... >> >>> >>> >>> On 11/29/2017 10:18 AM, Nir Soffer wrote: >>> >>> Do we have more info from Didi's debug plugin now? >>> >>> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> Hi, >>>> >>>> We have failed cq with ovirt-imageio failing to start on upgrade suite. >>>> I can still only see errors in the messages log. >>>> >>>> I'm writing the reported patch but I don't think it has anything to do >>>> with this issue. >>>> >>>> Link and headline of suspected patches: >>>> >>>> restapi: Enable update to no default network provider of cluster - >>>> https://gerrit.ovirt.org/#/c/84814/ >>>> >>>> Link to Job: >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ >>>> >>>> Link to all logs: >>>> >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ >>>> >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testRe... >>>> >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifa... >>>> >>>> (Relevant) error snippet from the log: >>>> >>>> <error> >>>> >>>> From messages log >>>> >>>> >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Started oVirt Engine. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Reloading. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked >>>> executable. Please remove executable permission bits. Proceeding anyway. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Starting oVirt Engine Data Warehouse... >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Started oVirt Engine Data Warehouse. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Reloading. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked >>>> executable. Please remove executable permission bits. Proceeding anyway. >>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>> Starting oVirt ImageIO Proxy... >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: Traceback (most recent call last): >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in >>>> <module> >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: status = image_proxy.main(args, config) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File >>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line >>>> 21, in main >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: image_server.start(config) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File >>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, >>>> in start >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: WSGIRequestHandler) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, >>>> in __init__ >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: self.server_bind() >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", >>>> line 48, in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: HTTPServer.server_bind(self) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line >>>> 108, in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, >>>> in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: self.socket.bind(self.server_address) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in >>>> meth >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service: main process exited, code=exited, >>>> status=1/FAILURE >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> Failed to start oVirt ImageIO Proxy. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> Unit ovirt-imageio-proxy.service entered failed state. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service failed. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> Starting oVirt ImageIO Proxy... >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: Traceback (most recent call last): >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in >>>> <module> >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: status = image_proxy.main(args, config) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File >>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line >>>> 21, in main >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: image_server.start(config) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File >>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, >>>> in start >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: WSGIRequestHandler) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419, >>>> in __init__ >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: self.server_bind() >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", >>>> line 48, in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: HTTPServer.server_bind(self) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line >>>> 108, in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430, >>>> in server_bind >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: self.socket.bind(self.server_address) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in >>>> meth >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args) >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service: main process exited, code=exited, >>>> status=1/FAILURE >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> Failed to start oVirt ImageIO Proxy. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> Unit ovirt-imageio-proxy.service entered failed state. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service failed. >>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart. >>>> >>>> </error> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >> -- >> Didi >> >> > > > -- > Didi