[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 20-11-2017 ] [ 001_initialize_engine.test_initialize_engine ]

Dafna Ron dron at redhat.com
Thu Nov 30 15:38:55 UTC 2017 

On 11/30/2017 03:03 PM, Yedidyah Bar David wrote:
> On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <didi at redhat.com> wrote:
>> On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <dron at redhat.com> wrote:
>>> we had a failure on 002_bootstrap.verify_add_hosts but the error is on
>>> imageio
>>>
>>> I looked at the host log that Nir added and I can only see that the
>>> address is in use which seems to be the same issue we have in initialize
>>> engine.
>>>
>>>
>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host-0/_var_log/ovirt-imageio-daemon/
>>>
>>> I cannot see anything in host-deploy.
>>> Didi, would we be able to see anything here?
>>
>> Sorry, seems like my plugin is not enough. Will have a look.
> Now merged an updated plugin, should hopefully pass changequeue soon.
> Let's see what happens next time a service fails.
> Search engine-setup/host-deploy logs for 'tcp connections'.
>
> Best regards,
thank you :)
will update once we have another failure.
>>>
>>> Thanks,
>>> Dafna
>>>
>>>
>>>
>>> On 11/29/2017 11:03 AM, Yedidyah Bar David wrote:
>>>
>>> On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <dron at redhat.com> wrote:
>>>> this is the plugin info from steup log but I don't see anything more than
>>>> we have seen except a timeout.
>>>>
>>>> https://pastebin.com/QVtNRNWV
>>>>
>>>>
>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20171128123116-mmjen3.log
>>>>
>>>> Didi, is there anywhere else I should look?
>>>
>>> Sadly, as already replied, not yet. Hopefully next time...
>>>
>>>>
>>>>
>>>> On 11/29/2017 10:18 AM, Nir Soffer wrote:
>>>>
>>>> Do we have more info from Didi's debug plugin now?
>>>>
>>>> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <dron at redhat.com> wrote:
>>>>> Hi,
>>>>>
>>>>> We have failed cq with ovirt-imageio failing to start on upgrade suite.
>>>>> I can still only see errors in the messages log.
>>>>>
>>>>> I'm writing the reported patch but I don't think it has anything to do
>>>>> with this issue.
>>>>>
>>>>> Link and headline of suspected patches:
>>>>>
>>>>> restapi: Enable update to no default network provider of cluster -
>>>>> https://gerrit.ovirt.org/#/c/84814/
>>>>>
>>>>> Link to Job:
>>>>>
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/
>>>>>
>>>>> Link to all logs:
>>>>>
>>>>>
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/
>>>>>
>>>>>
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/
>>>>>
>>>>>
>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/
>>>>>
>>>>> (Relevant) error snippet from the log:
>>>>>
>>>>> <error>
>>>>>
>>>>> From messages log
>>>>>
>>>>>
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Started oVirt Engine.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Reloading.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>>>> executable. Please remove executable permission bits. Proceeding anyway.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Starting oVirt Engine Data Warehouse...
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Started oVirt Engine Data Warehouse.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Reloading.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked
>>>>> executable. Please remove executable permission bits. Proceeding anyway.
>>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Starting oVirt ImageIO Proxy...
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
>>>>> <module>
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>>> 21, in main
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>>> in start
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419,
>>>>> in __init__
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>>> line 48, in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>>> 108, in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430,
>>>>> in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in
>>>>> meth
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>> status=1/FAILURE
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Failed to start oVirt ImageIO Proxy.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service failed.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Starting oVirt ImageIO Proxy...
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: Traceback (most recent call last):
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in
>>>>> <module>
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", line
>>>>> 21, in main
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: image_server.start(config)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File
>>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45,
>>>>> in start
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: WSGIRequestHandler)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 419,
>>>>> in __init__
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.server_bind()
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py",
>>>>> line 48, in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line
>>>>> 108, in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line 430,
>>>>> in server_bind
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in
>>>>> meth
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args)
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine
>>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service: main process exited, code=exited,
>>>>> status=1/FAILURE
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Failed to start oVirt ImageIO Proxy.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> Unit ovirt-imageio-proxy.service entered failed state.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service failed.
>>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd:
>>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart.
>>>>>
>>>>> </error>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>>>
>>> --
>>> Didi
>>>
>>>
>>
>>
>> --
>> Didi
>
>

More information about the Devel mailing list