[ovirt-users] Re: 503 Service Unavailable

Apologies, for got to add the below in the previous mail. I do see a few of these “Bad Gateway” errors in the engine.log as well, not sure what this means though: [root@virt ovirt-engine]# tail -f engine.log 2020-05-12 09:48:03,941+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [52bf537e] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}' 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 16 threads waiting for tasks and 0 tasks in queue. 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2020-05-12 09:53:00,274+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads waiting for tasks. 2020-05-12 09:53:03,951+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}' 2020-05-12 09:53:03,959+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Running command: SyncNetworkProviderCommand internal: true. 2020-05-12 09:53:04,018+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050) 2020-05-12 09:53:04,019+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}' 2020-05-12 09:58:04,028+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}' 2020-05-12 09:58:04,036+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Running command: SyncNetworkProviderCommand internal: true. 2020-05-12 09:58:04,327+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050) 2020-05-12 09:58:04,328+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}' Anton Louw Cloud Engineer: Storage and Virtualization at Vox ________________________________ T: 087 805 0000 | D: 087 805 1572 M: N/A E: anton.louw(a)voxtelecom.co.za A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg www.vox.co.za From: Anton Louw Sent: 12 May 2020 10:00 To: Yedidyah Bar David <didi(a)redhat.com&gt; Cc: users(a)ovirt.org Subject: RE: [ovirt-users] 503 Service Unavailable Thanks for the reply Didi. I have checked the logs, but it does not seem to show anything when I try and access the WebUI. I run tail -f engine.log and tail -f server.log, but it remains still. This is the same case with the httpd logs.

I checked the status of the engine from my node, and it displays the below: [root@jb2-client-node01 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host jb2-client-node01.kvm.voxvm.co.za (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : jb2-client-node01.kvm.voxvm.co.za Host ID : 1 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}

Score : 3400 stopped : False Local maintenance : False crc32 : 75d3e273 local_conf_timestamp : 3547513 Host timestamp : 3547513 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3547513 (Tue May 12 09:43:29 2020) host-id=1 score=3400 vm_conf_refresh_time=3547513 (Tue May 12 09:43:29 2020) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False Everything network related seems to run fine, I can connect to the engine via ssh etc. as well. From: Yedidyah Bar David <didi(a)redhat.com&gt; Sent: 12 May 2020 09:26 To: Anton Louw <Anton.Louw(a)voxtelecom.co.za&gt; Cc: users(a)ovirt.org Subject: Re: [ovirt-users] 503 Service Unavailable On Tue, May 12, 2020 at 9:53 AM Anton Louw via Users <users(a)ovirt.org&gt; wrote: > > > > Hi All, > > > > So due to some bad decisions etc. I have accidently removed the hosted engine setup by running engine-cleanup. I initially just wanted to remove the websocket proxy, and as advised by an article, this can be done by running engine-cleanup. Obviously this was not the case, and it removed everything. Not necessarily everything, but even if you ask to remove only something specific, it does more than that. We recently decided this is too risky and removed that question: https://bugzilla.redhat.com/show_bug.cgi?id=1792874 > > > > I at least had a backup that I took, so I restored the backup to the engine, and ran engine-setup. Everything runs fine, but when I try and access the webUI, I get an error: “503 Service Unavailable” “No server is available to handle this request” > > > > I have checked the ovirt-engine service, and it is running, as well as the httpd service. > > > > I have also attached the engine log as well as the server log. 1. When attaching complete logs (thanks for that!), please state the exact time you tried to do something that failed (access the web ui, in this case). Your attached logs have several different ERRORs, which might be due to unrelated things, and it's hard to tell where to look first. Once found, going backwards to try to find the root cause is usually easier. 2. Please check/share also httpd logs, and also /etc/httpd/conf.d - you can compare it with a fresh engine you can setup on some test vm somewhere, it should be mostly the same. 3. Also please check/share engine-setup logs, at /var/log/ovirt-engine/setup. You can search there for httpd or apache. 4. I have a feeling, just a guess, that you are missing /etc/httpd/conf.d/z-ovirt-engine-proxy.conf . Not sure what might have caused that. setup after cleanup should have created it. Best regards, > > > > Does anybody have any idea where I can check, or perhaps I missed something. > > > > Thank you > > > Anton Louw > Cloud Engineer: Storage and Virtualization at Vox > ________________________________ > T: 087 805 0000 | D: 087 805 1572 > M: N/A > E: anton.louw(a)voxtelecom.co.za > A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg > www.vox.co.za > > > > > > > > > Disclaimer > > The contents of this email are confidential to the sender and the intended recipient. Unless the contents are clearly and entirely of a personal nature, they are subject to copyright in favour of the holding company of the Vox group of companies. Any recipient who receives this email in error should immediately report the error to the sender and permanently delete this email from all storage devices. > > This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here. > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDUEFMIUCOM... -- Didi