Hi Didi,

 

So still not sure what the issue was. I did the clean-up again, restored from the backup and it seems to be working again after a few tweaks.

 

Thanks

 


Anton Louw
Cloud Engineer: Storage and Virtualization at Vox
T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

F
 
T
 
I
 
L
 
Y
 

From: Yedidyah Bar David <didi@redhat.com>
Sent: 12 May 2020 10:20
To: Anton Louw <Anton.Louw@voxtelecom.co.za>
Cc: users@ovirt.org
Subject: Re: [ovirt-users] 503 Service Unavailable

 

On Tue, May 12, 2020 at 11:02 AM Anton Louw <Anton.Louw@voxtelecom.co.za> wrote:
>
>
>
> Apologies, for got to add the below in the previous mail.
>
>
>
> I do see a few of these “Bad Gateway” errors in the engine.log as well, not sure what this means though:
>
>
>
> [root@virt ovirt-engine]# tail -f engine.log
>
> 2020-05-12 09:48:03,941+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [52bf537e] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}'
>
> 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks.
>
> 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 16 threads waiting for tasks and 0 tasks in queue.
>
> 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks.
>
> 2020-05-12 09:53:00,273+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks.
>
> 2020-05-12 09:53:00,274+02 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads waiting for tasks.
>
> 2020-05-12 09:53:03,951+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}'
>
> 2020-05-12 09:53:03,959+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Running command: SyncNetworkProviderCommand internal: true.
>
> 2020-05-12 09:53:04,018+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050)
>
> 2020-05-12 09:53:04,019+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [3546d22f] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}'
>
> 2020-05-12 09:58:04,028+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}'
>
> 2020-05-12 09:58:04,036+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Running command: SyncNetworkProviderCommand internal: true.
>
> 2020-05-12 09:58:04,327+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050)
>
> 2020-05-12 09:58:04,328+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [476ee6aa] Lock freed to object 'EngineLock:{exclusiveLocks='[44725bec-4d4a-496d-bd05-97eff7996a61=PROVIDER]', sharedLocks=''}'
>
>
>
>
> Anton Louw
> Cloud Engineer: Storage and Virtualization at Vox
> ________________________________
> T: 087 805 0000 | D: 087 805 1572
> M: N/A
> E: anton.louw@voxtelecom.co.za
> A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
> www.vox.co.za
>
>
>
>
>
>
>
> From: Anton Louw
> Sent: 12 May 2020 10:00
> To: Yedidyah Bar David <didi@redhat.com>
> Cc: users@ovirt.org
> Subject: RE: [ovirt-users] 503 Service Unavailable
>
>
>
> Thanks for the reply Didi.
>
>
>
> I have checked the logs, but it does not seem to show anything when I try and access the WebUI. I run tail -f engine.log and tail -f server.log, but it remains still. This is the same case with the httpd logs.

If you don't see anything in any of the httpd logs, it means the
request that your browser sent didn't arrive to httpd.
Perhaps indeed you have some networking issue?
Can you ping/ssh/etc from inside the engine some machine outside
(other than the host it's on)?
The other way around - from the engine vm to outside?

If not, you should first fix that.

That said, I'd not expect a 503 in that case, but a timeout on your browser.
So either you didn't check the right log file (or there is some other
problem, such as full disk, iptables, whatever) or some other machine
returns you 503.
You can also check with a sniffer (tcpdump) on the engine machine to
see if it gets traffic, e.g.

tcpdump -i any port 80 or port 443

>
>
>
> I checked the status of the engine from my node, and it displays the below:
>
>
>
> [root@jb2-client-node01 ~]# hosted-engine --vm-status
>
>
>
>
>
> !! Cluster is in GLOBAL MAINTENANCE mode !!
>
>
>
>
>
>
>
> --== Host jb2-client-node01.kvm.voxvm.co.za (id: 1) status ==--
>
>
>
> conf_on_shared_storage : True
>
> Status up-to-date : True
>
> Hostname : jb2-client-node01.kvm.voxvm.co.za
>
> Host ID : 1
>
> Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}

I am not sure the HA services show something different for "Can't
connect to the engine at all" vs "Health page didn't say it's
healthy".

>
> Score : 3400
>
> stopped : False
>
> Local maintenance : False
>
> crc32 : 75d3e273
>
> local_conf_timestamp : 3547513
>
> Host timestamp : 3547513
>
> Extra metadata (valid at timestamp):
>
> metadata_parse_version=1
>
> metadata_feature_version=1
>
> timestamp=3547513 (Tue May 12 09:43:29 2020)
>
> host-id=1
>
> score=3400
>
> vm_conf_refresh_time=3547513 (Tue May 12 09:43:29 2020)
>
> conf_on_shared_storage=True
>
> maintenance=False
>
> state=GlobalMaintenance
>
> stopped=False
>
>
>
> Everything network related seems to run fine, I can connect to the engine via ssh etc. as well.
>
>
>
>
>
>
>
> From: Yedidyah Bar David <didi@redhat.com>
> Sent: 12 May 2020 09:26
> To: Anton Louw <Anton.Louw@voxtelecom.co.za>
> Cc: users@ovirt.org
> Subject: Re: [ovirt-users] 503 Service Unavailable
>
>
>
> On Tue, May 12, 2020 at 9:53 AM Anton Louw via Users <users@ovirt.org> wrote:
> >
> >
> >
> > Hi All,
> >
> >
> >
> > So due to some bad decisions etc. I have accidently removed the hosted engine setup by running engine-cleanup. I initially just wanted to remove the websocket proxy, and as advised by an article, this can be done by running engine-cleanup. Obviously this was not the case, and it removed everything.
>
> Not necessarily everything, but even if you ask to remove only
> something specific, it does more than that. We recently decided this
> is too risky and removed that question:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1792874
>
> >
> >
> >
> > I at least had a backup that I took, so I restored the backup to the engine, and ran engine-setup. Everything runs fine, but when I try and access the webUI, I get an error: “503 Service Unavailable” “No server is available to handle this request”
> >
> >
> >
> > I have checked the ovirt-engine service, and it is running, as well as the httpd service.
> >
> >
> >
> > I have also attached the engine log as well as the server log.
>
> 1. When attaching complete logs (thanks for that!), please state the
> exact time you tried
> to do something that failed (access the web ui, in this case). Your
> attached logs have
> several different ERRORs, which might be due to unrelated things, and
> it's hard to tell
> where to look first. Once found, going backwards to try to find the
> root cause is usually
> easier.
>
> 2. Please check/share also httpd logs, and also /etc/httpd/conf.d -
> you can compare it
> with a fresh engine you can setup on some test vm somewhere, it should
> be mostly the
> same.
>
> 3. Also please check/share engine-setup logs, at
> /var/log/ovirt-engine/setup. You can search
> there for httpd or apache.
>
> 4. I have a feeling, just a guess, that you are missing
> /etc/httpd/conf.d/z-ovirt-engine-proxy.conf .
> Not sure what might have caused that. setup after cleanup should have
> created it.
>
> Best regards,
>
> >
> >
> >
> > Does anybody have any idea where I can check, or perhaps I missed something.
> >
> >
> >
> > Thank you
> >
> >
> > Anton Louw
> > Cloud Engineer: Storage and Virtualization at Vox
> > ________________________________
> > T: 087 805 0000 | D: 087 805 1572
> > M: N/A
> > E: anton.louw@voxtelecom.co.za
> > A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
> > www.vox.co.za
> >
> >
> >
> >
> >
> >
> >
> >
> > Disclaimer
> >
> > The contents of this email are confidential to the sender and the intended recipient. Unless the contents are clearly and entirely of a personal nature, they are subject to copyright in favour of the holding company of the Vox group of companies. Any recipient who receives this email in error should immediately report the error to the sender and permanently delete this email from all storage devices.
> >
> > This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more Click Here.
> >
> >
> >
> > _______________________________________________
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-leave@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDUEFMIUCOMC4RVFGWPSXF257QQZZU5D/
>
>
>
> --
> Didi
>
>


--
Didi