New Issue - Ovirt Status shows '?' or Down for all elements
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron. We are running version oVirt Engine Version: 3.5.0.1-1.el6. We are chasing it down and see this in the log: 2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing Any insight into what might be causing this or the fix would be appreciated. *** *Mark Steele*
Two other symptoms: - Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine - Seeing this in the log: 2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ? Best Regards, Strahil Nikolov
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one. It's production and we cannot take it down unless absolutely necessary. *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
More information: - the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com> wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
More information:
- the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com> wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues. I hope someone from the community can assist you. My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml Best Regards, Strahil Nikolov
All, We've read some things stating to try running the engine-setup again to get the certs inline. That is failing due to missing repo's - does anyone know of alternative or archived repo's we could point at to get past that part? [root@ovirt-01 ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Stage: Environment packages setup [ ERROR ] Yum Cannot queue package iproute: Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ ERROR ] Failed to execute stage 'Environment packages setup': Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ INFO ] Yum Performing yum transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200402072821-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Thu, Apr 2, 2020 at 12:43 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
More information:
- the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com> wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues.
I hope someone from the community can assist you.
My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml
Best Regards, Strahil Nikolov
As a followup - is it possible to move the hosted-engine to another HV manually? Meaning we shutdown the hosted-engine VM, and copy it's XML to another HV and fire it up over there where libvirt IS running? If that is possible, where do we find the live / running XML files on a host? Mark *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Thu, Apr 2, 2020 at 7:35 AM Mark Steele <msteele@telvue.com> wrote:
All,
We've read some things stating to try running the engine-setup again to get the certs inline. That is failing due to missing repo's - does anyone know of alternative or archived repo's we could point at to get past that part?
[root@ovirt-01 ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Stage: Environment packages setup [ ERROR ] Yum Cannot queue package iproute: Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ ERROR ] Failed to execute stage 'Environment packages setup': Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ INFO ] Yum Performing yum transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200402072821-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Thu, Apr 2, 2020 at 12:43 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
More information:
- the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com> wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
> Yesterday we had a storage crash that caused our hosted engine to stop and > not auto-restart. We manually restarted it and this morning, all elements > are showing a status of either question mark '?' or a red down chevron. > > We are running version oVirt Engine Version: 3.5.0.1-1.el6. > > We are chasing it down and see this in the log: > > 2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing > > > Any insight into what might be causing this or the fix would be > appreciated. > > *** > *Mark Steele* > >
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues.
I hope someone from the community can assist you.
My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml
Best Regards, Strahil Nikolov
On April 2, 2020 3:26:14 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote:
As a followup - is it possible to move the hosted-engine to another HV manually? Meaning we shutdown the hosted-engine VM, and copy it's XML to another HV and fire it up over there where libvirt IS running?
If that is possible, where do we find the live / running XML files on a host?
Mark
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Thu, Apr 2, 2020 at 7:35 AM Mark Steele <msteele@telvue.com> wrote:
All,
We've read some things stating to try running the engine-setup again to get the certs inline. That is failing due to missing repo's - does anyone know of alternative or archived repo's we could point at to get past that part?
[root@ovirt-01 ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Stage: Environment packages setup [ ERROR ] Yum Cannot queue package iproute: Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ ERROR ] Failed to execute stage 'Environment packages setup': Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ INFO ] Yum Performing yum transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200402072821-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Thu, Apr 2, 2020 at 12:43 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
More information:
- the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com> wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote: >Two other symptoms: > >- Some of the HV's respond 'Destination Host Prohibited' on
On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele@telvue.com> wrote: ping,
but
>SSH >works fine > >- Seeing this in the log: > >2020-04-01 10:34:31,839 ERROR >[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) >Unable to process messages: javax.net.ssl.SSLException: Received fatal >alert: certificate_expired > > >*** >*Mark Steele* >CIO / VP Technical Operations | TelVue Corporation >TelVue - We Share Your Vision >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >800.885.8886 | msteele@telvue.com | http://www.telvue.com >twitter: http://twitter.com/telvue | facebook: >https://www.facebook.com/telvue > > >On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote: > >> Yesterday we had a storage crash that caused our hosted engine to >stop and >> not auto-restart. We manually restarted it and this morning, all >elements >> are showing a status of either question mark '?' or a red down >chevron. >> >> We are running version oVirt Engine Version: 3.5.0.1-1.el6. >> >> We are chasing it down and see this in the log: >> >> 2020-04-01 10:15:53,060 WARN >[org.ovirt.engine.core.vdsbroker.VdsManager] >(DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = >e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, >continuing >> >> >> Any insight into what might be causing this or the fix would be >> appreciated. >> >> *** >> *Mark Steele* >> >>
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues.
I hope someone from the community can assist you.
My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml
Best Regards, Strahil Nikolov
Theoretically everything is possible, but as I'm trying to be a realist you can find the ovirt 3.6 repos somewhere on the web. Actually, I think you need the CentOS repos which should be in the Vault. Actually, you can get the HE vm's xml from the vdsm logs on the host it has started (and based on your experience - it was a long time ago). Also, the config (at least on v4.3) is in the OVF that the ovirt-ha-agent is extracting. Another approach is to get the xml via virsh (if it was running) and then moving the diak and xml to a KVM host is like a walk in the park. Best Rergards, Strahil Nikolov
There are ovirt and an ovirt-dependencies repos. You need both. Eric Evans Digital Data Services LLC. 304.660.9080 From: Mark Steele <msteele@telvue.com> Sent: Thursday, April 02, 2020 7:36 AM To: Strahil Nikolov <hunter86_bg@yahoo.com> Cc: users <users@ovirt.org>; Ian Easter <ieaster@telvue.com> Subject: [ovirt-users] Re: New Issue - Ovirt Status shows '?' or Down for all elements All, We've read some things stating to try running the engine-setup again to get the certs inline. That is failing due to missing repo's - does anyone know of alternative or archived repo's we could point at to get past that part? [root@ovirt-01 ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Stage: Environment packages setup [ ERROR ] Yum Cannot queue package iproute: Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ ERROR ] Failed to execute stage 'Environment packages setup': Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic [ INFO ] Yum Performing yum transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200402072821-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed *** Mark Steele CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com <mailto:msteele@telvue.com> | <http://www.telvue.com/> http://www.telvue.com twitter: <http://twitter.com/telvue> http://twitter.com/telvue | facebook: <https://www.facebook.com/telvue> https://www.facebook.com/telvue On Thu, Apr 2, 2020 at 12:43 AM Strahil Nikolov <hunter86_bg@yahoo.com <mailto:hunter86_bg@yahoo.com> > wrote: On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele@telvue.com <mailto:msteele@telvue.com> > wrote:
More information:
- the host that ovirt hosted engine is running on is not running libvirtd. When we attempt to start it, it complains about expired certs and fails. Any ideas on this new development?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com <mailto:msteele@telvue.com> | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele@telvue.com <mailto:msteele@telvue.com> > wrote:
A million years ago - we are going to be upgrading with a new cluster but for now, we have this one.
It's production and we cannot take it down unless absolutely necessary.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com <mailto:msteele@telvue.com> | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov <hunter86_bg@yahoo.com <mailto:hunter86_bg@yahoo.com> > wrote:
On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele@telvue.com <mailto:msteele@telvue.com> > wrote:
Two other symptoms:
- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine
- Seeing this in the log:
2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele@telvue.com <mailto:msteele@telvue.com> | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com <mailto:msteele@telvue.com> > wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.
We are running version oVirt Engine Version: 3.5.0.1-1.el6.
We are chasing it down and see this in the log:
2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing
Any insight into what might be causing this or the fix would be appreciated.
*** *Mark Steele*
Hey Mark, When was the oVirt last updated ?
Best Regards, Strahil Nikolov
That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues. I hope someone from the community can assist you. My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml Best Regards, Strahil Nikolov
participants (4)
-
eevans＠digitaldatatechs.com -
Mark Steele -
Strahil Nikolov -
Николаев Алексей