Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron. We are running version oVirt Engine Version: 3.5.0.1-1.el6. We are chasing it down and see this in the log: 2020-04-01 10:15:53,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing Any insight into what might be causing this or the fix would be appreciated. *** *Mark Steele*

On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele <msteele(a)telvue.com&gt; wrote: >Two other symptoms: > >- Some of the HV's respond 'Destination Host Prohibited' on ping, but >SSH >works fine > >- Seeing this in the log: > >2020-04-01 10:34:31,839 ERROR >[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) >Unable to process messages: javax.net.ssl.SSLException: Received fatal >alert: certificate_expired > > >*** >*Mark Steele* >CIO / VP Technical Operations | TelVue Corporation >TelVue - We Share Your Vision >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >twitter: http://twitter.com/telvue | facebook: >https://www.facebook.com/telvue > > >On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele(a)telvue.com&gt; wrote: > >> Yesterday we had a storage crash that caused our hosted engine to >stop and >> not auto-restart. We manually restarted it and this morning, all >elements >> are showing a status of either question mark '?' or a red down >chevron. >> >> We are running version oVirt Engine Version: 3.5.0.1-1.el6. >> >> We are chasing it down and see this in the log: >> >> 2020-04-01 10:15:53,060 WARN >[org.ovirt.engine.core.vdsbroker.VdsManager] >(DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = >e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, >continuing >> >> >> Any insight into what might be causing this or the fix would be >> appreciated. >> >> *** >> *Mark Steele* >> >> Hey Mark, When was the oVirt last updated ? Best Regards, Strahil Nikolov

On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele(a)telvue.com&gt; wrote: >More information: > >- the host that ovirt hosted engine is running on is not running >libvirtd. >When we attempt to start it, it complains about expired certs and >fails. >Any ideas on this new development? > >*** >*Mark Steele* >CIO / VP Technical Operations | TelVue Corporation >TelVue - We Share Your Vision >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >twitter: http://twitter.com/telvue | facebook: >https://www.facebook.com/telvue > > >On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele(a)telvue.com&gt; wrote: > >> A million years ago - we are going to be upgrading with a new cluster >but >> for now, we have this one. >> >> It's production and we cannot take it down unless absolutely >necessary. >> >> *** >> *Mark Steele* >> CIO / VP Technical Operations | TelVue Corporation >> TelVue - We Share Your Vision >> 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >> 800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >> twitter: http://twitter.com/telvue | facebook: >> https://www.facebook.com/telvue >> >> >> On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> wrote: >> >>> On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele &gt;&lt;msteele(a)telvue.com&gt; >>> wrote: >>> >Two other symptoms: >>> > >>> >- Some of the HV's respond 'Destination Host Prohibited' on ping, >but >>> >SSH >>> >works fine >>> > >>> >- Seeing this in the log: >>> > >>> >2020-04-01 10:34:31,839 ERROR >>> >[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp >Reactor) >>> >Unable to process messages: javax.net.ssl.SSLException: Received >fatal >>> >alert: certificate_expired >>> > >>> > >>> >*** >>> >*Mark Steele* >>> >CIO / VP Technical Operations | TelVue Corporation >>> >TelVue - We Share Your Vision >>> >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >>> >800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >>> >twitter: http://twitter.com/telvue | facebook: >>> >https://www.facebook.com/telvue >>> > >>> > >>> >On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele(a)telvue.com&gt; >wrote: >>> > >>> >> Yesterday we had a storage crash that caused our hosted engine to >>> >stop and >>> >> not auto-restart. We manually restarted it and this morning, all >>> >elements >>> >> are showing a status of either question mark '?' or a red down >>> >chevron. >>> >> >>> >> We are running version oVirt Engine Version: 3.5.0.1-1.el6. >>> >> >>> >> We are chasing it down and see this in the log: >>> >> >>> >> 2020-04-01 10:15:53,060 WARN >>> >[org.ovirt.engine.core.vdsbroker.VdsManager] >>> >(DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = >>> >e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, >>> >continuing >>> >> >>> >> >>> >> Any insight into what might be causing this or the fix would be >>> >> appreciated. >>> >> >>> >> *** >>> >> *Mark Steele* >>> >> >>> >> >>> >>> Hey Mark, >>> When was the oVirt last updated ? >>> >>> Best Regards, >>> Strahil Nikolov >>> >> That's why I asked about the update. The update process renews somee certificates, and as it wasn't updated for a long time - now you have issues. I hope someone from the community can assist you. My recommendations: 1. Check on all hosts all vdsm.log and extract the VMs' xml 2. Get all network's xml Best Regards, Strahil Nikolov

As a followup - is it possible to move the hosted-engine to another HV manually? Meaning we shutdown the hosted-engine VM, and copy it's XML to another HV and fire it up over there where libvirt IS running? If that is possible, where do we find the live / running XML files on a host? Mark *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 | msteele(a)telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Thu, Apr 2, 2020 at 7:35 AM Mark Steele <msteele(a)telvue.com&gt; wrote: > All, > > We've read some things stating to try running the engine-setup again to > get the certs inline. That is failing due to missing repo's - does anyone > know of alternative or archived repo's we could point at to get past that > part? > > [root@ovirt-01 ~]# engine-setup > [ INFO ] Stage: Initializing > [ INFO ] Stage: Environment setup > Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] > Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log > Version: otopi-1.3.0 (otopi-1.3.0-1.el6) > [ INFO ] Stage: Environment packages setup > [ ERROR ] Yum Cannot queue package iproute: Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic > [ ERROR ] Failed to execute stage 'Environment packages setup': Cannot find a valid baseurl for repo: ovirt-3.5-jpackage-6.0-generic > [ INFO ] Yum Performing yum transaction rollback > [ INFO ] Stage: Clean up > Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200402072811-h72nc0.log > [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200402072821-setup.conf' > [ INFO ] Stage: Pre-termination > [ INFO ] Stage: Termination > [ ERROR ] Execution of setup failed > > > *** > *Mark Steele* > CIO / VP Technical Operations | TelVue Corporation > TelVue - We Share Your Vision > 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 > 800.885.8886 | msteele(a)telvue.com | http://www.telvue.com > twitter: http://twitter.com/telvue | facebook: > https://www.facebook.com/telvue > > > On Thu, Apr 2, 2020 at 12:43 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > wrote: > >> On April 1, 2020 11:21:34 PM GMT+03:00, Mark Steele <msteele(a)telvue.com&gt; >> wrote: >> >More information: >> > >> >- the host that ovirt hosted engine is running on is not running >> >libvirtd. >> >When we attempt to start it, it complains about expired certs and >> >fails. >> >Any ideas on this new development? >> > >> >*** >> >*Mark Steele* >> >CIO / VP Technical Operations | TelVue Corporation >> >TelVue - We Share Your Vision >> >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >> >800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >> >twitter: http://twitter.com/telvue | facebook: >> >https://www.facebook.com/telvue >> > >> > >> >On Wed, Apr 1, 2020 at 1:43 PM Mark Steele <msteele(a)telvue.com&gt; wrote: >> > >> >> A million years ago - we are going to be upgrading with a new cluster >> >but >> >> for now, we have this one. >> >> >> >> It's production and we cannot take it down unless absolutely >> >necessary. >> >> >> >> *** >> >> *Mark Steele* >> >> CIO / VP Technical Operations | TelVue Corporation >> >> TelVue - We Share Your Vision >> >> 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >> >> 800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >> >> twitter: http://twitter.com/telvue | facebook: >> >> https://www.facebook.com/telvue >> >> >> >> >> >> On Wed, Apr 1, 2020 at 1:17 PM Strahil Nikolov >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> wrote: >> >> >> >>> On April 1, 2020 5:40:27 PM GMT+03:00, Mark Steele >> &gt;&lt;msteele(a)telvue.com&gt; >> >>> wrote: >> >>> >Two other symptoms: >> >>> > >> >>> >- Some of the HV's respond 'Destination Host Prohibited' on ping, >> >but >> >>> >SSH >> >>> >works fine >> >>> > >> >>> >- Seeing this in the log: >> >>> > >> >>> >2020-04-01 10:34:31,839 ERROR >> >>> >[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp >> >Reactor) >> >>> >Unable to process messages: javax.net.ssl.SSLException: Received >> >fatal >> >>> >alert: certificate_expired >> >>> > >> >>> > >> >>> >*** >> >>> >*Mark Steele* >> >>> >CIO / VP Technical Operations | TelVue Corporation >> >>> >TelVue - We Share Your Vision >> >>> >16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 >> >>> >800.885.8886 | msteele(a)telvue.com | http://www.telvue.com >> >>> >twitter: http://twitter.com/telvue | facebook: >> >>> >https://www.facebook.com/telvue >> >>> > >> >>> > >> >>> >On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele(a)telvue.com&gt; >> >wrote: >> >>> > >> >>> >> Yesterday we had a storage crash that caused our hosted engine to >> >>> >stop and >> >>> >> not auto-restart. We manually restarted it and this morning, all >> >>> >elements >> >>> >> are showing a status of either question mark '?' or a red down >> >>> >chevron. >> >>> >> >> >>> >> We are running version oVirt Engine Version: 3.5.0.1-1.el6. >> >>> >> >> >>> >> We are chasing it down and see this in the log: >> >>> >> >> >>> >> 2020-04-01 10:15:53,060 WARN >> >>> >[org.ovirt.engine.core.vdsbroker.VdsManager] >> >>> >(DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = >> >>> >e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, >> >>> >continuing >> >>> >> >> >>> >> >> >>> >> Any insight into what might be causing this or the fix would be >> >>> >> appreciated. >> >>> >> >> >>> >> *** >> >>> >> *Mark Steele* >> >>> >> >> >>> >> >> >>> >> >>> Hey Mark, >> >>> When was the oVirt last updated ? >> >>> >> >>> Best Regards, >> >>> Strahil Nikolov >> >>> >> >> >> >> That's why I asked about the update. >> The update process renews somee certificates, and as it wasn't updated >> for a long time - now you have issues. >> >> I hope someone from the community can assist you. >> >> My recommendations: >> 1. Check on all hosts all vdsm.log and extract the VMs' xml >> 2. Get all network's xml >> >> >> >> Best Regards, >> Strahil Nikolov >> >