8:41 p.m.
I suppose I just keep striking out on recent oVirt updates. Today (22 June 2019, all
previous updates have been installed successfully) I saw that an update was available for
my Enterprise Linux Host (main oVirt engine, CentOS 7 x64), and I attempted to update it.
The update failed, which had never happened before. I logged in via SSH and saw that the
updates had been halted due to "yum versionlock." I cleared versionlock and
proceeded with the update, which appeared to work successfully. I rebooted the system,
which came up a-ok. However, I can no longer reach the Administration Portal page. The
browser only hangs. I see the following:
- type in the IP address to the server and get the https://<ip
address>/ovirt-engine/sso/oauth/authorize page, which tells me:
"The FQDN used to access the system is not a valid engine FQDN. You must access the
system using the engine FQDN or one of the engine alternate FQDNs.
Click here to continue."
When I click the provided link, I get the same hanging behavior and it never loads the
login page.
- I was able to connect via Cockpit to https://<IP Address>:9090 and log in
successfully as root after SSH'ing in and restarting the engine. There are no major
issues displayed, and I was able to create a Diagnostic Report. Under Hostname > oVirt
Machines, it will actually redirect me to the login page at
https://fqdn/ovirt-engine/sso/login.html. The page will actually load after the redirect,
but when I enter my admin@internal credentials it just hangs and spins.
When I return to "Virtual Machines" in Cockpit, I have
Host/Cluster/Templates/VDSM options, I see "oVirt login in progress" with a
continually spinning circle, never actually able to authenticate.
In /var/log/ovirt-engine/ui.log, I see:
2019-06-22 19:08:02,533-04 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-4) []
Permutation name: C92E6928986552EDD0E1C99CDC0CC8AB
2019-06-22 19:08:02,533-04 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-4) []
Uncaught exception: com.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot
read property 'kh' of null
at
org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider.$lambda$4(AsyncDataProvider.java:387)
at
org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider$lambda$4$Type.executed(AsyncDataProvider.java:387)
at org.ovirt.engine.ui.frontend.Frontend$2.$onFailure(Frontend.java:329)
[frontend.jar:]
at org.ovirt.engine.ui.frontend.Frontend$2.onFailure(Frontend.java:329)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.$onFailure(OperationProcessor.java:184)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.onFailure(OperationProcessor.java:184)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider.$handleMultipleQueriesFailure(GWTRPCCommunicationProvider.java:305)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.onFailure(GWTRPCCommunicationProvider.java:263)
[frontend.jar:]
at
com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:198)
[gwt-servlet.jar:]
at com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:233)
[gwt-servlet.jar:]
at
com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409)
[gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)
at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)
In engine.log, I see some update related errors:
2019-06-22 18:46:54,053-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-22) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt2. command HSMGetAllTasksStatusesVDS failed:
Not SPM
2019-06-22 18:46:54,054-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-22) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = ovirt2.,
VdsIdVDSCommandParametersBase:{hostId='2f1da67f-66f7-40d6-81c4-dde82d6a6dd6'})'
execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not
SPM
2019-06-22 19:01:18,875-04 ERROR [org.ovirt.engine.core.bll.host.HostUpgradeManager]
(EE-ManagedThreadFactory-hostUpdatesChecker-Thread-2) [] Failed to run check-update of
host 'ovirt1.'. Error: fatal: [ovirt1.]: FAILED! => {"changed":
false, "msg": "yum lockfile is held by another process"}
2019-06-22 19:01:18,875-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpdatesChecker]
(EE-ManagedThreadFactory-hostUpdatesChecker-Thread-2) [] Failed to check if updates are
available for host 'ovirt1.' with error message 'Failed to run check-update of
host 'ovirt1.'. Error: fatal: [ovirt1.]: FAILED! => {"changed":
false, "msg": "yum lockfile is held by another process"}
2019-06-22 19:01:18,881-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-hostUpdatesChecker-Thread-2) [] EVENT_ID:
HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host ovirt1.
with message 'Failed to run check-update of host 'ovirt1.'. Error: fatal:
[ovirt1.]: FAILED! => {"changed": false, "msg": "yum lockfile
is held by another process"}
2019-06-22 19:08:04,848-04 ERROR [org.ovirt.engine.core.bll.SshHostRebootCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Command
'org.ovirt.engine.core.bll.SshHostRebootCommand' failed: ManagedThreadFactory is
stopped
2019-06-22 19:08:04,848-04 ERROR [org.ovirt.engine.core.bll.SshHostRebootCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Exception:
java.lang.IllegalStateException: ManagedThreadFactory is stopped
2019-06-22 19:08:04,857-04 ERROR [org.ovirt.engine.core.bll.SshHostRebootCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Error during log command:
org.ovirt.engine.core.bll.SshHostRebootCommand. Exception PreparedStatementCallback;
uncategorized SQLException for SQL [select * from getclusterbyclusterid(?, ?, ?)]; SQL
state [null]; error code [0]; IJ031013: Interrupted attempting lock:
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7b3be2cd; nested exception is
java.sql.SQLException: IJ031013: Interrupted attempting lock:
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7b3be2cd
2019-06-22 19:08:04,858-04 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Exception:
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback;
uncategorized SQLException for SQL [select * from checkifjobhastasks(?)]; SQL state
[null]; error code [0]; IJ031013: Interrupted attempting lock:
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7b3be2cd; nested exception is
java.sql.SQLException: IJ031013: Interrupted attempting lock:
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7b3be2cd
2019-06-22 19:08:04,869-04 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Failed to end Job
'376636f1-29cb-4bb7-adc4-1c8470ec1070', 'SshHostReboot': Failed to obtain
JDBC Connection; nested exception is java.sql.SQLException:
javax.resource.ResourceException: IJ000457: Unchecked throwable in
managedConnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@4a38ab13[state=NORMAL
managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7b3be2cd
connection handles=0 lastReturned=1561244884865 lastValidated=1561244613838
lastCheckedOut=1561244884865 trackByTx=false
pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@250efd73
mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@440e7f35[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@41bc50e8[connectionListener=4a38ab13
connectionManager=49ec2180 warned=false currentXid=null productName=Postgre
SQL productVersion=10.6 jndiName=java:/ENGINEDataSource] txSync=null]
2019-06-22 19:08:05,220-04 ERROR
[org.ovirt.engine.core.bll.hostdeploy.UpgradeHostInternalCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2) [6a9c283] Engine failed to restart
via ssh host 'ovirt1.' ('33138c5f-9e48-4426-b9e7-ae148ae16387') after
upgrade
2019-06-22 19:08:06,245-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-commandCoordinator-Thread-2)
[49770eb6-43f1-4048-af2c-a7ebadf282bb] EVENT_ID: HOST_UPGRADE_FAILED(841), Failed to
upgrade Host ovirt1. (User: admin@internal-authz).
2019-06-22 19:10:58,333-04 ERROR [org.ovirt.engine.core.bll.Backend] (ServerService Thread
Pool -- 52) [] Error during initialization: javax.ejb.EJBException:
java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
2019-06-22 19:24:36,687-04 ERROR [org.ovirt.engine.core.bll.Backend] (ServerService Thread
Pool -- 42) [] Error during initialization: javax.ejb.EJBException:
java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
2019-06-22 21:14:46,279-04 ERROR [org.ovirt.engine.core.bll.Backend] (ServerService Thread
Pool -- 55) [] Error during initialization: javax.ejb.EJBException:
java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
It appears that a lot of the system is still running, but I just can't reach the admin
page. Is there any way to roll this back or troubleshoot?