6:34 a.m.
Hi all,
We are running oVirt 4.1.1 on engine and nodes. The ISO storage domain is
NFS. Everything has been working great for years.
Now I need to move the ISO and Export domains to a different storage server.
When I put the ISO domain in maintenance the Master domain and the Export
domain start flipping active-non active. The Events tab in the Web UI shows
this message scrolling every 30 sec. or so:
"Datacenter is being initialized. Please wait for initialization to
complete"
... but it never completes.
It also starts cycling the SPM assignment from one node to another.
As soon as I activate back the ISO domain everything calms down and no more
errors are recorded in the logs.
In engine.log I see this "Unrecognized message received" error:
2019-02-25 19:45:20,467-08 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) []
Unable to process messages Unrecognized message received
2019-02-25 19:45:20,481-08 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler6) [76363b98] EVENT_ID:
IRS_BROKER_COMMAND_FAILURE(10,803), Correlation ID: null, Call Stack: null,
Custom E
vent ID: -1, Message: VDSM command GetStoragePoolInfoVDS failed:
Unrecognized message received
2019-02-25 19:45:20,481-08 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler6) [76363b98] ERROR,
org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand,
exception: VDSGenericExce
ption: VDSNetworkException: Unrecognized message received , log id: 1635c439
2019-02-25 19:45:20,481-08 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler6) [76363b98] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetwor
kException: Unrecognized message received
In vdsm.log on the nodes I see this:
2019-02-25 19:45:29,817-0800 ERROR (upgrade/b5b7a10) [storage.StoragePool]
Unhandled exception (utils:371)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in
wrapper
return f(*a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in
run
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
79, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 232, in _upgradePoolDomain
self._finalizePoolUpgradeIfNeeded()
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
77, in wrapper
raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
And I believe that after that error it moves the SPM to another host, gets
the same error and continues moving it to the next host and so on.
I really need to move the ISO and Export domains to a different storage
server.
Any idea what is causing this or how to fix it?
Appreciate any help.
Thanks,
-- Peter