Unrecognized message received error when ISO domain is put in Maintenance

Hi all, We are running oVirt 4.1.1 on engine and nodes. The ISO storage domain is NFS. Everything has been working great for years. Now I need to move the ISO and Export domains to a different storage server. When I put the ISO domain in maintenance the Master domain and the Export domain start flipping active-non active. The Events tab in the Web UI shows this message scrolling every 30 sec. or so: "Datacenter is being initialized. Please wait for initialization to complete" ... but it never completes. It also starts cycling the SPM assignment from one node to another. As soon as I activate back the ISO domain everything calms down and no more errors are recorded in the logs. In engine.log I see this "Unrecognized message received" error: 2019-02-25 19:45:20,467-08 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Unrecognized message received 2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [76363b98] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), Correlation ID: null, Call Stack: null, Custom E vent ID: -1, Message: VDSM command GetStoragePoolInfoVDS failed: Unrecognized message received 2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [76363b98] ERROR, org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand, exception: VDSGenericExce ption: VDSNetworkException: Unrecognized message received , log id: 1635c439 2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [76363b98] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetwor kException: Unrecognized message received In vdsm.log on the nodes I see this: 2019-02-25 19:45:29,817-0800 ERROR (upgrade/b5b7a10) [storage.StoragePool] Unhandled exception (utils:371) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper return f(*a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 232, in _upgradePoolDomain self._finalizePoolUpgradeIfNeeded() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state And I believe that after that error it moves the SPM to another host, gets the same error and continues moving it to the next host and so on. I really need to move the ISO and Export domains to a different storage server. Any idea what is causing this or how to fix it? Appreciate any help. Thanks, -- Peter
participants (1)
-
Wood Peter