Hi all,

We are running oVirt 4.1.1 on engine and nodes. The ISO storage domain is NFS. Everything has been working great for years.

Now I need to move the ISO and Export domains to a different storage server.

When I put the ISO domain in maintenance the Master domain and the Export domain start flipping active-non active. The Events tab in the Web UI shows this message scrolling every 30 sec. or so:

"Datacenter is being initialized. Please wait for initialization to complete"

... but it never completes.

It also starts cycling the SPM assignment from one node to another.

As soon as I activate back the ISO domain everything calms down and no more errors are recorded in the logs.

In engine.log I see this "Unrecognized message received" error:

2019-02-25 19:45:20,467-08 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Unrecognized message received

2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [76363b98] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), Correlation ID: null, Call Stack: null, Custom E

vent ID: -1, Message: VDSM command GetStoragePoolInfoVDS failed: Unrecognized message received

2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [76363b98] ERROR, org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand, exception: VDSGenericExce

ption: VDSNetworkException: Unrecognized message received , log id: 1635c439

2019-02-25 19:45:20,481-08 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [76363b98] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetwor

kException: Unrecognized message received

In vdsm.log on the nodes I see this:

2019-02-25 19:45:29,817-0800 ERROR (upgrade/b5b7a10) [storage.StoragePool] Unhandled exception (utils:371)

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper

    return f(*a, **kw)

  File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run

    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper

    return method(self, *args, **kwargs)

  File "/usr/share/vdsm/storage/sp.py", line 232, in _upgradePoolDomain

    self._finalizePoolUpgradeIfNeeded()

  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper

    raise SecureError("Secured object is not in safe state")

SecureError: Secured object is not in safe state

And I believe that after that error it moves the SPM to another host, gets the same error and continues moving it to the next host and so on.

I really need to move the ISO and Export domains to a different storage server.

Any idea what is causing this or how to fix it?

Appreciate any help.

Thanks,

-- Peter