[Users] Data Center Non Responsive / Contending

[Giorgio Bersano]

Hi everyone,
I'm asking for help again as testing my setup I put myself in a
situation in which I can't get out.

Layout: two hosts, an iSCSI storage, the engine installed as a
"regular" KVM guest on another host (external to the oVirt setup). All
CentOS 6.5, oVirt 3.4.0beta3.
One DC ("Default"), one Cluster ("Default"), default storage type iSCSI.
Moreover, ISO domain is another "external" KVM guest exposing an NFS
share; Export Domain is in fact a VM in this Cluster.

This is a preproduction setup.
Yesterday all was running fine until I needed to do some hardware maintenance.
So I decided to put the two hosts in maintenance from the webadmin
then shutdown them in the usual way. Engine was left operational.
Later i booted again one host, waited some time then tried to activate
it (from the webadmin) without success. Even switching on the second
host didn't change anything.
Now the Data Center status toggles between "Contending" and "Non Responsive".

None of the hosts is SPM and if I choose "Select as SPM" in the
webadmin the result is this popup:
----------
 Operation Canceled
----------
Error while executing action: Cannot force select SPM: Storage Domain
cannot be accessed.
-Please check that at least one Host is operational and Data Center state is up.
----------

The two hosts are operational but the DC isn't up (downpointing red
arrow - "Non Responsive", hourglass - "Contending").

Storage domains are in "Unknown" state and if I try to "Activate" the
Master Data Domain its status becomes "Locked" and then it fails with
these events:
. Invalid status on Data Center Default. Setting status to Non Responsive
. Failed to activate Storage Domain dt02clu6070 (Data Center Default) by admin

Connectivity seems OK. iSCSI connectivity seems OK.

I'm almost certain that if I had left one host active I would have had
zero problems.
But I also think shutting down the full system should not be a problem.

In the end I restarted the ovirt-engine service without results (as expected).

Any thought? Logs needed?

TIA,
Giorgio.