[ovirt-users] "Could not connect host to Data Center" after rebooting, how to resolve?

Wed Apr 30 14:09:39 UTC 2014

Hi list,

I had to do some fsck-related things last week on my ovirt box (centos,
single node and NFS on localhost).
Afterwards ovirt refused to start VMs  according to the webinterface
because it can't connect host to Data Center.

On my OS it works fine (I just replaced the IP by 'IP'):

[root at server ovirt-engine]# df
Filesystem            1K-blocks       Used  Available Use% Mounted on
/dev/mapper/vg_leiden-lv_root
                       51606140    6872592   42112108  15% /
tmpfs                   3978120          0    3978120   0% /dev/shm
/dev/sdb1                495844      99542     370702  22% /boot
/dev/mapper/vg_server-lv_home
                       62990260    2584272   57206196   5% /home
/dev/sda1            5814366992 4629615452 1184751540  80% /raid
IP:/raid/ovirt/data
                     5814367232 4629615616 1184751616  80%
/rhev/data-center/mnt/IP:_raid_ovirt_data
IP:/raid/ovirt/iso
                     5814367232 4629615616 1184751616  80%
/rhev/data-center/mnt/IP:_raid_ovirt_iso

in my event log in the webinterface I found correlation ID 7a735111 .
So grepping my logs for that one:

engine.log:2014-04-30 15:59:22,851 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(ajp--127.0.0.1-8702-8) [7a735111] Lock Acquired to object EngineLock
[exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE
engine.log:2014-04-30 15:59:22,888 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Running command:
ActivateStorageDomainCommand internal: false. Entities affected :  ID:
6bee0e2d-961c-453d-a266-e4623f91e162 Type: Storage
engine.log:2014-04-30 15:59:22,894 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Lock freed to object
EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162
value: STORAGE
engine.log:2014-04-30 15:59:22,895 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] ActivateStorage Domain.
Before Connect all hosts to pool. Time:4/30/14 3:59 PM
engine.log:2014-04-30 15:59:22,945 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ActivateStorageDomainVDSCommand( storagePoolId =
00000002-0002-0002-0002-0000000000ec, ignoreFailoverLimit = false,
storageDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162), log id: 5fecb439
engine.log:2014-04-30 15:59:23,011 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] hostFromVds::selectedVds
- server, spmStatus Unknown_Pool, storage pool Default
engine.log:2014-04-30 15:59:23,015 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ConnectStoragePoolVDSCommand(HostName = server, HostId =
ff23de79-f17c-439d-939e-d8f3d9672367, storagePoolId =
00000002-0002-0002-0002-0000000000ec, vds_spm_id = 1, masterDomainId =
6bee0e2d-961c-453d-a266-e4623f91e162, masterVersion = 1), log id: 15866a95
engine.log:2014-04-30 15:59:23,151 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ConnectStoragePoolVDSCommand, log id: 15866a95
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)
engine.log:2014-04-30 15:59:23,156 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ActivateStorageDomainVDSCommand, log id: 5fecb439
engine.log:2014-04-30 15:59:23,157 ERROR
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc
Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Could not connect host to Data Center(Storage issue) (Failed with error
ENGINE and code 5001)
engine.log:2014-04-30 15:59:23,162 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
[id=c2d8b433-f203-4d9e-b241-222eebf3dbae]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot [id=storagePoolId =
00000002-0002-0002-0002-0000000000ec, storageId =
6bee0e2d-961c-453d-a266-e4623f91e162, status=InActive].
engine.log:2014-04-30 15:59:23,170 INFO 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Correlation ID: 7a735111,
Job ID: 75dcda45-e0a3-46e3-b79f-8df9f0ed9d85, Call Stack: null, Custom
Event ID: -1, Message: Failed to activate Storage Domain data (Data
Center Default) by admin

I guess the most relevant error is:
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)

Okay so it can't connect to the data center. Well the NFS storage is
mounted and looks fine.
The directory is indeed looking fine:

[root at leiden data]# ls -al
total 12
drwxr-xr-x. 3 vdsm kvm 4096 Mar 30 03:27 .
drwxr-xr-x. 4 vdsm kvm 4096 Mar 30 03:26 ..
drwxr-xr-x. 5 vdsm kvm 4096 Mar 30 03:27
6bee0e2d-961c-453d-a266-e4623f91e162
-rwxr-xr-x. 1 vdsm kvm    0 Mar 30 03:27 __DIRECT_IO_TEST__

(and in the uuid there are indeed my VMs).

How should I solve this? To me everything looks just fine.

Cheers

Boudewijn Ector