7:09 p.m.
This is a multi-part message in MIME format.
--------------000906040806080001080808
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Hi list,
I had to do some fsck-related things last week on my ovirt box (centos,
single node and NFS on localhost).
Afterwards ovirt refused to start VMs according to the webinterface
because it can't connect host to Data Center.
On my OS it works fine (I just replaced the IP by 'IP'):
[root@server ovirt-engine]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_leiden-lv_root
51606140 6872592 42112108 15% /
tmpfs 3978120 0 3978120 0% /dev/shm
/dev/sdb1 495844 99542 370702 22% /boot
/dev/mapper/vg_server-lv_home
62990260 2584272 57206196 5% /home
/dev/sda1 5814366992 4629615452 1184751540 80% /raid
IP:/raid/ovirt/data
5814367232 4629615616 1184751616 80%
/rhev/data-center/mnt/IP:_raid_ovirt_data
IP:/raid/ovirt/iso
5814367232 4629615616 1184751616 80%
/rhev/data-center/mnt/IP:_raid_ovirt_iso
in my event log in the webinterface I found correlation ID 7a735111 .
So grepping my logs for that one:
engine.log:2014-04-30 15:59:22,851 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(ajp--127.0.0.1-8702-8) [7a735111] Lock Acquired to object EngineLock
[exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE
engine.log:2014-04-30 15:59:22,888 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Running command:
ActivateStorageDomainCommand internal: false. Entities affected : ID:
6bee0e2d-961c-453d-a266-e4623f91e162 Type: Storage
engine.log:2014-04-30 15:59:22,894 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Lock freed to object
EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162
value: STORAGE
engine.log:2014-04-30 15:59:22,895 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] ActivateStorage Domain.
Before Connect all hosts to pool. Time:4/30/14 3:59 PM
engine.log:2014-04-30 15:59:22,945 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ActivateStorageDomainVDSCommand( storagePoolId =
00000002-0002-0002-0002-0000000000ec, ignoreFailoverLimit = false,
storageDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162), log id: 5fecb439
engine.log:2014-04-30 15:59:23,011 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] hostFromVds::selectedVds
- server, spmStatus Unknown_Pool, storage pool Default
engine.log:2014-04-30 15:59:23,015 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ConnectStoragePoolVDSCommand(HostName = server, HostId =
ff23de79-f17c-439d-939e-d8f3d9672367, storagePoolId =
00000002-0002-0002-0002-0000000000ec, vds_spm_id = 1, masterDomainId =
6bee0e2d-961c-453d-a266-e4623f91e162, masterVersion = 1), log id: 15866a95
engine.log:2014-04-30 15:59:23,151 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ConnectStoragePoolVDSCommand, log id: 15866a95
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)
engine.log:2014-04-30 15:59:23,156 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ActivateStorageDomainVDSCommand, log id: 5fecb439
engine.log:2014-04-30 15:59:23,157 ERROR
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc
Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Could not connect host to Data Center(Storage issue) (Failed with error
ENGINE and code 5001)
engine.log:2014-04-30 15:59:23,162 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
[id=c2d8b433-f203-4d9e-b241-222eebf3dbae]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot [id=storagePoolId =
00000002-0002-0002-0002-0000000000ec, storageId =
6bee0e2d-961c-453d-a266-e4623f91e162, status=InActive].
engine.log:2014-04-30 15:59:23,170 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Correlation ID: 7a735111,
Job ID: 75dcda45-e0a3-46e3-b79f-8df9f0ed9d85, Call Stack: null, Custom
Event ID: -1, Message: Failed to activate Storage Domain data (Data
Center Default) by admin
I guess the most relevant error is:
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)
Okay so it can't connect to the data center. Well the NFS storage is
mounted and looks fine.
The directory is indeed looking fine:
[root@leiden data]# ls -al
total 12
drwxr-xr-x. 3 vdsm kvm 4096 Mar 30 03:27 .
drwxr-xr-x. 4 vdsm kvm 4096 Mar 30 03:26 ..
drwxr-xr-x. 5 vdsm kvm 4096 Mar 30 03:27
6bee0e2d-961c-453d-a266-e4623f91e162
-rwxr-xr-x. 1 vdsm kvm 0 Mar 30 03:27 __DIRECT_IO_TEST__
(and in the uuid there are indeed my VMs).
How should I solve this? To me everything looks just fine.
Cheers
Boudewijn Ector
--------------000906040806080001080808
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-text-plain" wrap="true"
graphical-quote="true"
style="font-family: -moz-fixed; font-size: 14px;"
lang="x-western">
<pre wrap="">Hi list,
I had to do some fsck-related things last week on my ovirt box (centos,
single node and NFS on localhost).
Afterwards ovirt refused to start VMs according to the webinterface
because it can't connect host to Data Center.
On my OS it works fine (I just replaced the IP by 'IP'):
[root@server ovirt-engine]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_leiden-lv_root
51606140 6872592 42112108 15% /
tmpfs 3978120 0 3978120 0% /dev/shm
/dev/sdb1 495844 99542 370702 22% /boot
/dev/mapper/vg_server-lv_home
62990260 2584272 57206196 5% /home
/dev/sda1 5814366992 4629615452 1184751540 80% /raid
IP:/raid/ovirt/data
5814367232 4629615616 1184751616 80%
/rhev/data-center/mnt/IP:_raid_ovirt_data
IP:/raid/ovirt/iso
5814367232 4629615616 1184751616 80%
/rhev/data-center/mnt/IP:_raid_ovirt_iso
in my event log in the webinterface I found correlation ID 7a735111 .
So grepping my logs for that one:
engine.log:2014-04-30 15:59:22,851 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(ajp--127.0.0.1-8702-8) [7a735111] Lock Acquired to object EngineLock
[exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE
engine.log:2014-04-30 15:59:22,888 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Running command:
ActivateStorageDomainCommand internal: false. Entities affected : ID:
6bee0e2d-961c-453d-a266-e4623f91e162 Type: Storage
engine.log:2014-04-30 15:59:22,894 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Lock freed to object
EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162
value: STORAGE
engine.log:2014-04-30 15:59:22,895 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] ActivateStorage Domain.
Before Connect all hosts to pool. Time:4/30/14 3:59 PM
engine.log:2014-04-30 15:59:22,945 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ActivateStorageDomainVDSCommand( storagePoolId =
00000002-0002-0002-0002-0000000000ec, ignoreFailoverLimit = false,
storageDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162), log id: 5fecb439
engine.log:2014-04-30 15:59:23,011 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] hostFromVds::selectedVds
- server, spmStatus Unknown_Pool, storage pool Default
engine.log:2014-04-30 15:59:23,015 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] START,
ConnectStoragePoolVDSCommand(HostName = server, HostId =
ff23de79-f17c-439d-939e-d8f3d9672367, storagePoolId =
00000002-0002-0002-0002-0000000000ec, vds_spm_id = 1, masterDomainId =
6bee0e2d-961c-453d-a266-e4623f91e162, masterVersion = 1), log id: 15866a95
engine.log:2014-04-30 15:59:23,151 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ConnectStoragePoolVDSCommand, log id: 15866a95
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)
engine.log:2014-04-30 15:59:23,156 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH,
ActivateStorageDomainVDSCommand, log id: 5fecb439
engine.log:2014-04-30 15:59:23,157 ERROR
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc
Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Could not connect host to Data Center(Storage issue) (Failed with error
ENGINE and code 5001)
engine.log:2014-04-30 15:59:23,162 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Command
[id=c2d8b433-f203-4d9e-b241-222eebf3dbae]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot [id=storagePoolId =
00000002-0002-0002-0002-0000000000ec, storageId =
6bee0e2d-961c-453d-a266-e4623f91e162, status=InActive].
engine.log:2014-04-30 15:59:23,170 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-49) [7a735111] Correlation ID: 7a735111,
Job ID: 75dcda45-e0a3-46e3-b79f-8df9f0ed9d85, Call Stack: null, Custom
Event ID: -1, Message: Failed to activate Storage Domain data (Data
Center Default) by admin
I guess the most relevant error is:
engine.log:2014-04-30 15:59:23,152 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(org.ovirt.thread.pool-6-thread-49) [7a735111]
IrsBroker::Failed::ActivateStorageDomainVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data
Center(Storage issue)
Okay so it can't connect to the data center. Well the NFS storage is
mounted and looks fine.
The directory is indeed looking fine:
[root@leiden data]# ls -al
total 12
drwxr-xr-x. 3 vdsm kvm 4096 Mar 30 03:27 .
drwxr-xr-x. 4 vdsm kvm 4096 Mar 30 03:26 ..
drwxr-xr-x. 5 vdsm kvm 4096 Mar 30 03:27
6bee0e2d-961c-453d-a266-e4623f91e162
-rwxr-xr-x. 1 vdsm kvm 0 Mar 30 03:27 __DIRECT_IO_TEST__
(and in the uuid there are indeed my VMs).
How should I solve this? To me everything looks just fine.
Cheers
Boudewijn Ector
</pre>
</div>
</body>
</html>
--------------000906040806080001080808--