4:42 a.m.
Hi,
Due creation of new disk in gluster-storage I got some errors. Finally the disk is not
being created.
engine.log says that there is no such file or directory, but the gluster domain
(124e6273-c6f5-471f-88e7-e5d9d37d7385) is active:
[root@ipet etc]# vdsClient -s 0 getStoragePoolInfo 5849b030-626e-47cb-ad90-3ce782d831b3
name = Test
isoprefix =
/rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f/images/11111111-1111-1111-1111-111111111111
pool_status = connected
lver = 0
spm_id = 1
master_uuid = 54f86ad7-2c12-4322-b2d1-f129f3d20e57
version = 3
domains =
9bdf01bd-78d6-4408-b3a9-e05469004d78:Active,e4a3928d-0475-4b99-bfb8-86606931296a:Active,c1582b82-9bfc-4e2b-ab2a-83551dcfba8f:Active,124e6273-c6f5-471f-88e7-e5d9d37d7385:Active,54f86ad7-2c12-4322-b2d1-f129f3d20e57:Active
type = FCP
master_ver = 8
9bdf01bd-78d6-4408-b3a9-e05469004d78 = {'status': 'Active',
'diskfree': '1043408617472', 'isoprefix': '',
'alerts': [], 'disktotal': '1099108974592', 'version': 3}
e4a3928d-0475-4b99-bfb8-86606931296a = {'status': 'Active',
'diskfree': '27244910346240', 'isoprefix': '',
'alerts': [], 'disktotal': '34573945667584', 'version':
3}
124e6273-c6f5-471f-88e7-e5d9d37d7385 = {'status': 'Active',
'diskfree': '12224677937152', 'isoprefix': '',
'alerts': [], 'disktotal': '16709735940096', 'version':
3}
c1582b82-9bfc-4e2b-ab2a-83551dcfba8f = {'status': 'Active',
'diskfree': '115282018304', 'isoprefix':
'/rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f/images/11111111-1111-1111-1111-111111111111',
'alerts': [], 'disktotal': '539016298496', 'version': 0}
54f86ad7-2c12-4322-b2d1-f129f3d20e57 = {'status': 'Active',
'diskfree': '148579024896', 'isoprefix': '',
'alerts': [], 'disktotal': '1197490569216', 'version': 3}
[root@ipet etc]# ll /rhev/data-center/mnt/glusterSD/moly\:_repo1/
insgesamt 0
drwxr-xr-x 4 vdsm kvm 96 17. Apr 10:02 124e6273-c6f5-471f-88e7-e5d9d37d7385
-rwxr-xr-x 1 vdsm kvm 0 11. Feb 12:52 __DIRECT_IO_TEST__
[root@ipet etc]# ll /rhev/data-center/
insgesamt 12
drwxr-xr-x 2 vdsm kvm 4096 14. Jul 10:42 5849b030-626e-47cb-ad90-3ce782d831b3
drwxr-xr-x 2 vdsm kvm 4096 16. Dez 2013 hsm-tasks
drwxr-xr-x 7 vdsm kvm 4096 14. Jul 10:24 mnt
[root@ipet etc]# ll /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/
insgesamt 16
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 54f86ad7-2c12-4322-b2d1-f129f3d20e57 ->
/rhev/data-center/mnt/blockSD/54f86ad7-2c12-4322-b2d1-f129f3d20e57
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 9bdf01bd-78d6-4408-b3a9-e05469004d78 ->
/rhev/data-center/mnt/blockSD/9bdf01bd-78d6-4408-b3a9-e05469004d78
lrwxrwxrwx 1 vdsm kvm 84 14. Jul 10:42 c1582b82-9bfc-4e2b-ab2a-83551dcfba8f ->
/rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 mastersd ->
/rhev/data-center/mnt/blockSD/54f86ad7-2c12-4322-b2d1-f129f3d20e57
Here i miss the directory of the gluster domain 124e6273-c6f5-471f-88e7-e5d9d37d7385 and
the symbolic link belonging to it. The directory is missed on every Host.
What can I do to fix this issue?
With best regards
Andreas
engine.log:
2014-07-17 05:30:33,347 INFO [org.ovirt.engine.core.bll.AddDiskCommand]
(ajp--127.0.0.1-8702-7) [19404214] Running command: AddDiskCommand internal: false.
Entities affected : ID: 124e6273-c6f5-471f-88e7-e5d9d37d7385 Type: Storage
2014-07-17 05:30:33,358 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand]
(ajp--127.0.0.1-8702-7) [3e7d9b07] Running command: AddImageFromScratchCommand internal:
true. Entities affected : ID: 124e6273-c6f5-471f-88e7-e5d9d37d7385 Type: Storage
2014-07-17 05:30:33,364 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7)
[3e7d9b07] START, CreateImageVDSCommand( storagePoolId =
5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId =
124e6273-c6f5-471f-88e7-e5d9d37d7385, imageGroupId = 7594000c-23a9-4941-8218-2c0654518a3d,
imageSizeInBytes = 68719476736, volumeFormat = RAW, newImageId =
c48e46cd-9dd5-4c52-94a4-db0378aecc3c, newImageDescription = ), log id: 642dadf4
2014-07-17 05:30:33,366 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7)
[3e7d9b07] -- executeIrsBrokerCommand: calling 'createVolume' with two new
parameters: description and UUID
2014-07-17 05:30:33,392 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7)
[3e7d9b07] FINISH, CreateImageVDSCommand, return: c48e46cd-9dd5-4c52-94a4-db0378aecc3c,
log id: 642dadf4
2014-07-17 05:30:33,403 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(ajp--127.0.0.1-8702-7) [3e7d9b07] CommandAsyncTask::Adding CommandMultiAsyncTasks object
for command 7429f506-e57d-45e7-bc66-42bbdc90174c
2014-07-17 05:30:33,404 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks]
(ajp--127.0.0.1-8702-7) [3e7d9b07] CommandMultiAsyncTasks::AttachTask: Attaching task
d957c5a9-518e-4668-b8cd-27a106952e7c to command 7429f506-e57d-45e7-bc66-42bbdc90174c.
2014-07-17 05:30:33,412 INFO [org.ovirt.engine.core.bll.AsyncTaskManager]
(ajp--127.0.0.1-8702-7) [3e7d9b07] Adding task d957c5a9-518e-4668-b8cd-27a106952e7c
(Parent Command AddDisk, Parameters Type
org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started
yet..
2014-07-17 05:30:33,419 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ajp--127.0.0.1-8702-7) [3e7d9b07] Correlation ID: 19404214, Job ID:
18601b0b-5bbe-4fc3-91f8-8a8dff0974fc, Call Stack: null, Custom Event ID: -1, Message:
Add-Disk operation of 're_Disk2' was initiated by EwertA.
2014-07-17 05:30:33,421 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(ajp--127.0.0.1-8702-7) [3e7d9b07] BaseAsyncTask::startPollingTask: Starting to poll task
d957c5a9-518e-4668-b8cd-27a106952e7c.
2014-07-17 05:30:38,432 INFO [org.ovirt.engine.core.bll.AsyncTaskManager]
(DefaultQuartzScheduler_Worker-70) Polling and updating Async Tasks: 1 tasks, 1 tasks to
poll now
2014-07-17 05:30:38,445 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-70) Failed in HSMGetAllTasksStatusesVDS method
2014-07-17 05:30:38,450 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-70) SPMAsyncTask::PollTask: Polling task
d957c5a9-518e-4668-b8cd-27a106952e7c (Parent Command AddDisk, Parameters Type
org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished,
result 'cleanSuccess'.
2014-07-17 05:30:38,467 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-70) BaseAsyncTask::LogEndTaskFailure: Task
d957c5a9-518e-4668-b8cd-27a106952e7c (Parent Command AddDisk, Parameters Type
org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:
-- Result: cleanSuccess
-- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS,
error = [Errno 2] No such file or directory:
'/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/124e6273-c6f5-471f-88e7-e5d9d37d7385/images/7594000c-23a9-4941-8218-2c0654518a3d',
code = 100,
-- Exception: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS,
error = [Errno 2] No such file or directory:
'/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/124e6273-c6f5-471f-88e7-e5d9d37d7385/images/7594000c-23a9-4941-8218-2c0654518a3d',
code = 100
2014-07-17 05:30:38,485 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(DefaultQuartzScheduler_Worker-70) CommandAsyncTask::EndActionIfNecessary: All tasks of
command 7429f506-e57d-45e7-bc66-42bbdc90174c has ended -> executing endAction
2014-07-17 05:30:38,487 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(DefaultQuartzScheduler_Worker-70) CommandAsyncTask::endAction: Ending action for 1 tasks
(command ID: 7429f506-e57d-45e7-bc66-42bbdc90174c): calling endAction .
2014-07-17 05:30:38,489 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction [within thread]
context: Attempting to endAction AddDisk, executionIndex: 0
2014-07-17 05:30:38,517 ERROR [org.ovirt.engine.core.bll.AddDiskCommand]
(org.ovirt.thread.pool-6-thread-42) [19404214] Ending command with failure:
org.ovirt.engine.core.bll.AddDiskCommand
2014-07-17 05:30:38,519 ERROR [org.ovirt.engine.core.bll.AddImageFromScratchCommand]
(org.ovirt.thread.pool-6-thread-42) [3e7d9b07] Ending command with failure:
org.ovirt.engine.core.bll.AddImageFromScratchCommand
2014-07-17 05:30:38,616 WARN [org.ovirt.engine.core.bll.AddDiskCommand]
(org.ovirt.thread.pool-6-thread-42) VmCommand::EndVmCommand: Vm is null - not performing
endAction on Vm
2014-07-17 05:30:38,638 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-42) Correlation ID: 19404214, Job ID:
18601b0b-5bbe-4fc3-91f8-8a8dff0974fc, Call Stack: null, Custom Event ID: -1, Message:
Operation Add-Disk failed to complete.
2014-07-17 05:30:38,645 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within
thread]: endAction for action type AddDisk completed, handling the result.
2014-07-17 05:30:38,652 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within
thread]: endAction for action type AddDisk succeeded, clearing tasks.
2014-07-17 05:30:38,663 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-42) SPMAsyncTask::ClearAsyncTask: Attempting to clear task
d957c5a9-518e-4668-b8cd-27a106952e7c
2014-07-17 05:30:38,669 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-42) START, SPMClearTaskVDSCommand( storagePoolId =
5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, taskId =
d957c5a9-518e-4668-b8cd-27a106952e7c), log id: f40f5de
2014-07-17 05:30:38,711 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-42) START, HSMClearTaskVDSCommand(HostName = ipet, HostId
= ec83e4e0-40cf-49e4-94e3-5efc5ef6f2e2, taskId=d957c5a9-518e-4668-b8cd-27a106952e7c), log
id: 71cdc9a4
2014-07-17 05:30:38,728 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-42) FINISH, HSMClearTaskVDSCommand, log id: 71cdc9a4
2014-07-17 05:30:38,733 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-42) FINISH, SPMClearTaskVDSCommand, log id: f40f5de
2014-07-17 05:30:38,741 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-42) BaseAsyncTask::RemoveTaskFromDB: Removed task
d957c5a9-518e-4668-b8cd-27a106952e7c from DataBase
2014-07-17 05:30:38,747 INFO [org.ovirt.engine.core.bll.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within
thread]: Removing CommandMultiAsyncTasks object for entity
7429f506-e57d-45e7-bc66-42bbdc90174c