Hi,
Due creation of new disk in gluster-storage I got some errors. Finally the disk is not being created.
engine.log says that there is no such file or directory, but the gluster domain (124e6273-c6f5-471f-88e7-e5d9d37d7385) is active:
[root@ipet etc]# vdsClient -s 0 getStoragePoolInfo 5849b030-626e-47cb-ad90-3ce782d831b3
name = Test
isoprefix = /rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f/images/11111111-1111-1111-1111-111111111111
pool_status = connected
lver = 0
spm_id = 1
master_uuid = 54f86ad7-2c12-4322-b2d1-f129f3d20e57
version = 3
domains = 9bdf01bd-78d6-4408-b3a9-e05469004d78:Active,e4a3928d-0475-4b99-bfb8-86606931296a:Active,c1582b82-9bfc-4e2b-ab2a-83551dcfba8f:Active,124e6273-c6f5-471f-88e7-e5d9d37d7385:Active,54f86ad7-2c12-4322-b2d1-f129f3d20e57:Active
type = FCP
master_ver = 8
9bdf01bd-78d6-4408-b3a9-e05469004d78 = {'status': 'Active', 'diskfree': '1043408617472', 'isoprefix': '', 'alerts': [], 'disktotal': '1099108974592', 'version': 3}
e4a3928d-0475-4b99-bfb8-86606931296a = {'status': 'Active', 'diskfree': '27244910346240', 'isoprefix': '', 'alerts': [], 'disktotal': '34573945667584', 'version': 3}
124e6273-c6f5-471f-88e7-e5d9d37d7385 = {'status': 'Active', 'diskfree': '12224677937152', 'isoprefix': '', 'alerts': [], 'disktotal': '16709735940096', 'version': 3}
c1582b82-9bfc-4e2b-ab2a-83551dcfba8f = {'status': 'Active', 'diskfree': '115282018304', 'isoprefix': '/rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '539016298496', 'version': 0}
54f86ad7-2c12-4322-b2d1-f129f3d20e57 = {'status': 'Active', 'diskfree': '148579024896', 'isoprefix': '', 'alerts': [], 'disktotal': '1197490569216', 'version': 3}
[root@ipet etc]# ll /rhev/data-center/mnt/glusterSD/moly\:_repo1/
insgesamt 0
drwxr-xr-x 4 vdsm kvm 96 17. Apr 10:02 124e6273-c6f5-471f-88e7-e5d9d37d7385
-rwxr-xr-x 1 vdsm kvm 0 11. Feb 12:52 __DIRECT_IO_TEST__
[root@ipet etc]# ll /rhev/data-center/
insgesamt 12
drwxr-xr-x 2 vdsm kvm 4096 14. Jul 10:42 5849b030-626e-47cb-ad90-3ce782d831b3
drwxr-xr-x 2 vdsm kvm 4096 16. Dez 2013 hsm-tasks
drwxr-xr-x 7 vdsm kvm 4096 14. Jul 10:24 mnt
[root@ipet etc]# ll /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/
insgesamt 16
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 54f86ad7-2c12-4322-b2d1-f129f3d20e57 -> /rhev/data-center/mnt/blockSD/54f86ad7-2c12-4322-b2d1-f129f3d20e57
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 9bdf01bd-78d6-4408-b3a9-e05469004d78 -> /rhev/data-center/mnt/blockSD/9bdf01bd-78d6-4408-b3a9-e05469004d78
lrwxrwxrwx 1 vdsm kvm 84 14. Jul 10:42 c1582b82-9bfc-4e2b-ab2a-83551dcfba8f -> /rhev/data-center/mnt/mixcoatl:_srv_mirror_ISOs/c1582b82-9bfc-4e2b-ab2a-83551dcfba8f
lrwxrwxrwx 1 vdsm kvm 66 14. Jul 10:42 mastersd -> /rhev/data-center/mnt/blockSD/54f86ad7-2c12-4322-b2d1-f129f3d20e57
Here i miss the directory of the gluster domain 124e6273-c6f5-471f-88e7-e5d9d37d7385 and the symbolic link belonging to it. The directory is missed on every Host.
What can I do to fix this issue?
With best regards
Andreas
engine.log:
2014-07-17 05:30:33,347 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [19404214] Running command: AddDiskCommand internal: false. Entities affected : ID: 124e6273-c6f5-471f-88e7-e5d9d37d7385 Type: Storage
2014-07-17 05:30:33,358 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [3e7d9b07] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 124e6273-c6f5-471f-88e7-e5d9d37d7385 Type: Storage
2014-07-17 05:30:33,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [3e7d9b07] START, CreateImageVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId = 124e6273-c6f5-471f-88e7-e5d9d37d7385, imageGroupId = 7594000c-23a9-4941-8218-2c0654518a3d, imageSizeInBytes = 68719476736, volumeFormat = RAW, newImageId = c48e46cd-9dd5-4c52-94a4-db0378aecc3c, newImageDescription = ), log id: 642dadf4
2014-07-17 05:30:33,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [3e7d9b07] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
2014-07-17 05:30:33,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [3e7d9b07] FINISH, CreateImageVDSCommand, return: c48e46cd-9dd5-4c52-94a4-db0378aecc3c, log id: 642dadf4
2014-07-17 05:30:33,403 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [3e7d9b07] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 7429f506-e57d-45e7-bc66-42bbdc90174c
2014-07-17 05:30:33,404 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [3e7d9b07] CommandMultiAsyncTasks::AttachTask: Attaching task d957c5a9-518e-4668-b8cd-27a106952e7c to command 7429f506-e57d-45e7-bc66-42bbdc90174c.
2014-07-17 05:30:33,412 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [3e7d9b07] Adding task d957c5a9-518e-4668-b8cd-27a106952e7c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
2014-07-17 05:30:33,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3e7d9b07] Correlation ID: 19404214, Job ID: 18601b0b-5bbe-4fc3-91f8-8a8dff0974fc, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 're_Disk2' was initiated by EwertA.
2014-07-17 05:30:33,421 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-7) [3e7d9b07] BaseAsyncTask::startPollingTask: Starting to poll task d957c5a9-518e-4668-b8cd-27a106952e7c.
2014-07-17 05:30:38,432 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-70) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
2014-07-17 05:30:38,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Failed in HSMGetAllTasksStatusesVDS method
2014-07-17 05:30:38,450 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-70) SPMAsyncTask::PollTask: Polling task d957c5a9-518e-4668-b8cd-27a106952e7c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'.
2014-07-17 05:30:38,467 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-70) BaseAsyncTask::LogEndTaskFailure: Task d957c5a9-518e-4668-b8cd-27a106952e7c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:
-- Result: cleanSuccess
-- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/124e6273-c6f5-471f-88e7-e5d9d37d7385/images/7594000c-23a9-4941-8218-2c0654518a3d', code = 100,
-- Exception: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/124e6273-c6f5-471f-88e7-e5d9d37d7385/images/7594000c-23a9-4941-8218-2c0654518a3d', code = 100
2014-07-17 05:30:38,485 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-70) CommandAsyncTask::EndActionIfNecessary: All tasks of command 7429f506-e57d-45e7-bc66-42bbdc90174c has ended -> executing endAction
2014-07-17 05:30:38,487 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-70) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 7429f506-e57d-45e7-bc66-42bbdc90174c): calling endAction .
2014-07-17 05:30:38,489 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0
2014-07-17 05:30:38,517 ERROR [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-6-thread-42) [19404214] Ending command with failure: org.ovirt.engine.core.bll.AddDiskCommand
2014-07-17 05:30:38,519 ERROR [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-6-thread-42) [3e7d9b07] Ending command with failure: org.ovirt.engine.core.bll.AddImageFromScratchCommand
2014-07-17 05:30:38,616 WARN [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-6-thread-42) VmCommand::EndVmCommand: Vm is null - not performing endAction on Vm
2014-07-17 05:30:38,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) Correlation ID: 19404214, Job ID: 18601b0b-5bbe-4fc3-91f8-8a8dff0974fc, Call Stack: null, Custom Event ID: -1, Message: Operation Add-Disk failed to complete.
2014-07-17 05:30:38,645 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result.
2014-07-17 05:30:38,652 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks.
2014-07-17 05:30:38,663 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-42) SPMAsyncTask::ClearAsyncTask: Attempting to clear task d957c5a9-518e-4668-b8cd-27a106952e7c
2014-07-17 05:30:38,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, SPMClearTaskVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, taskId = d957c5a9-518e-4668-b8cd-27a106952e7c), log id: f40f5de
2014-07-17 05:30:38,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, HSMClearTaskVDSCommand(HostName = ipet, HostId = ec83e4e0-40cf-49e4-94e3-5efc5ef6f2e2, taskId=d957c5a9-518e-4668-b8cd-27a106952e7c), log id: 71cdc9a4
2014-07-17 05:30:38,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, HSMClearTaskVDSCommand, log id: 71cdc9a4
2014-07-17 05:30:38,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, SPMClearTaskVDSCommand, log id: f40f5de
2014-07-17 05:30:38,741 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-42) BaseAsyncTask::RemoveTaskFromDB: Removed task d957c5a9-518e-4668-b8cd-27a106952e7c from DataBase
2014-07-17 05:30:38,747 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 7429f506-e57d-45e7-bc66-42bbdc90174c
