1:48 a.m.
Today I went through and re-loaded all the hosts from bare metal, ran all the updates,
created manual VDO storage, and manually created gluster using ovirt web interface, same
issue adding storage domain. Gluster bricks are
/gluster_bricks/ssd-storageone/ssd-storageone.
Checking the hosts, they're not mounting the gluster storage anywhere, I think
that's the problem, but can I run the gdeploy wizard and then increase the drive
size?
VDSM:
2019-01-02 17:11:41,485-0500 INFO (jsonrpc/2) [vdsm.api] START
connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options':
u'backup-volfile-servers=10.100.50.14:10.100.50.12', u'id':
u'00000000-0000-0000-0000-000000000000', u'connection':
u'10.100.50.16:/ssd-storageone', u'iqn': u'', u'user':
u'',
u'tpgt': u'1', u'vfs_type': u'glusterfs',
u'password': '********', u'port': u''}], options=None)
from=::ffff:172.16.9.202,58810, flow_id=5e9f8d6e-2b8e-4e6d-abda-d528fe7ae3a4,
task_id=b0072909-c539-47ad-b03a-d19a237adae8 (api:46)
2019-01-02 17:11:41,683-0500 INFO (jsonrpc/2) [storage.StorageServer.MountConnection]
Creating directory u'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone'
(storageServer:167)
2019-01-02 17:11:41,683-0500 INFO (jsonrpc/2) [storage.fileUtils] Creating directory:
/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone mode: None (fileUtils:197)
2019-01-02 17:11:41,684-0500 INFO (jsonrpc/2) [storage.Mount] mounting
10.100.50.16:/ssd-storageone at
/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone (mount:204)
2019-01-02 17:11:41,944-0500 INFO (itmap/0) [IOProcessClient]
(/glusterSD/10.100.50.16:_ssd-storageone) Starting client (__init__:308)
2019-01-02 17:11:41,952-0500 INFO (ioprocess/7805) [IOProcess]
(/glusterSD/10.100.50.16:_ssd-storageone) Starting ioprocess (__init__:434)
2019-01-02 17:11:42,097-0500 INFO (jsonrpc/4) [vdsm.api] START
connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options':
u'backup-volfile-servers=10.100.50.14:10.100.50.12', u'id':
u'ab4b5ccb-3d8b-4fa5-add3-7534e3672772', u'connection':
u'10.100.50.16:/ssd-storageone', u'iqn': u'', u'user':
u'',
u'tpgt': u'1', u'vfs_type': u'glusterfs',
u'password': '********', u'port': u''}], options=None)
from=::ffff:172.16.9.202,58810, flow_id=300c8c7a,
task_id=9d3955f7-c5d2-4242-b83b-10def6d2786b (api:46)
2019-01-02 17:11:42,107-0500 INFO (jsonrpc/6) [vdsm.api] START
createStorageDomain(storageType=7, sdUUID=u'c68931fd-42d5-4b1e-9ad3-5933311da71d',
domainName=u'Silver', typeSpecificArg=u'10.100.50.16:/ssd-storageone',
domClass=1, domVersion=u'4', options=None) from=::ffff:172.16.9.202,58810,
flow_id=300c8c7a, task_id=63f9d019-2df0-46f2-b50b-ac1
a42ee5a1a (api:46)
2019-01-02 17:11:42,466-0500 INFO (jsonrpc/6) [storage.StorageDomain]
sdUUID=c68931fd-42d5-4b1e-9ad3-5933311da71d domainName=Silver
remotePath=10.100.50.16:/ssd-storageone domClass=1 (nfsSD:70)
2019-01-02 17:11:42,500-0500 INFO (jsonrpc/6) [storage.StorageDomain] Creating domain
metadata directory
u'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md'
(fileSD:450)
2019-01-02 17:11:42,576-0500 INFO (jsonrpc/6) [vdsm.api] FINISH createStorageDomain
error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsy
nc'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0186115 s, 0.0
kB/s\n" from=::ffff:172.16.9.202,58810, flow_id=300c8c7a, task_id=6
3f9d019-2df0-46f2-b50b-ac1a42ee5a1a (api:50)
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data
-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0186115 s, 0.0
kB/s\n"
2019-01-02 17:11:42,577-0500 INFO (jsonrpc/6) [storage.TaskManager.Task]
(Task='63f9d019-2df0-46f2-b50b-ac1a42ee5a1a') aborting: Task is aborted:
u'Command [\'/usr/bin/dd\', \'iflag=fullblock\',
u\'of=/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases\',
\'oflag=direct,seek_bytes\',
\'seek=1048576\', \'bs=256512\', \'count=1\',
\'conv=notrunc,nocreat,fsync\'] failed with rc=1 out=\'[suppressed]\'
err="/usr/bin/dd: error writing
\'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases\':
Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.0
186115 s, 0.0 kB/s\\n"' - code 100 (task:1181)
2019-01-02 17:11:42,577-0500 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH
createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,n
ocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd:
error writing
'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0186115 s, 0.0
kB/s\n" (dispatcher:86)
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data
-center/mnt/glusterSD/10.100.50.16:_ssd-storageone/c68931fd-42d5-4b1e-9ad3-5933311da71d/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0186115 s, 0.0
kB/s\n"
2019-01-02 17:11:42,745-0500 INFO (jsonrpc/1) [vdsm.api] START
disconnectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options':
u'backup-volfile-servers=10.100.50.14:10.100.50.12', u'id':
u'ab4b5ccb-3d8b-4fa5-add3-7534e3672772', u'connection':
u'10.100.50.16:/ssd-storageone', u'iqn': u'', u'user':
u'
', u'tpgt': u'1', u'vfs_type': u'glusterfs',
u'password': '********', u'port': u''}], options=None)
from=::ffff:172.16.9.202,58810, flow_id=b4ebe5ed-9626-43ca-a307-b3b3e7d01f2b,
task_id=d8eb764a-bc46-46a6-9758-83cbc67529c3 (api:46)
2019-01-02 17:11:42,745-0500 INFO (jsonrpc/1) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone (mount:212)
2019-01-02 17:11:54,581-0500 INFO (jsonrpc/0) [vdsm.api] START
connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options':
u'backup-volfile-servers=10.100.50.14:10.100.50.12', u'id':
u'00000000-0000-0000-0000-000000000000', u'connection':
u'10.100.50.16:/ssd-storageone', u'iqn': u'', u'user':
u'',
u'tpgt': u'1', u'vfs_type': u'glusterfs',
u'password': '********', u'port': u''}], options=None)
from=::ffff:172.16.9.202,58810, flow_id=712a4a82-b947-4e18-b6e4-dacf0b09d4b4,
task_id=0e24152e-c7e9-44b4-bc26-21648c00986a (api:46)
2019-01-02 17:11:54,775-0500 INFO (jsonrpc/0) [storage.StorageServer.MountConnection]
Creating directory u'/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone'
(storageServer:167)
2019-01-02 17:11:54,775-0500 INFO (jsonrpc/0) [storage.fileUtils] Creating directory:
/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone mode: None (fileUtils:197)
2019-01-02 17:11:54,776-0500 INFO (jsonrpc/0) [storage.Mount] mounting
10.100.50.16:/ssd-storageone at
/rhev/data-center/mnt/glusterSD/10.100.50.16:_ssd-storageone (mount:204)
2019-01-02 17:11:55,068-0500 INFO (jsonrpc/6) [vdsm.api] START
connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options':
u'backup-volfile-servers=10.100.50.14:10.100.50.12', u'id':
u'ddab3ffa-1165-4495-a7d0-7515b21c3c8e', u'connection':
u'10.100.50.16:/ssd-storageone', u'iqn': u'', u'user':
u'',
u'tpgt': u'1', u'vfs_type': u'glusterfs',
u'password': '********', u'port': u''}], options=None)
from=::ffff:172.16.9.202,58810, flow_id=124752ae,
task_id=88fb682b-82b0-4d05-ac2f-90615901f616 (api:46)
2019-01-02 17:11:55,078-0500 INFO (jsonrpc/2) [vdsm.api] START
createStorageDomain(storageType=7, sdUUID=u'f5c5042a-3c8a-4629-aece-b5b50611399b',
domainName=u'Silver', typeSpecificArg=u'10.100.50.16:/ssd-storageone',
domClass=1, domVersion=u'4', options=None) from=::ffff:172.16.9.202,58810,
flow_id=124752ae, task_id=2163d5b7-49eb-4993-90c9-752
2bfd0708d (api:46)
Ovirt-engine.log
2019-01-02 17:04:32,999-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-130)
[41213d83-8594-41de-8e89-50786ecc7daa] EVENT_ID: GLUSTER_VOLUME_CREATE_FAILED(4,001),
Creation of Gluster Volume ssd-storageone failed on cluster Wander-Cluster.
2019-01-02 17:05:15,818-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[7efd0973-06ad-450e-b042-eb84f15de0f8] EVENT_ID: GLUSTER_VOLUME_CREATE_FAILED(4,001),
Creation of Gluster Volume ssd-storageone failed on cluster Wander-Cluster.
2019-01-02 17:08:21,392-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[44a5a53b-4687-4fdb-a5fc-ed18db00012b] EVENT_ID: GLUSTER_VOLUME_CREATE(4,000), Gluster
Volume ssd-storageone created on cluster Wander-Cluster.
2019-01-02 17:08:21,424-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (default
task-133) [7c032c48] START, SetGlusterVolumeOptionVDSCommand(HostName = Daedalus,
GlusterVolumeOptionVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 181d15dc
2019-01-02 17:08:22,205-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[7c032c48] EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option nfs.disable=on set
on ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:22,241-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (default
task-133) [7afc62eb] START, SetGlusterVolumeOptionVDSCommand(HostName = Daedalus,
GlusterVolumeOptionVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 30e0b58a
2019-01-02 17:08:22,793-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[7afc62eb] EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option user.cifs=disable
set on ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:22,822-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (default
task-133) [65a0f41f] START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 6b3968f0
2019-01-02 17:08:23,417-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[65a0f41f] EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option auth.allow=* set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:23,549-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Daedalus,
GlusterVolumeOptionVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 67ce8b8e
2019-01-02 17:08:24,641-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option group=virt set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:24,659-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 5cd0588e
2019-01-02 17:08:25,284-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option storage.owner-uid=36 set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:25,299-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 35488f82
2019-01-02 17:08:25,952-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option storage.owner-gid=36 set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:25,967-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 1068e722
2019-01-02 17:08:26,626-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option performance.strict-o-direct=on
set on ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:26,640-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Prometheus,
GlusterVolumeOptionVDSParameters:{hostId='8c0155dd-9c0a-4f88-b0aa-c777a57ef988',
volumeName='ssd-storageone'}), log id: 2869456
2019-01-02 17:08:27,312-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option network.remote-dio=off set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:27,326-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 1a4050bc
2019-01-02 17:08:27,960-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option
cluster.granular-entry-heal=enable set on ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:27,977-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
START, SetGlusterVolumeOptionVDSCommand(HostName = Prometheus,
GlusterVolumeOptionVDSParameters:{hostId='8c0155dd-9c0a-4f88-b0aa-c777a57ef988',
volumeName='ssd-storageone'}), log id: 25bd2afd
2019-01-02 17:08:28,576-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16572) [d8e919c7-77af-4d44-9391-c451d50c03b5]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option network.ping-timeout=30 set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:08:29,722-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.shd-max-threads=8 on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,722-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.shd-max-threads'='8' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,725-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option performance.io-cache=off on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,725-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'performance.io-cache'='off' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,728-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.locking-scheme=granular on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,728-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.locking-scheme'='granular' set on volume 'ssd-storageone'
from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,731-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option performance.client-io-threads=off on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,731-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'performance.client-io-threads'='off' set on volume
'ssd-storageone' from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,733-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.eager-lock=enable on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,733-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.eager-lock'='enable' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,736-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option performance.low-prio-threads=32 on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,736-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'performance.low-prio-threads'='32' set on volume 'ssd-storageone'
from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,739-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.quorum-type=auto on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,739-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.quorum-type'='auto' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,741-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option features.shard=on on volume ssd-storageone of cluster Wander-Cluster,
and added it to engine DB.
2019-01-02 17:08:29,741-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option 'features.shard'='on' set
on volume 'ssd-storageone' from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,745-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option performance.read-ahead=off on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,745-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'performance.read-ahead'='off' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,747-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.server-quorum-type=server on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,747-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.server-quorum-type'='server' set on volume
'ssd-storageone' from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,750-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option performance.quick-read=off on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,750-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'performance.quick-read'='off' set on volume 'ssd-storageone' from
gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,753-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.data-self-heal-algorithm=full on volume ssd-storageone of
cluster Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,753-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.data-self-heal-algorithm'='full' set on volume
'ssd-storageone' from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,755-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FROM_CLI(4,028),
Detected new option cluster.shd-wait-qlength=10000 on volume ssd-storageone of cluster
Wander-Cluster, and added it to engine DB.
2019-01-02 17:08:29,756-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] New option
'cluster.shd-wait-qlength'='10000' set on volume 'ssd-storageone'
from gluster CLI. Updating engine DB accordingly.
2019-01-02 17:08:29,761-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler4) [552ce8c5] EVENT_ID:
GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI(4,041), Detected change in value of option
user.cifs from disable to off on volume ssd-storageone of cluster Wander-Cluster, and
updated it to engine DB.
2019-01-02 17:08:29,761-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] Detected change in value of option
'user.cifs=disable' of volume 'ssd-storageone' from 'disable' to
'off'. Updating engine DB accordingly.
2019-01-02 17:08:29,762-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler4) [552ce8c5] Detected option 'group' reset on volume
'ssd-storageone'. Removing it from engine DB as well.
2019-01-02 17:09:19,743-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.StartGlusterVolumeVDSCommand] (default task-134)
[65340ca2-b521-4819-8b5a-9c5c7ef6e849] START, StartGlusterVolumeVDSCommand(HostName =
Icarus,
GlusterVolumeActionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 6226ecb2
2019-01-02 17:09:23,243-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-133)
[] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
10.100.50.16:/gluster_bricks/ssd-storageone/ssd-storageone of volume ssd-storageone of
cluster Wander-Cluster from UP to UP via gluster event.
2019-01-02 17:09:27,844-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(default task-134) [65340ca2-b521-4819-8b5a-9c5c7ef6e849] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Icarus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 49051252
2019-01-02 17:09:29,513-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-134)
[65340ca2-b521-4819-8b5a-9c5c7ef6e849] EVENT_ID: GLUSTER_VOLUME_START(4,004), Gluster
Volume ssd-storageone of cluster Wander-Cluster started.
2019-01-02 17:11:41,483-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-134) [5e9f8d6e-2b8e-4e6d-abda-d528fe7ae3a4] START,
ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='null',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 5250290c
2019-01-02 17:11:42,098-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-134) [300c8c7a] START, ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='ab4b5ccb-3d8b-4fa5-add3-7534e3672772',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 1e400372
2019-01-02 17:11:42,112-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-134) [300c8c7a] START, CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='c68931fd-42d5-4b1e-9ad3-5933311da71d'}',
args='10.100.50.16:/ssd-storageone'}), log id: 7a64fc6d
2019-01-02 17:11:42,590-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-134)
[300c8c7a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM Icarus command
CreateStorageDomainVDS failed: Error creating a storage domain: (u'storageType=7,
sdUUID=c68931fd-42d5-4b1e-9ad3-5933311da71d, domainName=Silver, domClass=1,
typeSpecificArg=10.100.50.16:/ssd-storageone domVersion=4',)
2019-01-02 17:11:42,590-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-134) [300c8c7a] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=351, message=Error creating a storage
domain: (u'storageType=7, sdUUID=c68931fd-42d5-4b1e-9ad3-5933311da71d,
domainName=Silver, domClass=1, typeSpecificArg=10.100.50.16:/ssd-storageone
domVersion=4',)]]'
2019-01-02 17:11:42,590-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-134) [300c8c7a] Command 'CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='c68931fd-42d5-4b1e-9ad3-5933311da71d'}',
args='10.100.50.16:/ssd-storageone'})' execution failed: VDSGenericException:
VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage
domain: (u'storageType=7, sdUUID=c68931fd-42d5-4b1e-9ad3-5933311da71d,
domainName=Silver, domClass=1, typeSpecificArg=10.100.50.16:/ssd-storageone
domVersion=4',), code = 351
2019-01-02 17:11:42,590-05 ERROR
[org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand] (default
task-134) [300c8c7a] Command
'org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand'
failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error
creating a storage domain: (u'storageType=7,
sdUUID=c68931fd-42d5-4b1e-9ad3-5933311da71d, domainName=Silver, domClass=1,
typeSpecificArg=10.100.50.16:/ssd-storageone domVersion=4',), code = 351 (Failed with
error StorageDomainCreationError and code 351)
2019-01-02 17:11:42,745-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default
task-133) [b4ebe5ed-9626-43ca-a307-b3b3e7d01f2b] START,
DisconnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='ab4b5ccb-3d8b-4fa5-add3-7534e3672772',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 45987e16
2019-01-02 17:11:54,583-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-133) [712a4a82-b947-4e18-b6e4-dacf0b09d4b4] START,
ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='null',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 66814b6c
2019-01-02 17:11:55,070-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-133) [124752ae] START, ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='ddab3ffa-1165-4495-a7d0-7515b21c3c8e',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: fe2bebd
2019-01-02 17:11:55,082-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [124752ae] START, CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='f5c5042a-3c8a-4629-aece-b5b50611399b'}',
args='10.100.50.16:/ssd-storageone'}), log id: 7f955a00
2019-01-02 17:11:55,557-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [124752ae] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=361, message=Storage domain is not empty
- requires cleaning: (u'10.100.50.16:/ssd-storageone',)]]'
2019-01-02 17:11:55,557-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [124752ae] Command 'CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='f5c5042a-3c8a-4629-aece-b5b50611399b'}',
args='10.100.50.16:/ssd-storageone'})' execution failed: VDSGenericException:
VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain
is not empty - requires cleaning: (u'10.100.50.16:/ssd-storageone',)
2019-01-02 17:11:55,557-05 ERROR
[org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand] (default
task-133) [124752ae] Command
'org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand'
failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS,
error = Storage domain is not empty - requires cleaning:
(u'10.100.50.16:/ssd-storageone',) (Failed with error StorageDomainNotEmpty and
code 361)
2019-01-02 17:11:55,747-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default
task-133) [6d58781b-8723-48b9-a1a7-36e93fcdc62b] START,
DisconnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='ddab3ffa-1165-4495-a7d0-7515b21c3c8e',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 300afade
2019-01-02 17:12:04,754-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-133) [ea7c92c2-14ec-447d-a501-4ed3b38327fc] START,
ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='null',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 3ba89582
2019-01-02 17:12:05,295-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-133) [126de8b0] START, ConnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='fe0cdd72-07e8-4476-8bdf-184d32396600',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 14a5863
2019-01-02 17:12:05,309-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [126de8b0] START, CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='c78cece3-70b3-4f5f-9ef5-ba5790d573dc'}',
args='10.100.50.16:/ssd-storageone'}), log id: 498dbaf7
2019-01-02 17:12:05,694-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [126de8b0] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=361, message=Storage domain is not empty
- requires cleaning: (u'10.100.50.16:/ssd-storageone',)]]'
2019-01-02 17:12:05,694-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default
task-133) [126de8b0] Command 'CreateStorageDomainVDSCommand(HostName = Icarus,
CreateStorageDomainVDSCommandParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storageDomain='StorageDomainStatic:{name='Silver',
id='c78cece3-70b3-4f5f-9ef5-ba5790d573dc'}',
args='10.100.50.16:/ssd-storageone'})' execution failed: VDSGenericException:
VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain
is not empty - requires cleaning: (u'10.100.50.16:/ssd-storageone',)
2019-01-02 17:12:05,694-05 ERROR
[org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand] (default
task-133) [126de8b0] Command
'org.ovirt.engine.core.bll.storage.domain.AddGlusterFsStorageDomainCommand'
failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS,
error = Storage domain is not empty - requires cleaning:
(u'10.100.50.16:/ssd-storageone',) (Failed with error StorageDomainNotEmpty and
code 361)
2019-01-02 17:12:05,795-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default
task-134) [17f3ac07-5069-4ab7-b286-a5fc200a7ba9] START,
DisconnectStorageServerVDSCommand(HostName = Icarus,
StorageServerConnectionManagementVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='GLUSTERFS',
connectionList='[StorageServerConnections:{id='fe0cdd72-07e8-4476-8bdf-184d32396600',
connection='10.100.50.16:/ssd-storageone', iqn='null',
vfsType='glusterfs',
mountOptions='backup-volfile-servers=10.100.50.14:10.100.50.12',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 57e9da28
2019-01-02 17:12:16,983-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Prometheus,
GlusterVolumeOptionVDSParameters:{hostId='8c0155dd-9c0a-4f88-b0aa-c777a57ef988',
volumeName='ssd-storageone'}), log id: 3284cb5e
2019-01-02 17:12:18,080-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option group=virt set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:12:18,093-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 5b67ca4a
2019-01-02 17:12:18,735-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option storage.owner-uid=36 set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:12:18,756-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Daedalus,
GlusterVolumeOptionVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 199be138
2019-01-02 17:12:19,463-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option storage.owner-gid=36 set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:12:19,475-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Daedalus,
GlusterVolumeOptionVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 22879e9f
2019-01-02 17:12:20,140-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option performance.strict-o-direct=on
set on ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:12:20,153-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Icarus,
GlusterVolumeOptionVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 2ce70403
2019-01-02 17:12:20,807-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_ADDED(4,002), Volume Option network.remote-dio=off set on
ssd-storageone of cluster Wander-Cluster.
2019-01-02 17:12:20,820-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
START, SetGlusterVolumeOptionVDSCommand(HostName = Prometheus,
GlusterVolumeOptionVDSParameters:{hostId='8c0155dd-9c0a-4f88-b0aa-c777a57ef988',
volumeName='ssd-storageone'}), log id: 70b3aa2f
2019-01-02 17:12:21,043-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-16591) [bb744659-cef6-4bca-bfd8-af3d0832aca6]
EVENT_ID: GLUSTER_VOLUME_OPTION_SET_FAILED(4,003), Volume Option
cluster.granular-entry-heal=enable could not be set on ssd-storageone of cluster
Wander-Cluster.
2019-01-02 17:12:21,229-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeHealInfoVDSCommand]
(DefaultQuartzScheduler4) [8575d05] START, GetGlusterVolumeHealInfoVDSCommand(HostName =
Daedalus,
GlusterVolumeVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 4c5a9b16
2019-01-02 17:12:24,244-05 INFO [org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler8) [ae1e46c] Detected option 'group' reset on volume
'ssd-storageone'. Removing it from engine DB as well.
2019-01-02 17:12:39,565-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [79337af0] EVENT_ID:
GLUSTER_VOLUME_SNAPSHOT_VOLUME_CONFIG_DETECTED_NEW(4,155), Found new gluster volume
snapshot configuration snap-max-hard-limit with value 256 for volume ssd-storageone on
cluster Wander-Cluster, and added it to engine DB."
2019-01-02 17:14:09,946-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler10) [58c36899] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Daedalus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 5be1d3da
2019-01-02 17:19:21,957-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler3) [4d122ce5] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Icarus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 3ccc9205
2019-01-02 17:22:21,975-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeHealInfoVDSCommand]
(DefaultQuartzScheduler2) [314f1fee] START, GetGlusterVolumeHealInfoVDSCommand(HostName =
Icarus, GlusterVolumeVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 5dbfffbb
2019-01-02 17:24:33,947-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler2) [5f1db4b6] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Daedalus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 3d76cc06
2019-01-02 17:29:45,860-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler7) [338cc9b2] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Daedalus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='64991852-0106-4daf-baa6-b2b4b7272385',
volumeName='ssd-storageone'}), log id: 7e372372
2019-01-02 17:32:22,568-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeHealInfoVDSCommand]
(DefaultQuartzScheduler8) [6221273c] START, GetGlusterVolumeHealInfoVDSCommand(HostName =
Icarus, GlusterVolumeVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 2f21cea7
2019-01-02 17:34:57,879-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler4) [72252954] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Prometheus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='8c0155dd-9c0a-4f88-b0aa-c777a57ef988',
volumeName='ssd-storageone'}), log id: c6e4fcd
2019-01-02 17:40:10,045-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler9) [21075cfb] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Icarus,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='5864fabe-37a2-4682-97b9-097dfd2b79c2',
volumeName='ssd-storageone'}), log id: 13cb600d