4:56 a.m.
Hey guys,
I have recently updated (yesterday) my platform to latest available (v4.3.3.7) and
upgraded to gluster v6.1 .The setup is hyperconverged 3 node cluster with ovirt1/gluster1
& ovirt2/gluster2 as replica nodes (glusterX is for gluster communication) while
ovirt3 is the arbiter.
Today I have tried to add new domain storages but they fail with the following:
2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [vdsm.api] FINISH createStorageDomain
error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n" from=::ffff:192.168.1.2,43864, flow_id=4a54578a,
task_id=d2535d0f-c7f7-4f31-a10f-704923ce1790 (api:52)
2019-05-16 10:15:21,296+0300 ERROR (jsonrpc/2) [storage.TaskManager.Task]
(Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in createStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2614, in
createStorageDomain
storageType, domVersion, block_size, alignment)
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 106, in
create
block_size)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 466, in
_prepareMetadata
cls.format_external_leases(sdUUID, xleases_path)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 1255, in
format_external_leases
xlease.format_index(lockspace, backend)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 681, in
format_index
index.dump(file)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 843, in
dump
file.pwrite(INDEX_BASE, self._buf)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1076, in
pwrite
self._run(args, data=buf[:])
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1093, in
_run
raise cmdutils.Error(args, rc, "[suppressed]", err)
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n"
2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [storage.TaskManager.Task]
(Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') aborting: Task is aborted:
u'Command [\'/usr/bin/dd\', \'iflag=fullblock\',
u\'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\',
\'oflag=direct,seek_bytes\', \'seek=1048576\', \'bs=256512\',
\'count=1\', \'conv=notrunc,nocreat,fsync\'] failed with rc=1
out=\'[suppressed]\' err="/usr/bin/dd: error writing
\'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\':
Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.0138582 s,
0.0 kB/s\\n"' - code 100 (task:1181)
2019-05-16 10:15:21,297+0300 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH
createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n" (dispatcher:87)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74,
in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
prepare
raise self.error
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n"
2019-05-16 10:15:21,297+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
StorageDomain.create failed (error 351) in 0.45 seconds (__init__:312)
2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [vdsm.api] START
disconnectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options': u'backup-volfile-servers=gluster2:ovirt3',
u'id': u'7442e9ab-dc54-4b9a-95d9-5d98a1e81b05', u'connection':
u'gluster1:/data_fast2', u'iqn': u'', u'user':
u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
u'vfs_type': u'glusterfs', u'password': '********',
u'port': u''}], options=None) from=::ffff:192.168.1.2,43864,
flow_id=33ced9b2-cdd5-4147-a223-d0eb398a2daf, task_id=a9a8f90a-1603-40c6-a959-3cbff29d1d7b
(api:48)
2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2 (mount:212)
I have tested manually mounting and trying it again:
[root@ovirt1 logs]# mount -t glusterfs -o backupvolfile-server=gluster2:ovirt3
gluster1:/data_fast2 /mnt
[root@ovirt1 logs]# cd /mnt/
[root@ovirt1 mnt]# ll
total 0
[root@ovirt1 mnt]# dd if=/dev/zero of=file bs=4M status=progress count=250
939524096 bytes (940 MB) copied, 8.145447 s, 115 MB/s
250+0 records in
250+0 records out
1048576000 bytes (1.0 GB) copied, 9.08347 s, 115 MB/s
[root@ovirt1 mnt]# /usr/bin/dd iflag=fullblock of=file oflag=direct,seek_bytes
seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync status=progress
^C0+0 records in
0+0 records out
0 bytes (0 B) copied, 46.5877 s, 0.0 kB/s
Can someone give a hint ? Maybe it's related to gluster v6 ?
Can someone test with older version of Gluster ?
Best Regards,Strahil Nikolov
5:07 a.m.
Why did you move to gluster v6? For the kicks? :-) The devs are currently evaluating for
themselves whether they can switch to V6 for the upcoming releases.
6:59 a.m.
New subject: ovirt 4.3.3.7 cannot create a gluster storage domain
Due to the issue with dom_md/ids not getting in sync and always pending heal on
ovirt2/gluster2 & ovirt3
Best Regards,Strahil Nikolov
В четвъртък, 16 май 2019 г., 6:08:44 ч. Гринуич-4, Andreas Elvers
<andreas.elvers+ovirtforum(a)solutions.work> написа:
Why did you move to gluster v6? For the kicks? :-) The devs are currently evaluating for
themselves whether they can switch to V6 for the upcoming releases.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JYZRKA4QBTX...
6:45 a.m.
New subject: ovirt 4.3.3.7 cannot create a gluster storage domain
Yeah. But I think this ist just an artefact of the current version. All images are in
sync.
dom_md/ids is an obsolete file anyway as the docs say.
see vdsm/block-storage-domains
https://www.ovirt.org/develop/developer-guide/vdsm/block-storage-domains....
10:13 a.m.
On Fri, May 17, 2019 at 2:47 PM Andreas Elvers <
andreas.elvers+ovirtforum(a)solutions.work> wrote:
Yeah. But I think this ist just an artefact of the current version.
All
images are in sync.
dom_md/ids is an obsolete file anyway as the docs say.
This page was correct about 10 years ago, the ids file is used for sanlock
delta leases, which are
the core infrastructure of oVirt. Without this file, you will not have any
kind of storage.
Please use RHV documentation:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/
And the source:
https://github.com/ovirt
Anything else is not reliable source for information.
Nir
10:22 a.m.
On Fri, May 17, 2019 at 6:13 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, May 17, 2019 at 2:47 PM Andreas Elvers <
andreas.elvers+ovirtforum(a)solutions.work> wrote:
> Yeah. But I think this ist just an artefact of the current version. All
> images are in sync.
> dom_md/ids is an obsolete file anyway as the docs say.
>
This page was correct about 10 years ago, the ids file is used for sanlock
delta leases, which are
the core infrastructure of oVirt. Without this file, you will not have any
kind of storage.
Should be fixed in:
https://github.com/oVirt/ovirt-site/pull/1994
Please use RHV documentation:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/
And the source:
https://github.com/ovirt
Anything else is not reliable source for information.
Nir
12:05 p.m.
On Mon, May 20, 2019 at 12:46 PM Andreas Elvers <
andreas.elvers+ovirtforum(a)solutions.work> wrote:
> Without this file [dom_md/ids], you will not have any
> kind of storage.
Ok. Sounds I'm kind of in trouble with that file being un-healable by
gluster?
Yes, but the good news is that you can easily initialize this file if it
got corrupted,
since it keeps only temporary hosts status.
https://lists.ovirt.org/pipermail/users/2016-February/038046.html
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FIR74ALG7WJ...
7:23 a.m.
New subject: Fw: ovirt 4.3.3.7 cannot create a gluster storage domain
It seems that the issue is within the 'dd' command as it stays waiting for
input:
[root@ovirt1 mnt]# /usr/bin/dd iflag=fullblock of=file oflag=direct,seek_bytes
seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync ^C0+0 records in
0+0 records out
0 bytes (0 B) copied, 19.3282 s, 0.0 kB/s
Changing the dd command works and shows that the gluster is working:
[root@ovirt1 mnt]# cat /dev/urandom | /usr/bin/dd of=file oflag=direct,seek_bytes
seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync 0+1 records in
0+1 records out
131072 bytes (131 kB) copied, 0.00705081 s, 18.6 MB/s
Best Regards,Strahil Nikolov
----- Препратено съобщение ----- От: Strahil Nikolov <hunter86_bg(a)yahoo.xn--com>-5gg4g:
Users <users(a)ovirt.xn--org>-8fg9bsk7dnkp7a: четвъртък, 16 май 2019 г., 5:56:44 ч.
Гринуич-4Тема: ovirt 4.3.3.7 cannot create a gluster storage domain
Hey guys,
I have recently updated (yesterday) my platform to latest available (v4.3.3.7) and
upgraded to gluster v6.1 .The setup is hyperconverged 3 node cluster with ovirt1/gluster1
& ovirt2/gluster2 as replica nodes (glusterX is for gluster communication) while
ovirt3 is the arbiter.
Today I have tried to add new domain storages but they fail with the following:
2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [vdsm.api] FINISH createStorageDomain
error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n" from=::ffff:192.168.1.2,43864, flow_id=4a54578a,
task_id=d2535d0f-c7f7-4f31-a10f-704923ce1790 (api:52)
2019-05-16 10:15:21,296+0300 ERROR (jsonrpc/2) [storage.TaskManager.Task]
(Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in createStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2614, in
createStorageDomain
storageType, domVersion, block_size, alignment)
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 106, in
create
block_size)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 466, in
_prepareMetadata
cls.format_external_leases(sdUUID, xleases_path)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 1255, in
format_external_leases
xlease.format_index(lockspace, backend)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 681, in
format_index
index.dump(file)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 843, in
dump
file.pwrite(INDEX_BASE, self._buf)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1076, in
pwrite
self._run(args, data=buf[:])
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1093, in
_run
raise cmdutils.Error(args, rc, "[suppressed]", err)
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n"
2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [storage.TaskManager.Task]
(Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') aborting: Task is aborted:
u'Command [\'/usr/bin/dd\', \'iflag=fullblock\',
u\'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\',
\'oflag=direct,seek_bytes\', \'seek=1048576\', \'bs=256512\',
\'count=1\', \'conv=notrunc,nocreat,fsync\'] failed with rc=1
out=\'[suppressed]\' err="/usr/bin/dd: error writing
\'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\':
Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.0138582 s,
0.0 kB/s\\n"' - code 100 (task:1181)
2019-05-16 10:15:21,297+0300 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH
createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n" (dispatcher:87)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74,
in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
prepare
raise self.error
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0
kB/s\n"
2019-05-16 10:15:21,297+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
StorageDomain.create failed (error 351) in 0.45 seconds (__init__:312)
2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [vdsm.api] START
disconnectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'mnt_options': u'backup-volfile-servers=gluster2:ovirt3',
u'id': u'7442e9ab-dc54-4b9a-95d9-5d98a1e81b05', u'connection':
u'gluster1:/data_fast2', u'iqn': u'', u'user':
u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
u'vfs_type': u'glusterfs', u'password': '********',
u'port': u''}], options=None) from=::ffff:192.168.1.2,43864,
flow_id=33ced9b2-cdd5-4147-a223-d0eb398a2daf, task_id=a9a8f90a-1603-40c6-a959-3cbff29d1d7b
(api:48)
2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2 (mount:212)
I have tested manually mounting and trying it again:
[root@ovirt1 logs]# mount -t glusterfs -o backupvolfile-server=gluster2:ovirt3
gluster1:/data_fast2 /mnt
[root@ovirt1 logs]# cd /mnt/
[root@ovirt1 mnt]# ll
total 0
[root@ovirt1 mnt]# dd if=/dev/zero of=file bs=4M status=progress count=250
939524096 bytes (940 MB) copied, 8.145447 s, 115 MB/s
250+0 records in
250+0 records out
1048576000 bytes (1.0 GB) copied, 9.08347 s, 115 MB/s
[root@ovirt1 mnt]# /usr/bin/dd iflag=fullblock of=file oflag=direct,seek_bytes
seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync status=progress
^C0+0 records in
0+0 records out
0 bytes (0 B) copied, 46.5877 s, 0.0 kB/s
Can someone give a hint ? Maybe it's related to gluster v6 ?
Can someone test with older version of Gluster ?
Best Regards,Strahil Nikolov
1823
days inactive
1829
days old
8 comments
3 participants
participants (3)
-
Andreas Elvers -
Nir Soffer -
Strahil Nikolov