[ovirt-users] Cannot find master domain
David Gossage
dgossage at carouselchecks.com
Thu Jul 28 14:49:12 UTC 2016
On Thu, Jul 28, 2016 at 9:38 AM, Siavash Safi <siavash.safi at gmail.com>
wrote:
> file system: xfs
> features.shard: off
>
Ok was just seeing if matched up to the issues latest 3.7.x releases have
with zfs and sharding but doesn't look like your issue.
In your logs I see it mounts with thee commands. What happens if you use
same to a test dir?
/usr/bin/mount -t glusterfs -o backup-volfile-servers=172.16.0.12:172.16.0.13
172.16.0.11:/ovirt /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt
It then umounts it and complains short while later of permissions.
StorageServerAccessPermissionError: Permission settings on the specified
path do not allow access to the storage. Verify permission settings on the
specified storage path.: 'path =
/rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'
Are the permissions of dirs to
/rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt
as expected?
How about on the bricks anything out of place?
Is gluster still using same options as before? could it have reset the
user and group to not be 36?
>
> On Thu, Jul 28, 2016 at 7:03 PM David Gossage <dgossage at carouselchecks.com>
> wrote:
>
>> On Thu, Jul 28, 2016 at 9:28 AM, Siavash Safi <siavash.safi at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Jul 28, 2016 at 6:29 PM David Gossage <
>>> dgossage at carouselchecks.com> wrote:
>>>
>>>> On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi <siavash.safi at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Issue: Cannot find master domain
>>>>> Changes applied before issue started to happen: replaced 172.16.0.12:/data/brick1/brick1
>>>>> with 172.16.0.12:/data/brick3/brick3, did minor package upgrades for
>>>>> vdsm and glusterfs
>>>>>
>>>>> vdsm log: https://paste.fedoraproject.org/396842/
>>>>>
>>>>
>>>>
>>>> Any errrors in glusters brick or server logs? The client gluster logs
>>>> from ovirt?
>>>>
>>> Brick errors:
>>> [2016-07-28 14:03:25.002396] E [MSGID: 113091]
>>> [posix.c:178:posix_lookup] 0-ovirt-posix: null gfid for path (null)
>>> [2016-07-28 14:03:25.002430] E [MSGID: 113018]
>>> [posix.c:196:posix_lookup] 0-ovirt-posix: lstat on null failed [Invalid
>>> argument]
>>> (Both repeated many times)
>>>
>>> Server errors:
>>> None
>>>
>>> Client errors:
>>> None
>>>
>>>
>>>>
>>>>> yum log: https://paste.fedoraproject.org/396854/
>>>>>
>>>>
>>>> What version of gluster was running prior to update to 3.7.13?
>>>>
>>> 3.7.11-1 from gluster.org repository(after update ovirt switched to
>>> centos repository)
>>>
>>
>> What file system do your bricks reside on and do you have sharding
>> enabled?
>>
>>
>>>> Did it create gluster mounts on server when attempting to start?
>>>>
>>> As I checked the master domain is not mounted on any nodes.
>>> Restarting vdsmd generated following errors:
>>>
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,661::fileUtils::143::Storage.fileUtils::(createdir) Creating
>>> directory: /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt mode: None
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,661::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option)
>>> Using bricks: ['172.16.0.11', '172.16.0.12', '172.16.0.13']
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,662::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/systemd-run --scope
>>> --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o
>>> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt
>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,789::__init__::318::IOProcessClient::(_run) Starting IOProcess...
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,802::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/umount -f -l
>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
>>> jsonrpc.Executor/5::ERROR::2016-07-28
>>> 18:50:57,813::hsm::2473::Storage.HSM::(connectStorageServer) Could not
>>> connect to storageServer
>>> Traceback (most recent call last):
>>> File "/usr/share/vdsm/storage/hsm.py", line 2470, in
>>> connectStorageServer
>>> conObj.connect()
>>> File "/usr/share/vdsm/storage/storageServer.py", line 248, in connect
>>> six.reraise(t, v, tb)
>>> File "/usr/share/vdsm/storage/storageServer.py", line 241, in connect
>>> self.getMountObj().getRecord().fs_file)
>>> File "/usr/share/vdsm/storage/fileSD.py", line 79, in validateDirAccess
>>> raise se.StorageServerAccessPermissionError(dirPath)
>>> StorageServerAccessPermissionError: Permission settings on the specified
>>> path do not allow access to the storage. Verify permission settings on the
>>> specified storage path.: 'path =
>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,817::hsm::2497::Storage.HSM::(connectStorageServer) knownSDs: {}
>>> jsonrpc.Executor/5::INFO::2016-07-28
>>> 18:50:57,817::logUtils::51::dispatcher::(wrapper) Run and protect:
>>> connectStorageServer, Return response: {'statuslist': [{'status': 469,
>>> 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]}
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,817::task::1191::Storage.TaskManager.Task::(prepare)
>>> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::finished: {'statuslist':
>>> [{'status': 469, 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]}
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,817::task::595::Storage.TaskManager.Task::(_updateState)
>>> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::moving from state preparing ->
>>> state finished
>>>
>>> I can manually mount the gluster volume on the same server.
>>>
>>>
>>>>
>>>>
>>>>> Setup:
>>>>> engine running on a separate node
>>>>> 3 x kvm/glusterd nodes
>>>>>
>>>>> Status of volume: ovirt
>>>>> Gluster process TCP Port RDMA Port
>>>>> Online Pid
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Brick 172.16.0.11:/data/brick1/brick1 49152 0 Y
>>>>> 17304
>>>>> Brick 172.16.0.12:/data/brick3/brick3 49155 0 Y
>>>>> 9363
>>>>> Brick 172.16.0.13:/data/brick1/brick1 49152 0 Y
>>>>> 23684
>>>>> Brick 172.16.0.11:/data/brick2/brick2 49153 0 Y
>>>>> 17323
>>>>> Brick 172.16.0.12:/data/brick2/brick2 49153 0 Y
>>>>> 9382
>>>>> Brick 172.16.0.13:/data/brick2/brick2 49153 0 Y
>>>>> 23703
>>>>> NFS Server on localhost 2049 0 Y
>>>>> 30508
>>>>> Self-heal Daemon on localhost N/A N/A Y
>>>>> 30521
>>>>> NFS Server on 172.16.0.11 2049 0 Y
>>>>> 24999
>>>>> Self-heal Daemon on 172.16.0.11 N/A N/A Y
>>>>> 25016
>>>>> NFS Server on 172.16.0.13 2049 0 Y
>>>>> 25379
>>>>> Self-heal Daemon on 172.16.0.13 N/A N/A Y
>>>>> 25509
>>>>>
>>>>> Task Status of Volume ovirt
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Task : Rebalance
>>>>> ID : 84d5ab2a-275e-421d-842b-928a9326c19a
>>>>> Status : completed
>>>>>
>>>>> Thanks,
>>>>> Siavash
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160728/ae4cc53e/attachment-0001.html>
More information about the Users
mailing list