Re: [ovirt-users] Cannot find master domain

On Thu, Jul 28, 2016 at 10:28 AM, Siavash Safi <siavash.safi(a)gmail.com&gt; wrote: > I created the directory with correct permissions: > drwxr-xr-x. 2 vdsm kvm 6 Jul 28 19:51 > /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt/ > > It was removed after I tried to activate the storage from web. > > Is dir missing on all 3 oVirt nodes? Did you create on all 3? When you did test mount with oVirts mount options did permissions on files after mount look proper? Can you read/write to mount? > Engine displays the master storage as inactive: > [image: oVirt_Engine_Web_Administration.png] > > > On Thu, Jul 28, 2016 at 7:40 PM David Gossage < > dgossage(a)carouselchecks.com&gt; wrote: > >> On Thu, Jul 28, 2016 at 10:00 AM, Siavash Safi <siavash.safi(a)gmail.com&gt; >> wrote: >> >>> >>> >>> On Thu, Jul 28, 2016 at 7:19 PM David Gossage < >>> dgossage(a)carouselchecks.com&gt; wrote: >>> >>>> On Thu, Jul 28, 2016 at 9:38 AM, Siavash Safi <siavash.safi(a)gmail.com&gt; >>>> wrote: >>>> >>>>> file system: xfs >>>>> features.shard: off >>>>> >>>> >>>> Ok was just seeing if matched up to the issues latest 3.7.x releases >>>> have with zfs and sharding but doesn't look like your issue. >>>> >>>> In your logs I see it mounts with thee commands. What happens if you >>>> use same to a test dir? >>>> >>>> /usr/bin/mount -t glusterfs -o backup-volfile-servers=172.16.0.12:172.16.0.13 >>>> 172.16.0.11:/ovirt /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt >>>> >>> >>> It mounts successfully: >>> [root@node1 ~]# /usr/bin/mount -t glusterfs -o >>> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt /mnt >>> [root@node1 ~]# ls /mnt/ >>> 4697fbde-45fb-4f91-ac4c-5516bc59f683 __DIRECT_IO_TEST__ >>> >>> >>>> It then umounts it and complains short while later of permissions. >>>> >>>> StorageServerAccessPermissionError: Permission settings on the >>>> specified path do not allow access to the storage. Verify permission >>>> settings on the specified storage path.: 'path = >>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt' >>>> >>>> Are the permissions of dirs to >>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt as expected? >>>> >>> >>> /rhev/data-center/mnt/glusterSD/ is empty. Maybe it remove the >>> directory after failure to cleanup? >>> >> >> Maybe though I don't recall it ever being deleted unless you maybe >> destroy detach storage. What if you create that directory and permissions >> appropriately on any node missing then try and activate storage? >> >> In engine is it still displaying the master storage domain? >> >> >>> How about on the bricks anything out of place? >>>> >>> >>> I didn't notice anything. >>> >>> >>>> Is gluster still using same options as before? could it have reset >>>> the user and group to not be 36? >>>> >>> >>> All options seem to be correct, to make sure I ran "Optimize for Virt >>> Store" from web. >>> >>> Volume Name: ovirt >>> Type: Distributed-Replicate >>> Volume ID: b224d9bc-d120-4fe1-b233-09089e5ca0b2 >>> Status: Started >>> Number of Bricks: 2 x 3 = 6 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 172.16.0.11:/data/brick1/brick1 >>> Brick2: 172.16.0.12:/data/brick3/brick3 >>> Brick3: 172.16.0.13:/data/brick1/brick1 >>> Brick4: 172.16.0.11:/data/brick2/brick2 >>> Brick5: 172.16.0.12:/data/brick2/brick2 >>> Brick6: 172.16.0.13:/data/brick2/brick2 >>> Options Reconfigured: >>> performance.readdir-ahead: on >>> nfs.disable: off >>> user.cifs: enable >>> auth.allow: * >>> performance.quick-read: off >>> performance.read-ahead: off >>> performance.io-cache: off >>> performance.stat-prefetch: off >>> cluster.eager-lock: enable >>> network.remote-dio: enable >>> cluster.quorum-type: auto >>> cluster.server-quorum-type: server >>> storage.owner-uid: 36 >>> storage.owner-gid: 36 >>> server.allow-insecure: on >>> network.ping-timeout: 10 >>> >>> >>>>> On Thu, Jul 28, 2016 at 7:03 PM David Gossage < >>>>> dgossage(a)carouselchecks.com&gt; wrote: >>>>> >>>>>> On Thu, Jul 28, 2016 at 9:28 AM, Siavash Safi < >>>>>> siavash.safi(a)gmail.com&gt; wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 28, 2016 at 6:29 PM David Gossage < >>>>>>> dgossage(a)carouselchecks.com&gt; wrote: >>>>>>> >>>>>>>> On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi < >>>>>>>> siavash.safi(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Issue: Cannot find master domain >>>>>>>>> Changes applied before issue started to happen: replaced >>>>>>>>> 172.16.0.12:/data/brick1/brick1 with 172.16.0.12:/data/brick3/brick3, >>>>>>>>> did minor package upgrades for vdsm and glusterfs >>>>>>>>> >>>>>>>>> vdsm log: https://paste.fedoraproject.org/396842/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Any errrors in glusters brick or server logs? The client gluster >>>>>>>> logs from ovirt? >>>>>>>> >>>>>>> Brick errors: >>>>>>> [2016-07-28 14:03:25.002396] E [MSGID: 113091] >>>>>>> [posix.c:178:posix_lookup] 0-ovirt-posix: null gfid for path (null) >>>>>>> [2016-07-28 14:03:25.002430] E [MSGID: 113018] >>>>>>> [posix.c:196:posix_lookup] 0-ovirt-posix: lstat on null failed [Invalid >>>>>>> argument] >>>>>>> (Both repeated many times) >>>>>>> >>>>>>> Server errors: >>>>>>> None >>>>>>> >>>>>>> Client errors: >>>>>>> None >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> yum log: https://paste.fedoraproject.org/396854/ >>>>>>>>> >>>>>>>> >>>>>>>> What version of gluster was running prior to update to 3.7.13? >>>>>>>> >>>>>>> 3.7.11-1 from gluster.org repository(after update ovirt switched >>>>>>> to centos repository) >>>>>>> >>>>>> >>>>>> What file system do your bricks reside on and do you have sharding >>>>>> enabled? >>>>>> >>>>>> >>>>>>>> Did it create gluster mounts on server when attempting to start? >>>>>>>> >>>>>>> As I checked the master domain is not mounted on any nodes. >>>>>>> Restarting vdsmd generated following errors: >>>>>>> >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,661::fileUtils::143::Storage.fileUtils::(createdir) Creating >>>>>>> directory: /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt >>>>>>> mode: None >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,661::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option) >>>>>>> Using bricks: ['172.16.0.11', '172.16.0.12', '172.16.0.13'] >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,662::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset >>>>>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/systemd-run --scope >>>>>>> --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o >>>>>>> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt >>>>>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None) >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,789::__init__::318::IOProcessClient::(_run) Starting IOProcess... >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,802::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset >>>>>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/umount -f -l >>>>>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None) >>>>>>> jsonrpc.Executor/5::ERROR::2016-07-28 >>>>>>> 18:50:57,813::hsm::2473::Storage.HSM::(connectStorageServer) Could not >>>>>>> connect to storageServer >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/share/vdsm/storage/hsm.py", line 2470, in >>>>>>> connectStorageServer >>>>>>> conObj.connect() >>>>>>> File "/usr/share/vdsm/storage/storageServer.py", line 248, in >>>>>>> connect >>>>>>> six.reraise(t, v, tb) >>>>>>> File "/usr/share/vdsm/storage/storageServer.py", line 241, in >>>>>>> connect >>>>>>> self.getMountObj().getRecord().fs_file) >>>>>>> File "/usr/share/vdsm/storage/fileSD.py", line 79, in >>>>>>> validateDirAccess >>>>>>> raise se.StorageServerAccessPermissionError(dirPath) >>>>>>> StorageServerAccessPermissionError: Permission settings on the >>>>>>> specified path do not allow access to the storage. Verify permission >>>>>>> settings on the specified storage path.: 'path = >>>>>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt' >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,817::hsm::2497::Storage.HSM::(connectStorageServer) knownSDs: {} >>>>>>> jsonrpc.Executor/5::INFO::2016-07-28 >>>>>>> 18:50:57,817::logUtils::51::dispatcher::(wrapper) Run and protect: >>>>>>> connectStorageServer, Return response: {'statuslist': [{'status': 469, >>>>>>> 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]} >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,817::task::1191::Storage.TaskManager.Task::(prepare) >>>>>>> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::finished: {'statuslist': >>>>>>> [{'status': 469, 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]} >>>>>>> jsonrpc.Executor/5::DEBUG::2016-07-28 >>>>>>> 18:50:57,817::task::595::Storage.TaskManager.Task::(_updateState) >>>>>>> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::moving from state preparing -> >>>>>>> state finished >>>>>>> >>>>>>> I can manually mount the gluster volume on the same server. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Setup: >>>>>>>>> engine running on a separate node >>>>>>>>> 3 x kvm/glusterd nodes >>>>>>>>> >>>>>>>>> Status of volume: ovirt >>>>>>>>> Gluster process TCP Port RDMA Port >>>>>>>>> Online Pid >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>> Brick 172.16.0.11:/data/brick1/brick1 49152 0 >>>>>>>>> Y 17304 >>>>>>>>> Brick 172.16.0.12:/data/brick3/brick3 49155 0 >>>>>>>>> Y 9363 >>>>>>>>> Brick 172.16.0.13:/data/brick1/brick1 49152 0 >>>>>>>>> Y 23684 >>>>>>>>> Brick 172.16.0.11:/data/brick2/brick2 49153 0 >>>>>>>>> Y 17323 >>>>>>>>> Brick 172.16.0.12:/data/brick2/brick2 49153 0 >>>>>>>>> Y 9382 >>>>>>>>> Brick 172.16.0.13:/data/brick2/brick2 49153 0 >>>>>>>>> Y 23703 >>>>>>>>> NFS Server on localhost 2049 0 >>>>>>>>> Y 30508 >>>>>>>>> Self-heal Daemon on localhost N/A N/A >>>>>>>>> Y 30521 >>>>>>>>> NFS Server on 172.16.0.11 2049 0 >>>>>>>>> Y 24999 >>>>>>>>> Self-heal Daemon on 172.16.0.11 N/A N/A >>>>>>>>> Y 25016 >>>>>>>>> NFS Server on 172.16.0.13 2049 0 >>>>>>>>> Y 25379 >>>>>>>>> Self-heal Daemon on 172.16.0.13 N/A N/A >>>>>>>>> Y 25509 >>>>>>>>> >>>>>>>>> Task Status of Volume ovirt >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>> Task : Rebalance >>>>>>>>> ID : 84d5ab2a-275e-421d-842b-928a9326c19a >>>>>>>>> Status : completed >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Siavash >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>>>> >>>>>>>>