One brick was at a point down for replacement. It has been replaced and all vols are up Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------ ------------------ Brick ovirt0:/gluster/brick3/data 49152 0 Y 22467 Brick ovirt2:/gluster/brick3/data 49152 0 Y 20736 Brick ovirt3:/gluster/brick3/data 49152 0 Y 23148 Brick ovirt0:/gluster/brick4/data 49153 0 Y 22497 Brick ovirt2:/gluster/brick4/data 49153 0 Y 20742 Brick ovirt3:/gluster/brick4/data 49153 0 Y 23158 Brick ovirt0:/gluster/brick5/data 49154 0 Y 22473 Brick ovirt2:/gluster/brick5/data 49154 0 Y 20748 Brick ovirt3:/gluster/brick5/data 49154 0 Y 23156 Brick ovirt0:/gluster/brick6/data 49155 0 Y 22479 Brick ovirt2:/gluster/brick6_1/data 49161 0 Y 21203 Brick ovirt3:/gluster/brick6/data 49155 0 Y 23157 Brick ovirt0:/gluster/brick7/data 49156 0 Y 22485 Brick ovirt2:/gluster/brick7/data 49156 0 Y 20763 Brick ovirt3:/gluster/brick7/data 49156 0 Y 23155 Brick ovirt0:/gluster/brick8/data 49157 0 Y 22491 Brick ovirt2:/gluster/brick8/data 49157 0 Y 20771 Brick ovirt3:/gluster/brick8/data 49157 0 Y 23154 Self-heal Daemon on localhost N/A N/A Y 23238 Bitrot Daemon on localhost N/A N/A Y 24870 Scrubber Daemon on localhost N/A N/A Y 24889 Self-heal Daemon on ovirt2 N/A N/A Y 24271 Bitrot Daemon on ovirt2 N/A N/A Y 24856 Scrubber Daemon on ovirt2 N/A N/A Y 24866 Self-heal Daemon on ovirt0 N/A N/A Y 29409 Bitrot Daemon on ovirt0 N/A N/A Y 5457 Scrubber Daemon on ovirt0 N/A N/A Y 5468 Task Status of Volume data ------------------------------------------------------------ ------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------ ------------------ Brick ovirt0:/gluster/brick1/engine 49158 0 Y 22511 Brick ovirt2:/gluster/brick1/engine 49158 0 Y 20780 Brick ovirt3:/gluster/brick1/engine 49158 0 Y 23199 Self-heal Daemon on localhost N/A N/A Y 23238 Self-heal Daemon on ovirt0 N/A N/A Y 29409 Self-heal Daemon on ovirt2 N/A N/A Y 24271 Task Status of Volume engine ------------------------------------------------------------ ------------------ There are no active volume tasks Status of volume: iso Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------ ------------------ Brick ovirt0:/gluster/brick2/iso 49159 0 Y 22520 Brick ovirt2:/gluster/brick2/iso 49159 0 Y 20789 Brick ovirt3:/gluster/brick2/iso 49159 0 Y 23208 NFS Server on localhost N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 23238 NFS Server on ovirt2 N/A N/A N N/A Self-heal Daemon on ovirt2 N/A N/A Y 24271 NFS Server on ovirt0 N/A N/A N N/A Self-heal Daemon on ovirt0 N/A N/A Y 29409 Task Status of Volume iso ------------------------------------------------------------ ------------------ There are no active volume tasks 2018-01-17 8:13 GMT+01:00 Gobinda Das <godas(a)redhat.com&gt;: > Hi, > I can see some error in log: > [2018-01-14 11:19:49.886571] E [socket.c:2309:socket_connect_finish] > 0-engine-client-0: connection to 10.2.0.120:24007 failed (Connection > timed out) > [2018-01-14 11:20:05.630669] E [socket.c:2309:socket_connect_finish] > 0-engine-client-0: connection to 10.2.0.120:24007 failed (Connection > timed out) > [2018-01-14 12:01:09.089925] E [MSGID: 114058] > [client-handshake.c:1527:client_query_portmap_cbk] 0-engine-client-0: > failed to get the port number for remote subvolume. Please run 'gluster > volume status' on server to see if brick process is running. > [2018-01-14 12:01:09.090048] I [MSGID: 114018] > [client.c:2280:client_rpc_notify] 0-engine-client-0: disconnected from > engine-client-0. Client process will keep trying to connect to glusterd > until brick's port is available > > Can you please check gluster volume status and see if all bricks are up? > > On Wed, Jan 17, 2018 at 12:24 PM, Endre Karlson <endre.karlson(a)gmail.com&gt; > wrote: > >> It's there now for each of the hosts. ovirt1 is not in service yet. >> >> 2018-01-17 5:52 GMT+01:00 Gobinda Das <godas(a)redhat.com&gt;: >> >>> In the above url only data and iso mnt log present,But there is no >>> engine and vmstore mount log. >>> >>> On Wed, Jan 17, 2018 at 1:26 AM, Endre Karlson <endre.karlson(a)gmail.com >>> > wrote: >>> >>>> Hi, all logs are located here: https://www.dropbox.com/ >>>> sh/3qzmwe76rkt09fk/AABzM9rJKbH5SBPWc31Npxhma?dl=0 for the mounts >>>> >>>> additionally we replaced a broken disk that is now resynced. >>>> >>>> 2018-01-15 11:17 GMT+01:00 Gobinda Das <godas(a)redhat.com&gt;: >>>> >>>>> Hi Endre, >>>>> Mount logs will be in below format inside /var/log/glusterfs : >>>>> >>>>> /var/log/glusterfs/rhev-data-center-mnt-glusterSD-*\:_engine.log >>>>> /var/log/glusterfs/rhev-data-center-mnt-glusterSD-*\:_data.log >>>>> /var/log/glusterfs/rhev-data-center-mnt-glusterSD-*\:_vmstore.log >>>>> >>>>> On Mon, Jan 15, 2018 at 11:57 AM, Endre Karlson < >>>>> endre.karlson(a)gmail.com&gt; wrote: >>>>> >>>>>> Hi. >>>>>> >>>>>> What are the gluster mount logs ? >>>>>> >>>>>> I have these gluster logs. >>>>>> cli.log etc-glusterfs-glusterd.vol.log >>>>>> glfsheal-engine.log glusterd.log nfs.log >>>>>> rhev-data-center-mnt-glusterSD-ovirt0:_engine.log >>>>>> rhev-data-center-mnt-glusterSD-ovirt3:_iso.log >>>>>> cmd_history.log glfsheal-data.log glfsheal-iso.log >>>>>> glustershd.log rhev-data-center-mnt-glusterSD-ovirt0:_data.log >>>>>> rhev-data-center-mnt-glusterSD-ovirt0:_iso.log statedump.log >>>>>> >>>>>> >>>>>> I am running version >>>>>> glusterfs-server-3.12.4-1.el7.x86_64 >>>>>> glusterfs-geo-replication-3.12.4-1.el7.x86_64 >>>>>> libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.7.x86_64 >>>>>> glusterfs-libs-3.12.4-1.el7.x86_64 >>>>>> glusterfs-api-3.12.4-1.el7.x86_64 >>>>>> python2-gluster-3.12.4-1.el7.x86_64 >>>>>> glusterfs-client-xlators-3.12.4-1.el7.x86_64 >>>>>> glusterfs-cli-3.12.4-1.el7.x86_64 >>>>>> glusterfs-events-3.12.4-1.el7.x86_64 >>>>>> glusterfs-rdma-3.12.4-1.el7.x86_64 >>>>>> vdsm-gluster-4.20.9.3-1.el7.centos.noarch >>>>>> glusterfs-3.12.4-1.el7.x86_64 >>>>>> glusterfs-fuse-3.12.4-1.el7.x86_64 >>>>>> >>>>>> // Endre >>>>>> >>>>>> 2018-01-15 6:11 GMT+01:00 Gobinda Das <godas(a)redhat.com&gt;: >>>>>> >>>>>>> Hi Endre, >>>>>>> Can you please provide glusterfs mount logs? >>>>>>> >>>>>>> On Mon, Jan 15, 2018 at 6:16 AM, Darrell Budic < >>>>>>> budic(a)onholyground.com&gt; wrote: >>>>>>> >>>>>>>> What version of gluster are you running? I’ve seen a few of these >>>>>>>> since moving my storage cluster to 12.3, but still haven’t been able to >>>>>>>> determine what’s causing it. Seems to be happening most often on VMs that >>>>>>>> haven’t been switches over to libgfapi mounts yet, but even one of those >>>>>>>> has paused once so far. They generally restart fine from the GUI, and >>>>>>>> nothing seems to need healing. >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> *From:* Endre Karlson <endre.karlson(a)gmail.com&gt; >>>>>>>> *Subject:* [ovirt-users] Problems with some vms >>>>>>>> *Date:* January 14, 2018 at 12:55:45 PM CST >>>>>>>> *To:* users >>>>>>>> >>>>>>>> Hi, we are getting some errors with some of our vms in a 3 node >>>>>>>> server setup. >>>>>>>> >>>>>>>> 2018-01-14 15:01:44,015+0100 INFO (libvirt/events) [virt.vm] >>>>>>>> (vmId='2c34f52d-140b-4dbe-a4bd-d2cb467b0b7c') abnormal vm stop >>>>>>>> device virtio-disk0 error eother (vm:4880) >>>>>>>> >>>>>>>> We are running glusterfs for shared storage. >>>>>>>> >>>>>>>> I have tried setting global maintenance on the first server and >>>>>>>> then issuing a 'hosted-engine --vm-start' but that leads to nowhere. >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Gobinda >>>>>>> +91-9019047912 <+91%2090190%2047912> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Gobinda >>>>> +91-9019047912 <+91%2090190%2047912> >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Gobinda >>> +91-9019047912 <+91%2090190%2047912> >>> >> >> > > > -- > Thanks, > Gobinda > +91-9019047912 <+91%2090190%2047912> >