<div dir="ltr">One brick was at a point down for replacement.<div><br></div><div>It has been replaced and all vols are up</div><div><br></div><div><div>Status of volume: data</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick ovirt0:/gluster/brick3/data 49152 0 Y 22467</div><div>Brick ovirt2:/gluster/brick3/data 49152 0 Y 20736</div><div>Brick ovirt3:/gluster/brick3/data 49152 0 Y 23148</div><div>Brick ovirt0:/gluster/brick4/data 49153 0 Y 22497</div><div>Brick ovirt2:/gluster/brick4/data 49153 0 Y 20742</div><div>Brick ovirt3:/gluster/brick4/data 49153 0 Y 23158</div><div>Brick ovirt0:/gluster/brick5/data 49154 0 Y 22473</div><div>Brick ovirt2:/gluster/brick5/data 49154 0 Y 20748</div><div>Brick ovirt3:/gluster/brick5/data 49154 0 Y 23156</div><div>Brick ovirt0:/gluster/brick6/data 49155 0 Y 22479</div><div>Brick ovirt2:/gluster/brick6_1/data 49161 0 Y 21203</div><div>Brick ovirt3:/gluster/brick6/data 49155 0 Y 23157</div><div>Brick ovirt0:/gluster/brick7/data 49156 0 Y 22485</div><div>Brick ovirt2:/gluster/brick7/data 49156 0 Y 20763</div><div>Brick ovirt3:/gluster/brick7/data 49156 0 Y 23155</div><div>Brick ovirt0:/gluster/brick8/data 49157 0 Y 22491</div><div>Brick ovirt2:/gluster/brick8/data 49157 0 Y 20771</div><div>Brick ovirt3:/gluster/brick8/data 49157 0 Y 23154</div><div>Self-heal Daemon on localhost N/A N/A Y 23238</div><div>Bitrot Daemon on localhost N/A N/A Y 24870</div><div>Scrubber Daemon on localhost N/A N/A Y 24889</div><div>Self-heal Daemon on ovirt2 N/A N/A Y 24271</div><div>Bitrot Daemon on ovirt2 N/A N/A Y 24856</div><div>Scrubber Daemon on ovirt2 N/A N/A Y 24866</div><div>Self-heal Daemon on ovirt0 N/A N/A Y 29409</div><div>Bitrot Daemon on ovirt0 N/A N/A Y 5457 </div><div>Scrubber Daemon on ovirt0 N/A N/A Y 5468 </div><div> </div><div>Task Status of Volume data</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div> </div><div>Status of volume: engine</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick ovirt0:/gluster/brick1/engine 49158 0 Y 22511</div><div>Brick ovirt2:/gluster/brick1/engine 49158 0 Y 20780</div><div>Brick ovirt3:/gluster/brick1/engine 49158 0 Y 23199</div><div>Self-heal Daemon on localhost N/A N/A Y 23238</div><div>Self-heal Daemon on ovirt0 N/A N/A Y 29409</div><div>Self-heal Daemon on ovirt2 N/A N/A Y 24271</div><div> </div><div>Task Status of Volume engine</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div> </div><div>Status of volume: iso</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick ovirt0:/gluster/brick2/iso 49159 0 Y 22520</div><div>Brick ovirt2:/gluster/brick2/iso 49159 0 Y 20789</div><div>Brick ovirt3:/gluster/brick2/iso 49159 0 Y 23208</div><div>NFS Server on localhost N/A N/A N N/A </div><div>Self-heal Daemon on localhost N/A N/A Y 23238</div><div>NFS Server on ovirt2 N/A N/A N N/A </div><div>Self-heal Daemon on ovirt2 N/A N/A Y 24271</div><div>NFS Server on ovirt0 N/A N/A N N/A </div><div>Self-heal Daemon on ovirt0 N/A N/A Y 29409</div><div> </div><div>Task Status of Volume iso</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2018-01-17 8:13 GMT+01:00 Gobinda Das <span dir="ltr"><<a href="mailto:godas@redhat.com" target="_blank">godas@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div> I can see some error in log:</div><div><div>[2018-01-14 11:19:49.886571] E [socket.c:2309:socket_connect_<wbr>finish] 0-engine-client-0: connection to <a href="http://10.2.0.120:24007" target="_blank">10.2.0.120:24007</a> failed (Connection timed out)</div><div>[2018-01-14 11:20:05.630669] E [socket.c:2309:socket_connect_<wbr>finish] 0-engine-client-0: connection to <a href="http://10.2.0.120:24007" target="_blank">10.2.0.120:24007</a> failed (Connection timed out)</div><div>[2018-01-14 12:01:09.089925] E [MSGID: 114058] [client-handshake.c:1527:clien<wbr>t_query_portmap_cbk] 0-engine-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.</div><div>[2018-01-14 12:01:09.090048] I [MSGID: 114018] [client.c:2280:client_rpc_noti<wbr>fy] 0-engine-client-0: disconnected from engine-client-0. Client process will keep trying to connect to glusterd until brick's port is available</div></div><div><br></div><div>Can you please check gluster volume status and see if all bricks are up?</div></div><div class="gmail_extra"><div><div class="gmail-m_-796592407099418223h5"><br><div class="gmail_quote">On Wed, Jan 17, 2018 at 12:24 PM, Endre Karlson <span dir="ltr"><<a href="mailto:endre.karlson@gmail.com" target="_blank">endre.karlson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It's there now for each of the hosts. ovirt1 is not in service yet.</div><div class="gmail-m_-796592407099418223m_-6332070998189379465HOEnZb"><div class="gmail-m_-796592407099418223m_-6332070998189379465h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-01-17 5:52 GMT+01:00 Gobinda Das <span dir="ltr"><<a href="mailto:godas@redhat.com" target="_blank">godas@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In the above url only data and iso mnt log present,But there is no engine and vmstore mount log. </div><div class="gmail_extra"><div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685h5"><br><div class="gmail_quote">On Wed, Jan 17, 2018 at 1:26 AM, Endre Karlson <span dir="ltr"><<a href="mailto:endre.karlson@gmail.com" target="_blank">endre.karlson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><span><div dir="ltr" style="font-size:12.8px">Hi, all logs are located here: <a href="https://www.dropbox.com/sh/3qzmwe76rkt09fk/AABzM9rJKbH5SBPWc31Npxhma?dl=0" target="_blank">https://www.dropbox.com/<wbr>sh/3qzmwe76rkt09fk/AABzM9rJKbH<wbr>5SBPWc31Npxhma?dl=0</a> for the mounts</div><div dir="ltr" style="font-size:12.8px"><br></div></span><div style="font-size:12.8px">additionally we replaced a broken disk that is now resynced.</div><div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181h5"><div style="font-size:12.8px"><br></div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-yj6qo gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-ajU" style="font-size:12.8px"></div><div class="gmail_quote">2018-01-15 11:17 GMT+01:00 Gobinda Das <span dir="ltr"><<a href="mailto:godas@redhat.com" target="_blank">godas@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Endre,<div> Mount logs will be in below format inside /var/log/glusterfs :</div><div><br></div><div> /var/log/glusterfs/rhev-data-<wbr>center-mnt-glusterSD-*\:_engin<wbr>e.log</div><div> /var/log/glusterfs/rhev-data-c<wbr>enter-mnt-glusterSD-*\:_data.l<wbr>og<br></div><div> /var/log/glusterfs/rhev-data-c<wbr>enter-mnt-glusterSD-*\:_vmstor<wbr>e.log<br></div></div><div class="gmail_extra"><div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-h5"><br><div class="gmail_quote">On Mon, Jan 15, 2018 at 11:57 AM, Endre Karlson <span dir="ltr"><<a href="mailto:endre.karlson@gmail.com" target="_blank">endre.karlson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi.<div><br></div><div>What are the gluster mount logs ?</div><div><br></div><div>I have these gluster logs.</div><div><div>cli.log etc-glusterfs-glusterd.vol.log<wbr> glfsheal-engine.log glusterd.log nfs.log rhev-data-center-mnt-glusterSD<wbr>-ovirt0:_engine.log rhev-data-center-mnt-glusterSD<wbr>-ovirt3:_iso.log</div><div>cmd_history.log glfsheal-data.log glfsheal-iso.log glustershd.log rhev-data-center-mnt-glusterSD<wbr>-ovirt0:_data.log rhev-data-center-mnt-glusterSD<wbr>-ovirt0:_iso.log statedump.log</div></div><div><br></div><div><br></div><div>I am running version</div><div><div>glusterfs-server-3.12.4-1.el7.<wbr>x86_64</div><div>glusterfs-geo-replication-3.12<wbr>.4-1.el7.x86_64</div><div>libvirt-daemon-driver-storage-<wbr>gluster-3.2.0-14.el7_4.7.x86_6<wbr>4</div><div>glusterfs-libs-3.12.4-1.el7.x8<wbr>6_64</div><div>glusterfs-api-3.12.4-1.el7.x86<wbr>_64</div><div>python2-gluster-3.12.4-1.el7.x<wbr>86_64</div><div>glusterfs-client-xlators-3.12.<wbr>4-1.el7.x86_64</div><div>glusterfs-cli-3.12.4-1.el7.x86<wbr>_64</div><div>glusterfs-events-3.12.4-1.el7.<wbr>x86_64</div><div>glusterfs-rdma-3.12.4-1.el7.x8<wbr>6_64</div><div>vdsm-gluster-4.20.9.3-1.el7.ce<wbr>ntos.noarch</div><div>glusterfs-3.12.4-1.el7.x86_64</div><div>glusterfs-fuse-3.12.4-1.el7.x8<wbr>6_64</div></div><span class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085HOEnZb"><font color="#888888"><div><br></div><div>// Endre</div></font></span></div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085HOEnZb"><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-01-15 6:11 GMT+01:00 Gobinda Das <span dir="ltr"><<a href="mailto:godas@redhat.com" target="_blank">godas@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Endre,<div> Can you please provide <span style="font-size:12.8px">glusterfs mount logs?</span></div></div><div class="gmail_extra"><div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027h5"><br><div class="gmail_quote">On Mon, Jan 15, 2018 at 6:16 AM, Darrell Budic <span dir="ltr"><<a href="mailto:budic@onholyground.com" target="_blank">budic@onholyground.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">What version of gluster are you running? I’ve seen a few of these since moving my storage cluster to 12.3, but still haven’t been able to determine what’s causing it. Seems to be happening most often on VMs that haven’t been switches over to libgfapi mounts yet, but even one of those has paused once so far. They generally restart fine from the GUI, and nothing seems to need healing.<div><br><div><blockquote type="cite"><hr style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:0px;margin:10px 0px 5px"><span id="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027m_-5619438456864580468m_-4352343014308123976RwhHeaderAttributes"><font face="Helvetica" size="4" color="#000000" style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Helvetica;color:rgb(0,0,0)"><b>From:</b> Endre Karlson <<a href="mailto:endre.karlson@gmail.com" target="_blank">endre.karlson@gmail.com</a>></font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Helvetica;color:rgb(0,0,0)"><b>Subject:</b> [ovirt-users] Problems with some vms</font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Helvetica;color:rgb(0,0,0)"><b>Date:</b> January 14, 2018 at 12:55:45 PM CST</font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Helvetica;color:rgb(0,0,0)"><b>To:</b> users</font></span><br>
<br class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027m_-5619438456864580468m_-4352343014308123976Apple-interchange-newline"><div><div><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027m_-5619438456864580468h5"><div dir="ltr">Hi, we are getting some errors with some of our vms in a 3 node server setup.<div><br></div><div><div>2018-01-14 15:01:44,015+0100 INFO (libvirt/events) [virt.vm] (vmId='2c34f52d-140b-4dbe-a4bd<wbr>-d2cb467b0b7c') abnormal vm stop device virtio-disk0 error eother (vm:4880)</div><div><br></div><div>We are running glusterfs for shared storage.</div><div><br></div><div>I have tried setting global maintenance on the first server and then issuing a 'hosted-engine --vm-start' but that leads to nowhere. </div></div></div></div></div>
______________________________<wbr>_________________<br>Users mailing list<br><a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br></div></blockquote></div><br></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027HOEnZb"><font color="#888888">-- <br><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085m_-615385065311145027m_-5619438456864580468gmail_signature"><div dir="ltr">Thanks,<div>Gobinda</div><div><a href="tel:+91%2090190%2047912" value="+919019047912" target="_blank">+91-9019047912</a></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-HOEnZb"><font color="#888888">-- <br><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181m_-4382999187967402779gmail-m_1430353528837899085gmail_signature"><div dir="ltr">Thanks,<div>Gobinda</div><div><a href="tel:+91%2090190%2047912" value="+919019047912" target="_blank">+91-9019047912</a></div></div></div>
</font></span></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685HOEnZb"><font color="#888888">-- <br><div class="gmail-m_-796592407099418223m_-6332070998189379465m_4657464254736145685m_7742980644771417181gmail_signature"><div dir="ltr">Thanks,<div>Gobinda</div><div><a href="tel:+91%2090190%2047912" value="+919019047912" target="_blank">+91-9019047912</a></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="gmail-m_-796592407099418223HOEnZb"><font color="#888888">-- <br><div class="gmail-m_-796592407099418223m_-6332070998189379465gmail_signature"><div dir="ltr">Thanks,<div>Gobinda</div><div><a href="tel:+91%2090190%2047912" value="+919019047912" target="_blank">+91-9019047912</a></div></div></div>
</font></span></div>
</blockquote></div><br></div></div></div>