<div dir="ltr">You'd need to allow some more time to dig into the logs. I'll try to get back on this by Monday.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 7, 2017 at 2:23 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="h5"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jul 6, 2017 at 3:22 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="m_8155841319646511931gmail-"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jul 6, 2017 at 2:16 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_8155841319646511931gmail-m_5986593071559009587gmail-">On Thu, Jul 6, 2017 at 5:26 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><div class="gmail_extra"><div class="gmail_quote">On Thu, Jul 6, 2017 at 8:38 AM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="m_8155841319646511931gmail-m_5986593071559009587gmail-m_8126099012281831575m_3272918658699115455gmail-"><div class="gmail_extra"><div class="gmail_quote"><br></div></div></span><div class="gmail_extra"><div class="gmail_extra"><div>Eventually I can destroy and recreate this "export" volume again with the old names (ovirt0N.localdomain.local) if you give me the sequence of commands, then enable debug and retry the reset-brick command</div><span class="m_8155841319646511931gmail-m_5986593071559009587gmail-m_8126099012281831575m_3272918658699115455gmail-HOEnZb"><font color="#888888"><div><br></div><div>Gianluca</div></font></span></div></div></div>
</blockquote></div><br></div><div class="gmail_extra"><br></div></span><div class="gmail_extra">So it seems I was able to destroy and re-create.</div><div class="gmail_extra">Now I see that the volume creation uses by default the new ip, so I reverted the hostnames roles in the commands after putting glusterd in debug mode on the host where I execute the reset-brick command (do I have to set debug for the the nodes too?)</div></div></blockquote><div><br></div></span><div>You have to set the log level to debug for glusterd instance where the commit fails and share the glusterd log of that particular node.<br> </div></div></div></div></blockquote></div></div><div class="gmail_extra"><br></div></span><div class="gmail_extra">Ok, done.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Command executed on ovirt01 with timestamp "2017-07-06 13:04:12" in glusterd log files</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluste<wbr>r/brick3/export start</div><span class="m_8155841319646511931gmail-"><div class="gmail_extra">volume reset-brick: success: reset-brick start operation successful</div><div class="gmail_extra"><br></div></span><div class="gmail_extra">[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluste<wbr>r/brick3/export ovirt01.localdomain.local:/glu<wbr>ster/brick3/export commit force</div><span class="m_8155841319646511931gmail-"><div class="gmail_extra">volume reset-brick: failed: Commit failed on ovirt02.localdomain.local. Please check log file for details.</div><div class="gmail_extra">Commit failed on ovirt03.localdomain.local. Please check log file for details.</div></span><div class="gmail_extra">[root@ovirt01 export]#</div><div class="gmail_extra"><br></div><div class="gmail_extra">See glusterd log files for the 3 nodes in debug mode here:</div><div class="gmail_extra">ovirt01: <a href="https://drive.google.com/file/d/0BwoPbcrMv8mvY1RTTGp3RUhScm8/view?usp=sharing" target="_blank">https://drive.google.<wbr>com/file/d/0BwoPbcrMv8mvY1RTTG<wbr>p3RUhScm8/view?usp=sharing</a></div><div class="gmail_extra">ovirt02: <a href="https://drive.google.com/file/d/0BwoPbcrMv8mvSVpJUHNhMzhMSU0/view?usp=sharing" target="_blank">https://drive.google.<wbr>com/file/d/0BwoPbcrMv8mvSVpJUH<wbr>NhMzhMSU0/view?usp=sharing</a></div><div class="gmail_extra">ovirt03: <a href="https://drive.google.com/file/d/0BwoPbcrMv8mvT2xiWEdQVmJNb0U/view?usp=sharing" target="_blank">https://drive.google.<wbr>com/file/d/0BwoPbcrMv8mvT2xiWE<wbr>dQVmJNb0U/view?usp=sharing</a></div><div class="gmail_extra"><br></div><div class="gmail_extra">HIH debugging</div><span class="m_8155841319646511931gmail-HOEnZb"><font color="#888888"><div class="gmail_extra">Gianluca</div></font></span></div><div class="gmail_extra"><br></div></div>
</blockquote></div><br></div></div></div><div class="gmail_extra">Hi Atin,</div><div class="gmail_extra">did you have time to see the logs?</div><div class="gmail_extra">Comparing debug enabled messages with previous ones, I see these added lines on nodes where commit failed after running the commands</div><div class="gmail_extra"><br></div><div class="gmail_extra"><span class=""><div class="gmail_extra">gluster volume reset-brick export gl01.localdomain.local:/<wbr>gluster/brick3/export start</div></span><span class=""><div class="gmail_extra">gluster volume reset-brick export gl01.localdomain.local:/<wbr>gluster/brick3/export ovirt01.localdomain.local:/<wbr>gluster/brick3/export commit force</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div></span><div class="gmail_extra">[2017-07-06 13:04:30.221872] D [MSGID: 0] [glusterd-peer-utils.c:674:gd_<wbr>peerinfo_find_from_hostname] 0-management: Friend ovirt01.localdomain.local found.. state: 3</div><div class="gmail_extra">[2017-07-06 13:04:30.221882] D [MSGID: 0] [glusterd-peer-utils.c:167:<wbr>glusterd_hostname_to_uuid] 0-management: returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.221888] D [MSGID: 0] [glusterd-utils.c:1039:<wbr>glusterd_resolve_brick] 0-management: Returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.221908] D [MSGID: 0] [glusterd-utils.c:998:<wbr>glusterd_brickinfo_new] 0-management: Returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.221915] D [MSGID: 0] [glusterd-utils.c:1195:<wbr>glusterd_brickinfo_new_from_<wbr>brick] 0-management: Returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.222187] D [MSGID: 0] [glusterd-peer-utils.c:167:<wbr>glusterd_hostname_to_uuid] 0-management: returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.222201] D [MSGID: 0] [glusterd-utils.c:1486:<wbr>glusterd_volume_brickinfo_get] 0-management: Returning -1</div><div class="gmail_extra">[2017-07-06 13:04:30.222207] D [MSGID: 0] [store.c:459:gf_store_handle_<wbr>destroy] 0-: Returning 0</div><div class="gmail_extra">[2017-07-06 13:04:30.222242] D [MSGID: 0] [glusterd-utils.c:1512:<wbr>glusterd_volume_brickinfo_get_<wbr>by_brick] 0-glusterd: Returning -1</div><div class="gmail_extra">[2017-07-06 13:04:30.222250] D [MSGID: 0] [glusterd-replace-brick.c:416:<wbr>glusterd_op_perform_replace_<wbr>brick] 0-glusterd: Returning -1</div><div class="gmail_extra">[2017-07-06 13:04:30.222257] C [MSGID: 106074] [glusterd-reset-brick.c:372:<wbr>glusterd_op_reset_brick] 0-management: Unable to add dst-brick: ovirt01.localdomain.local:/<wbr>gluster/brick3/export to volume: export</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Does it share up more light?</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra">Gianluca</div></div></div>
</blockquote></div><br></div>