On Thu, Jul 6, 2017 at 3:22 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Thu, Jul 6, 2017 at 2:16 PM, Atin Mukherjee <amukherj@redhat.com> wrote:

On Thu, Jul 6, 2017 at 5:26 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Jul 6, 2017 at 8:38 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

Eventually I can destroy and recreate this "export" volume again with the old names (ovirt0N.localdomain.local) if you give me the sequence of commands, then enable debug and retry the reset-brick command

Gianluca

So it seems I was able to destroy and re-create.
Now I see that the volume creation uses by default the new ip, so I reverted the hostnames roles in the commands after putting glusterd in debug mode on the host where I execute the reset-brick command (do I have to set debug for the the nodes too?)

You have to set the log level to debug for glusterd instance where the commit fails and share the glusterd log of that particular node.

Ok, done.

Command executed on ovirt01 with timestamp "2017-07-06 13:04:12" in glusterd log files

[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful

[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export ovirt01.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on ovirt02.localdomain.local. Please check log file for details.
Commit failed on ovirt03.localdomain.local. Please check log file for details.
[root@ovirt01 export]#

See glusterd log files for the 3 nodes in debug mode here:
ovirt01: https://drive.google.com/file/d/0BwoPbcrMv8mvY1RTTGp3RUhScm8/view?usp=sharing
ovirt02: https://drive.google.com/file/d/0BwoPbcrMv8mvSVpJUHNhMzhMSU0/view?usp=sharing
ovirt03: https://drive.google.com/file/d/0BwoPbcrMv8mvT2xiWEdQVmJNb0U/view?usp=sharing

HIH debugging
Gianluca

Hi Atin,

did you have time to see the logs?

Comparing debug enabled messages with previous ones, I see these added lines on nodes where commit failed after running the commands

gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export start

gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export ovirt01.localdomain.local:/gluster/brick3/export commit force

[2017-07-06 13:04:30.221872] D [MSGID: 0] [glusterd-peer-utils.c:674:gd_peerinfo_find_from_hostname] 0-management: Friend ovirt01.localdomain.local found.. state: 3

[2017-07-06 13:04:30.221882] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0

[2017-07-06 13:04:30.221888] D [MSGID: 0] [glusterd-utils.c:1039:glusterd_resolve_brick] 0-management: Returning 0

[2017-07-06 13:04:30.221908] D [MSGID: 0] [glusterd-utils.c:998:glusterd_brickinfo_new] 0-management: Returning 0

[2017-07-06 13:04:30.221915] D [MSGID: 0] [glusterd-utils.c:1195:glusterd_brickinfo_new_from_brick] 0-management: Returning 0

[2017-07-06 13:04:30.222187] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0

[2017-07-06 13:04:30.222201] D [MSGID: 0] [glusterd-utils.c:1486:glusterd_volume_brickinfo_get] 0-management: Returning -1

[2017-07-06 13:04:30.222207] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0

[2017-07-06 13:04:30.222242] D [MSGID: 0] [glusterd-utils.c:1512:glusterd_volume_brickinfo_get_by_brick] 0-glusterd: Returning -1

[2017-07-06 13:04:30.222250] D [MSGID: 0] [glusterd-replace-brick.c:416:glusterd_op_perform_replace_brick] 0-glusterd: Returning -1

[2017-07-06 13:04:30.222257] C [MSGID: 106074] [glusterd-reset-brick.c:372:glusterd_op_reset_brick] 0-management: Unable to add dst-brick: ovirt01.localdomain.local:/gluster/brick3/export to volume: export

Does it share up more light?

Thanks,

Gianluca