11:53 a.m.
On Thu, Jul 6, 2017 at 3:22 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Thu, Jul 6, 2017 at 2:16 PM, Atin Mukherjee
<amukherj(a)redhat.com>
wrote:
>
>
> On Thu, Jul 6, 2017 at 5:26 PM, Gianluca Cecchi <
> gianluca.cecchi(a)gmail.com> wrote:
>
>> On Thu, Jul 6, 2017 at 8:38 AM, Gianluca Cecchi <
>> gianluca.cecchi(a)gmail.com> wrote:
>>
>>>
>>> Eventually I can destroy and recreate this "export" volume again
with
>>> the old names (ovirt0N.localdomain.local) if you give me the sequence of
>>> commands, then enable debug and retry the reset-brick command
>>>
>>> Gianluca
>>>
>>
>>
>> So it seems I was able to destroy and re-create.
>> Now I see that the volume creation uses by default the new ip, so I
>> reverted the hostnames roles in the commands after putting glusterd in
>> debug mode on the host where I execute the reset-brick command (do I have
>> to set debug for the the nodes too?)
>>
>
> You have to set the log level to debug for glusterd instance where the
> commit fails and share the glusterd log of that particular node.
>
>
Ok, done.
Command executed on ovirt01 with timestamp "2017-07-06 13:04:12" in
glusterd log files
[root@ovirt01 export]# gluster volume reset-brick export
gl01.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful
[root@ovirt01 export]# gluster volume reset-brick export
gl01.localdomain.local:/gluster/brick3/export
ovirt01.localdomain.local:/gluster/brick3/export
commit force
volume reset-brick: failed: Commit failed on ovirt02.localdomain.local.
Please check log file for details.
Commit failed on ovirt03.localdomain.local. Please check log file for
details.
[root@ovirt01 export]#
See glusterd log files for the 3 nodes in debug mode here:
ovirt01: https://drive.google.com/file/d/0BwoPbcrMv8mvY1RTTGp3RUhScm8/
view?usp=sharing
ovirt02: https://drive.google.com/file/d/0BwoPbcrMv8mvSVpJUHNhMzhMSU0/
view?usp=sharing
ovirt03: https://drive.google.com/file/d/0BwoPbcrMv8mvT2xiWEdQVmJNb0U/
view?usp=sharing
HIH debugging
Gianluca
Hi Atin,
did you have time to see the logs?
Comparing debug enabled messages with previous ones, I see these added
lines on nodes where commit failed after running the commands
gluster volume reset-brick export
gl01.localdomain.local:/gluster/brick3/export start
gluster volume reset-brick export
gl01.localdomain.local:/gluster/brick3/export
ovirt01.localdomain.local:/gluster/brick3/export commit force
[2017-07-06 13:04:30.221872] D [MSGID: 0]
[glusterd-peer-utils.c:674:gd_peerinfo_find_from_hostname] 0-management:
Friend ovirt01.localdomain.local found.. state: 3
[2017-07-06 13:04:30.221882] D [MSGID: 0]
[glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management:
returning 0
[2017-07-06 13:04:30.221888] D [MSGID: 0]
[glusterd-utils.c:1039:glusterd_resolve_brick] 0-management: Returning 0
[2017-07-06 13:04:30.221908] D [MSGID: 0]
[glusterd-utils.c:998:glusterd_brickinfo_new] 0-management: Returning 0
[2017-07-06 13:04:30.221915] D [MSGID: 0]
[glusterd-utils.c:1195:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2017-07-06 13:04:30.222187] D [MSGID: 0]
[glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management:
returning 0
[2017-07-06 13:04:30.222201] D [MSGID: 0]
[glusterd-utils.c:1486:glusterd_volume_brickinfo_get] 0-management:
Returning -1
[2017-07-06 13:04:30.222207] D [MSGID: 0]
[store.c:459:gf_store_handle_destroy] 0-: Returning 0
[2017-07-06 13:04:30.222242] D [MSGID: 0]
[glusterd-utils.c:1512:glusterd_volume_brickinfo_get_by_brick] 0-glusterd:
Returning -1
[2017-07-06 13:04:30.222250] D [MSGID: 0]
[glusterd-replace-brick.c:416:glusterd_op_perform_replace_brick]
0-glusterd: Returning -1
[2017-07-06 13:04:30.222257] C [MSGID: 106074]
[glusterd-reset-brick.c:372:glusterd_op_reset_brick] 0-management: Unable
to add dst-brick: ovirt01.localdomain.local:/gluster/brick3/export to
volume: export
Does it share up more light?
Thanks,
Gianluca