unable to bring up gluster bricks after 4.5 upgrade
Hello All, I've been struggling with a few issues upgrading my 3-node HCI custer from 4.4 to 4.5. At present the self hosted engine VM is properly running oVirt 4.5 on CentOS 8x stream. First host node, I set in maintenance and installed new node-ng image. I ran into issue with rescue mode on boot which appears to have been related to LVM devices bug. I was able to work past that and get the node to boot. The node running 4.5.2 image is booting properly and gluster/lvm mounts etc all look good. I am able to activate the host and run VMs on it etc. however, oVirt cli is showing that all bricks on host are DOWN. I was unable to get the bricks back up even after doing a force start of the volumes. Here is the glusterd log from the host in question when I try force start on the engine volume (other volumes are similar: ==> glusterd.log <== The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: bitd service is stopped" repeated 2 times between [2022-08-29 18:09:56.027147 +0000] and [2022-08-29 18:10:34.694144 +0000] [2022-08-29 18:10:34.695348 +0000] I [MSGID: 106618] [glusterd-svc-helper.c:909:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=engine) to existing process with pid 2473 [2022-08-29 18:10:34.695669 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped [2022-08-29 18:10:34.695691 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: scrub service is stopped [2022-08-29 18:10:34.695832 +0000] I [MSGID: 106617] [glusterd-svc-helper.c:698:glusterd_svc_attach_cbk] 0-management: svc glustershd of volume engine attached successfully to pid 2473 [2022-08-29 18:10:34.703718 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster2.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703774 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster1.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703797 +0000] E [MSGID: 106664] [glusterd-mgmt.c:1969:glusterd_mgmt_v3_post_commit] 0-management: Post commit failed on peers [2022-08-29 18:10:34.703800 +0000] E [MSGID: 106664] [glusterd-mgmt.c:2664:glusterd_mgmt_v3_initiate_all_phases] 0-management: Post commit Op Failed If I run start command manually on host cli: gluster volume start engine force volume start: engine: failed: Post commit failed on gluster1.xxxx. Please check log file for details. Post commit failed on gluster2.xxxx. Please check log file for details. I feel like this may be some issue with the difference in major versions of GlusterFS on the nodes but I am unsure. The other nodes are running ovirt-node-ng-4.4.6.3 At this point I am afraid to bring down any other node to attempt upgrading it without the bricks in UP status on the first host. I do not want to lose quorum and potentially disrupt running VMs. Any idea why I can't seem to start the volumes on the upgraded host? Thanks!
A bit more info from the host's brick log [2022-08-29 18:43:44.251198 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-barrier: reconfigured [2022-08-29 18:43:44.251203 +0000] D [MSGID: 0] [options.c:1133:xlator_reconfigure_rec] 0-engine-index: No reconfigure() found [2022-08-29 18:43:44.251207 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-index: reconfigured [2022-08-29 18:43:44.251214 +0000] I [MSGID: 0] [options.c:1251:xlator_option_reconf_bool] 0-engine-quota: option deem-statfs using set value off [2022-08-29 18:43:44.251221 +0000] I [MSGID: 0] [options.c:1251:xlator_option_reconf_bool] 0-engine-quota: option server-quota using set value off [2022-08-29 18:43:44.251248 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-quota: reconfigured [2022-08-29 18:44:04.899452 +0000] E [MSGID: 113072] [posix-inode-fd-ops.c:2087:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2022-08-29 18:44:04.899542 +0000] E [MSGID: 115067] [server-rpc-fops_v2.c:1324:server4_writev_cbk] 0-engine-server: WRITE info [{frame=358765}, {WRITEV_fd_no=5}, {uuid_utoa=c816cdf3-12e6-45c0-ae0f-2cf03e0f7299}, {client=CTX_ID:11b78775-07c9-47ff-b426-b44f3f88a3f7-GRAPH_ID:0-PID:25622-HOST:host1.xxxxx-PC_NAME:engine-client-0-RECON_NO:-2}, {error-xlator=engine-posix}, {errno=22}, {error=Invalid argument}] [2022-08-29 18:44:14.876436 +0000] E [MSGID: 113002] [posix-entry-ops.c:769:posix_mkdir] 0-engine-posix: gfid is null for (null) [Invalid argument] [2022-08-29 18:44:14.876503 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-engine-server: MKDIR info [{frame=359508}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:37199949-8cf2-4bbe-938e-e9ef3bd98486-GRAPH_ID:3-PID:2473-HOST:host0.xxxxx-PC_NAME:engine-client-0-RECON_NO:-0}, {error-xlator=engine-posix}, {errno=22}, {error=Invalid argument}] On Mon, Aug 29, 2022 at 3:18 PM Jayme <jaymef@gmail.com> wrote:
Hello All,
I've been struggling with a few issues upgrading my 3-node HCI custer from 4.4 to 4.5.
At present the self hosted engine VM is properly running oVirt 4.5 on CentOS 8x stream.
First host node, I set in maintenance and installed new node-ng image. I ran into issue with rescue mode on boot which appears to have been related to LVM devices bug. I was able to work past that and get the node to boot.
The node running 4.5.2 image is booting properly and gluster/lvm mounts etc all look good. I am able to activate the host and run VMs on it etc. however, oVirt cli is showing that all bricks on host are DOWN.
I was unable to get the bricks back up even after doing a force start of the volumes.
Here is the glusterd log from the host in question when I try force start on the engine volume (other volumes are similar:
==> glusterd.log <== The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: bitd service is stopped" repeated 2 times between [2022-08-29 18:09:56.027147 +0000] and [2022-08-29 18:10:34.694144 +0000] [2022-08-29 18:10:34.695348 +0000] I [MSGID: 106618] [glusterd-svc-helper.c:909:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=engine) to existing process with pid 2473 [2022-08-29 18:10:34.695669 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped [2022-08-29 18:10:34.695691 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: scrub service is stopped [2022-08-29 18:10:34.695832 +0000] I [MSGID: 106617] [glusterd-svc-helper.c:698:glusterd_svc_attach_cbk] 0-management: svc glustershd of volume engine attached successfully to pid 2473 [2022-08-29 18:10:34.703718 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster2.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703774 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster1.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703797 +0000] E [MSGID: 106664] [glusterd-mgmt.c:1969:glusterd_mgmt_v3_post_commit] 0-management: Post commit failed on peers [2022-08-29 18:10:34.703800 +0000] E [MSGID: 106664] [glusterd-mgmt.c:2664:glusterd_mgmt_v3_initiate_all_phases] 0-management: Post commit Op Failed
If I run start command manually on host cli:
gluster volume start engine force volume start: engine: failed: Post commit failed on gluster1.xxxx. Please check log file for details. Post commit failed on gluster2.xxxx. Please check log file for details.
I feel like this may be some issue with the difference in major versions of GlusterFS on the nodes but I am unsure. The other nodes are running ovirt-node-ng-4.4.6.3
At this point I am afraid to bring down any other node to attempt upgrading it without the bricks in UP status on the first host. I do not want to lose quorum and potentially disrupt running VMs.
Any idea why I can't seem to start the volumes on the upgraded host?
Thanks!
It appears that I may have resolved the issue after putting host into maintenance again and rebooting a second time. I'm really not sure why but all bricks are up now On Mon, Aug 29, 2022 at 3:45 PM Jayme <jaymef@gmail.com> wrote:
A bit more info from the host's brick log
[2022-08-29 18:43:44.251198 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-barrier: reconfigured [2022-08-29 18:43:44.251203 +0000] D [MSGID: 0] [options.c:1133:xlator_reconfigure_rec] 0-engine-index: No reconfigure() found [2022-08-29 18:43:44.251207 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-index: reconfigured [2022-08-29 18:43:44.251214 +0000] I [MSGID: 0] [options.c:1251:xlator_option_reconf_bool] 0-engine-quota: option deem-statfs using set value off [2022-08-29 18:43:44.251221 +0000] I [MSGID: 0] [options.c:1251:xlator_option_reconf_bool] 0-engine-quota: option server-quota using set value off [2022-08-29 18:43:44.251248 +0000] D [MSGID: 0] [options.c:1113:xlator_reconfigure_rec] 0-engine-quota: reconfigured [2022-08-29 18:44:04.899452 +0000] E [MSGID: 113072] [posix-inode-fd-ops.c:2087:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2022-08-29 18:44:04.899542 +0000] E [MSGID: 115067] [server-rpc-fops_v2.c:1324:server4_writev_cbk] 0-engine-server: WRITE info [{frame=358765}, {WRITEV_fd_no=5}, {uuid_utoa=c816cdf3-12e6-45c0-ae0f-2cf03e0f7299}, {client=CTX_ID:11b78775-07c9-47ff-b426-b44f3f88a3f7-GRAPH_ID:0-PID:25622-HOST:host1.xxxxx-PC_NAME:engine-client-0-RECON_NO:-2}, {error-xlator=engine-posix}, {errno=22}, {error=Invalid argument}] [2022-08-29 18:44:14.876436 +0000] E [MSGID: 113002] [posix-entry-ops.c:769:posix_mkdir] 0-engine-posix: gfid is null for (null) [Invalid argument] [2022-08-29 18:44:14.876503 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-engine-server: MKDIR info [{frame=359508}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:37199949-8cf2-4bbe-938e-e9ef3bd98486-GRAPH_ID:3-PID:2473-HOST:host0.xxxxx-PC_NAME:engine-client-0-RECON_NO:-0}, {error-xlator=engine-posix}, {errno=22}, {error=Invalid argument}]
On Mon, Aug 29, 2022 at 3:18 PM Jayme <jaymef@gmail.com> wrote:
Hello All,
I've been struggling with a few issues upgrading my 3-node HCI custer from 4.4 to 4.5.
At present the self hosted engine VM is properly running oVirt 4.5 on CentOS 8x stream.
First host node, I set in maintenance and installed new node-ng image. I ran into issue with rescue mode on boot which appears to have been related to LVM devices bug. I was able to work past that and get the node to boot.
The node running 4.5.2 image is booting properly and gluster/lvm mounts etc all look good. I am able to activate the host and run VMs on it etc. however, oVirt cli is showing that all bricks on host are DOWN.
I was unable to get the bricks back up even after doing a force start of the volumes.
Here is the glusterd log from the host in question when I try force start on the engine volume (other volumes are similar:
==> glusterd.log <== The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: bitd service is stopped" repeated 2 times between [2022-08-29 18:09:56.027147 +0000] and [2022-08-29 18:10:34.694144 +0000] [2022-08-29 18:10:34.695348 +0000] I [MSGID: 106618] [glusterd-svc-helper.c:909:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=engine) to existing process with pid 2473 [2022-08-29 18:10:34.695669 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped [2022-08-29 18:10:34.695691 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: scrub service is stopped [2022-08-29 18:10:34.695832 +0000] I [MSGID: 106617] [glusterd-svc-helper.c:698:glusterd_svc_attach_cbk] 0-management: svc glustershd of volume engine attached successfully to pid 2473 [2022-08-29 18:10:34.703718 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster2.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703774 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster1.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703797 +0000] E [MSGID: 106664] [glusterd-mgmt.c:1969:glusterd_mgmt_v3_post_commit] 0-management: Post commit failed on peers [2022-08-29 18:10:34.703800 +0000] E [MSGID: 106664] [glusterd-mgmt.c:2664:glusterd_mgmt_v3_initiate_all_phases] 0-management: Post commit Op Failed
If I run start command manually on host cli:
gluster volume start engine force volume start: engine: failed: Post commit failed on gluster1.xxxx. Please check log file for details. Post commit failed on gluster2.xxxx. Please check log file for details.
I feel like this may be some issue with the difference in major versions of GlusterFS on the nodes but I am unsure. The other nodes are running ovirt-node-ng-4.4.6.3
At this point I am afraid to bring down any other node to attempt upgrading it without the bricks in UP status on the first host. I do not want to lose quorum and potentially disrupt running VMs.
Any idea why I can't seem to start the volumes on the upgraded host?
Thanks!
participants (1)
-
Jayme