12:17 p.m.
Team,
I've been having issues all my VMs cant be started, error is "VM Has been
paused due to storage I/O error"
Any ideas are welcome as all my VMs are down
Gluster log (gluster version 6.8):
[2020-05-27 09:10:28.132619] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204,
fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.132694] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3286: READV 3
(8674ab5f-56b9-4136-9b30-a65ca86be204), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:10:28.211930] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204,
fd=0x7f12b825ed18, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.211995] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3306: READV 2
(8674ab5f-56b9-4136-9b30-a65ca86be204), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:10:28.226451] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1,
fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.226511] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3317: READV 2
(755664db-2d04-4ed0-9333-251c6cc3dcb1), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:10:28.232122] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1,
fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.232181] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3319: READV 2
(755664db-2d04-4ed0-9333-251c6cc3dcb1), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:10:28.237043] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204,
fd=0x7f12b82038b8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.237100] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3321: READV 3
(8674ab5f-56b9-4136-9b30-a65ca86be204), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:10:28.242176] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv]
0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1,
fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument]
[2020-05-27 09:10:28.242235] E [MSGID: 115068]
[server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3323: READV 2
(755664db-2d04-4ed0-9333-251c6cc3dcb1), client:
CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0,
error-xlator: vmstore-posix [Invalid argument]
[2020-05-27 09:11:18.990877] I [MSGID: 115036] [server.c:499:server_rpc_notify]
0-vmstore-server: disconnecting connection from
CTX_ID:87270d80-4310-4795-9f4d-2c1a61d16cee-GRAPH_ID:0-PID:28815-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
[2020-05-27 09:11:18.991896] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-vmstore-server: Shutting down connection
CTX_ID:87270d80-4310-4795-9f4d-2c1a61d16cee-GRAPH_ID:0-PID:28815-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
[2020-05-27 09:11:19.329120] I [MSGID: 115036] [server.c:499:server_rpc_notify]
0-vmstore-server: disconnecting connection from
CTX_ID:5858e269-923e-4a38-9c6b-62f337a6abac-GRAPH_ID:0-PID:20791-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
[2020-05-27 09:11:19.329419] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-vmstore-server: Shutting down connection
CTX_ID:5858e269-923e-4a38-9c6b-62f337a6abac-GRAPH_ID:0-PID:20791-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
[2020-05-27 09:11:21.044186] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/vmstore/vmstore: allowed = "*", received addr =
"192.168.0.59"
[2020-05-27 09:11:21.044227] I [login.c:110:gf_auth] 0-auth/login: allowed user names:
c2ef048a-b354-4f91-8e41-a2aab6e65dbc
[2020-05-27 09:11:21.044265] I [MSGID: 115029] [server-handshake.c:553:server_setvolume]
0-vmstore-server: accepted client from
CTX_ID:c2426c68-ad6c-4d3a-b6a4-decce8d18b75-GRAPH_ID:0-PID:1558-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
(version: 6.8) with subvol /gluster_bricks/vmstore/vmstore
[2020-05-27 09:11:21.383131] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/vmstore/vmstore: allowed = "*", received addr =
"192.168.0.60"
[2020-05-27 09:11:21.383168] I [login.c:110:gf_auth] 0-auth/login: allowed user names:
c2ef048a-b354-4f91-8e41-a2aab6e65dbc
[2020-05-27 09:11:21.383190] I [MSGID: 115029] [server-handshake.c:553:server_setvolume]
0-vmstore-server: accepted client from
CTX_ID:5a363377-0bc7-40eb-a7cb-f4f5757a102b-GRAPH_ID:0-PID:27173-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0
(version: 6.8) with subvol /gluster_bricks/vmstore/vmstore