Team, I've been having issues all my VMs cant be started, error is "VM Has been paused due to storage I/O error" Any ideas are welcome as all my VMs are down Gluster log (gluster version 6.8): [2020-05-27 09:10:28.132619] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204, fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.132694] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3286: READV 3 (8674ab5f-56b9-4136-9b30-a65ca86be204), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:10:28.211930] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204, fd=0x7f12b825ed18, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.211995] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3306: READV 2 (8674ab5f-56b9-4136-9b30-a65ca86be204), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:10:28.226451] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1, fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.226511] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3317: READV 2 (755664db-2d04-4ed0-9333-251c6cc3dcb1), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:10:28.232122] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1, fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.232181] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3319: READV 2 (755664db-2d04-4ed0-9333-251c6cc3dcb1), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:10:28.237043] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=8674ab5f-56b9-4136-9b30-a65ca86be204, fd=0x7f12b82038b8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.237100] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3321: READV 3 (8674ab5f-56b9-4136-9b30-a65ca86be204), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:10:28.242176] E [MSGID: 113040] [posix-inode-fd-ops.c:1572:posix_readv] 0-vmstore-posix: read failed on gfid=755664db-2d04-4ed0-9333-251c6cc3dcb1, fd=0x7f12b807e6d8, offset=0 size=1, buf=0x7f134fbd9000 [Invalid argument] [2020-05-27 09:10:28.242235] E [MSGID: 115068] [server-rpc-fops_v2.c:1425:server4_readv_cbk] 0-vmstore-server: 3323: READV 2 (755664db-2d04-4ed0-9333-251c6cc3dcb1), client: CTX_ID:03354525-5de3-4390-b775-3db7a85c0022-GRAPH_ID:0-PID:29372-HOST:jrz-061-ovirt3.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument] [2020-05-27 09:11:18.990877] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-vmstore-server: disconnecting connection from CTX_ID:87270d80-4310-4795-9f4d-2c1a61d16cee-GRAPH_ID:0-PID:28815-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 [2020-05-27 09:11:18.991896] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-vmstore-server: Shutting down connection CTX_ID:87270d80-4310-4795-9f4d-2c1a61d16cee-GRAPH_ID:0-PID:28815-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 [2020-05-27 09:11:19.329120] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-vmstore-server: disconnecting connection from CTX_ID:5858e269-923e-4a38-9c6b-62f337a6abac-GRAPH_ID:0-PID:20791-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 [2020-05-27 09:11:19.329419] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-vmstore-server: Shutting down connection CTX_ID:5858e269-923e-4a38-9c6b-62f337a6abac-GRAPH_ID:0-PID:20791-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 [2020-05-27 09:11:21.044186] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/vmstore/vmstore: allowed = "*", received addr = "192.168.0.59" [2020-05-27 09:11:21.044227] I [login.c:110:gf_auth] 0-auth/login: allowed user names: c2ef048a-b354-4f91-8e41-a2aab6e65dbc [2020-05-27 09:11:21.044265] I [MSGID: 115029] [server-handshake.c:553:server_setvolume] 0-vmstore-server: accepted client from CTX_ID:c2426c68-ad6c-4d3a-b6a4-decce8d18b75-GRAPH_ID:0-PID:1558-HOST:jrz-059-ovirt1.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 (version: 6.8) with subvol /gluster_bricks/vmstore/vmstore [2020-05-27 09:11:21.383131] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/vmstore/vmstore: allowed = "*", received addr = "192.168.0.60" [2020-05-27 09:11:21.383168] I [login.c:110:gf_auth] 0-auth/login: allowed user names: c2ef048a-b354-4f91-8e41-a2aab6e65dbc [2020-05-27 09:11:21.383190] I [MSGID: 115029] [server-handshake.c:553:server_setvolume] 0-vmstore-server: accepted client from CTX_ID:5a363377-0bc7-40eb-a7cb-f4f5757a102b-GRAPH_ID:0-PID:27173-HOST:jrz-060-ovirt2.example.com-PC_NAME:vmstore-client-2-RECON_NO:-0 (version: 6.8) with subvol /gluster_bricks/vmstore/vmstore