Hi Jean-Louis, Sorry for the delay. I was waiting for it to occur again. While waiting for it to occur again I've updated it to the latest oVirt 4.5.5 snapshot sometime late last week.
Do you have discard enabled on the disks?
No. Discard is not enabled. This is a thin provisioned disk, however. Here is what I found: Event log: Oct 23, 2023, 1:00:00 PM VM win-guest-vm1 has been paused due to unknown storage error. Virsh: ~]$ sudo /usr/bin/virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domblkerror win-guest-vm1 sda: unspecified error Log files: File: vdsm.log 2023-10-23 13:00:00,787-0400 INFO (libvirt/events) [virt.vm] (vmId='9bc1b5cd-7a28-463a-bb47-e315d16aa02b') abnormal vm stop device ua-bf610b0d-fffc-4569-bf31-9ab76d95e5c0 error (vm:5192) 2023-10-23 13:00:00,787-0400 INFO (libvirt/events) [virt.vm] (vmId='9bc1b5cd-7a28-463a-bb47-e315d16aa02b') CPU stopped: onIOError (vm:6073) 2023-10-23 13:00:00,790-0400 INFO (libvirt/events) [virt.vm] (vmId='9bc1b5cd-7a28-463a-bb47-e315d16aa02b') CPU stopped: onSuspend (vm:6073) 2023-10-23 13:00:00,807-0400 WARN (libvirt/events) [virt.vm] (vmId='9bc1b5cd-7a28-463a-bb47-e315d16aa02b') device sda reported I/O error (vm:3990) ... snip ... 2023-10-23 14:02:04,886-0400 WARN (jsonrpc/2) [virt.vm] (vmId='9bc1b5cd-7a28-463a-bb47-e315d16aa02b') monitor became unresponsive (command timeout, age=3731.5355101790046) (vm:6086) File mom.log 2023-10-23 13:00:01,929 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:01,929 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:07,645 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:07,646 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:09,092 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:09,093 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:11,165 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:11,165 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:12,367 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:12,368 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:14,785 - mom.Collectors.GuestMemory - WARNING - getVmMemoryStats(): The ovirt-guest-agent is not active 2023-10-23 13:00:19,349 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:19,365 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:21,372 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:21,373 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:22,763 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:22,763 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:27,380 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:27,381 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:29,597 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:29,598 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:29,798 - mom.GuestMonitor.Thread - INFO - GuestMonitor-win-guest-vm1 ending 2023-10-23 13:00:31,546 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:31,547 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:37,886 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:37,887 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:39,808 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:39,808 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:41,718 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:41,719 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:42,379 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:42,379 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:49,996 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:49,997 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:51,898 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:51,899 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:53,015 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:53,016 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:57,390 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:57,391 - mom.RPCServer - INFO - getStatistics() 2023-10-23 13:00:59,244 - mom.RPCServer - INFO - ping() 2023-10-23 13:00:59,245 - mom.RPCServer - INFO - getStatistics() File: rhev-data-center-mnt-glusterSD-hypervisor-0.gluster.stor.prd.lab:_vmstore.log [2023-10-23 16:53:02.685763 +0000] I [MSGID: 133022] [shard.c:3838:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=c3b441d4-e635-4791-9f19-029e2314c4f5 from backend [2023-10-23 16:58:02.985859 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-4: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 16:58:02.985889 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-3: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 16:58:02.985905 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-5: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 16:58:03.986544 +0000] W [fuse-bridge.c:3067:fuse_writev_cbk] 0-glusterfs-fuse: 38146917: WRITE => -1 gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed fd=0x56486713f738 (Invalid argume nt) [2023-10-23 16:58:03.994009 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-0: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.994035 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-1: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.994067 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-3: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.994067 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-2: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.994092 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-4: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.994102 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-5: remote operation failed. [{path=<gfid:b1fc43e0-2348-42ed-a7e7-cf832 bc4f2ed>}, {gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed}, {errno=2}, {error=No such file or directory}] [2023-10-23 16:58:03.996116 +0000] I [MSGID: 133022] [shard.c:3838:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=b1fc43e0-2348-42ed-a7e7-cf832bc4f2ed from backend [2023-10-23 17:00:00.781887 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2683:client4_0_readv_cbk] 0-vmstore-client-5: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.782585 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-5: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.782602 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-3: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.782641 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-4: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784639 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-0: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784640 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-1: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784667 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-2: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784694 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-4: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784704 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-3: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784765 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-5: remote operation failed. [{path=/.shard/6990a191-bd6c-4078-876c-77cdf853bc89.263}, {gfid=2a686544-d463-4c8d-adfc-540fd26d5889}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784801 +0000] E [MSGID: 109038] [dht-helper.c:1367:dht_migration_complete_check_task] 0-vmstore-dht: failed to lookup the file [{tmp=2a686544-d463-4c8d-adfc-540fd26d5889}, {name=vmstore-dht}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784838 +0000] W [fuse-bridge.c:2979:fuse_readv_cbk] 0-glusterfs-fuse: 38149792: READ => -1 gfid=6990a191-bd6c-4078-876c-77cdf853bc89 fd=0x5648663a7698 (No such file or directory) [2023-10-23 17:00:00.784062 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-4: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784145 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-3: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:00:00.784340 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:1277:client4_0_fstat_cbk] 0-vmstore-client-5: remote operation failed. [{errno=2}, {error=No such file or directory}] [2023-10-23 17:03:04.290996 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-4: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 17:03:04.291057 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-5: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 17:03:04.291076 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-vmstore-client-3: remote operation failed. [{errno=22}, {error=Invalid argument}] [2023-10-23 17:03:05.291822 +0000] W [fuse-bridge.c:3067:fuse_writev_cbk] 0-glusterfs-fuse: 38167016: WRITE => -1 gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210 fd=0x564867504798 (Invalid argument) [2023-10-23 17:03:05.298181 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-2: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.298204 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-0: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.298207 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-3: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.298216 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-1: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.298223 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-4: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.298241 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-vmstore-client-5: remote operation failed. [{path=<gfid:8810a5ea-6bd6-4da8-8901-abee03e8a210>}, {gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210}, {errno=2}, {error=No such file or directory}] [2023-10-23 17:03:05.299577 +0000] I [MSGID: 133022] [shard.c:3838:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=8810a5ea-6bd6-4da8-8901-abee03e8a210 from backend