On Tue, Apr 7, 2020 at 7:36 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

OK. So I set log at least at INFO level on all subsystems and tried a redeploy of Openshift with 3 mater nodes and 7 worker nodes.
One worker got the error and VM in paused mode

Apr 7, 2020, 3:27:28 PM VM worker-6 has been paused due to unknown storage error.

The vm has only one 100Gb virtual disk on gluster volume named vmstore


Here below all the logs around time at the different layers.
Let me know if you need another log file not yet considered.

From what I see, the matching error is found in

- rhev-data-center-mnt-glusterSD-ovirtst.mydomain.storage:_vmstore.log

[2020-04-07 13:27:28.721262] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-vmstore-shard: Lookup on shard 523 failed. Base file gfid = d22530cf-2e50-4059-8924-0aafe38497b1 [No such file or directory]
[2020-04-07 13:27:28.721432] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4435189: WRITE => -1 gfid=d22530cf-2e50-4059-8924-0aafe38497b1 fd=0x7f3c4c07ab38 (No such file or directory)


This ^^, right here is the reason the VM paused. Are you using a plain distribute volume here?
Can you share some of the log messages that occur right above these errors?
Also, can you check if the file $VMSTORE_BRICKPATH/.glusterfs/d2/25/d22530cf-2e50-4059-8924-0aafe38497b1 exists on the brick?

-Krutika

and

- gluster_bricks-vmstore-vmstore.log

[2020-04-07 13:27:28.719391] W [MSGID: 113020] [posix-helpers.c:1051:posix_gfid_set] 0-vmstore-posix: setting GFID on /gluster_bricks/vmstore
/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed  [File exists]
[2020-04-07 13:27:28.719978] E [MSGID: 113020] [posix-entry-ops.c:517:posix_mknod] 0-vmstore-posix: setting gfid on /gluster_bricks/vmstore/v
mstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File exists]


Here below all the files checked.
Any hint?

Gianluca

- qemu.log of the vm

2020-04-07T12:30:29.954084Z qemu-kvm: -drive file=/rhev/data-center/mnt/glusterSD/ovirtst.mydomain.storage:_vmstore/81b97244-4b69-4d49-84c4-c822387adc6a/images/04ec5a8d-4ee6-4661-a832-0c094968aa5b/716fdc98-2d0e-44c3-b5fe-c0187cdad751,format=raw,if=none,id=drive-ua-04ec5a8d-4ee6-4661-a832-0c094968aa5b,serial=04ec5a8d-4ee6-4661-a832-0c094968aa5b,werror=stop,rerror=stop,cache=none,aio=threads: 'serial' is deprecated, please use the corresponding option of '-device' instead
Spice-Message: 14:30:29.963: setting TLS option 'CipherString' to 'kECDHE+FIPS:kDHE+FIPS:kRSA+FIPS:!eNULL:!aNULL' from /etc/pki/tls/spice.cnf configuration file
2020-04-07T12:30:29.976216Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 8 [socket-id: 2, core-id: 0, thread-id: 0], CPU 9 [socket-id: 2, core-id: 1, thread-id: 0], CPU 10 [socket-id: 2, core-id: 2, thread-id: 0], CPU 11 [socket-id: 2, core-id: 3, thread-id: 0], CPU 12 [socket-id: 3, core-id: 0, thread-id: 0], CPU 13 [socket-id: 3, core-id: 1, thread-id: 0], CPU 14 [socket-id: 3, core-id: 2, thread-id: 0], CPU 15 [socket-id: 3, core-id: 3, thread-id: 0], CPU 16 [socket-id: 4, core-id: 0, thread-id: 0], CPU 17 [socket-id: 4, core-id: 1, thread-id: 0], CPU 18 [socket-id: 4, core-id: 2, thread-id: 0], CPU 19 [socket-id: 4, core-id: 3, thread-id: 0], CPU 20 [socket-id: 5, core-id: 0, thread-id: 0], CPU 21 [socket-id: 5, core-id: 1, thread-id: 0], CPU 22 [socket-id: 5, core-id: 2, thread-id: 0], CPU 23 [socket-id: 5, core-id: 3, thread-id: 0], CPU 24 [socket-id: 6, core-id: 0, thread-id: 0], CPU 25 [socket-id: 6, core-id: 1, thread-id: 0], CPU 26 [socket-id: 6, core-id: 2, thread-id: 0], CPU 27 [socket-id: 6, core-id: 3, thread-id: 0], CPU 28 [socket-id: 7, core-id: 0, thread-id: 0], CPU 29 [socket-id: 7, core-id: 1, thread-id: 0], CPU 30 [socket-id: 7, core-id: 2, thread-id: 0], CPU 31 [socket-id: 7, core-id: 3, thread-id: 0], CPU 32 [socket-id: 8, core-id: 0, thread-id: 0], CPU 33 [socket-id: 8, core-id: 1, thread-id: 0], CPU 34 [socket-id: 8, core-id: 2, thread-id: 0], CPU 35 [socket-id: 8, core-id: 3, thread-id: 0], CPU 36 [socket-id: 9, core-id: 0, thread-id: 0], CPU 37 [socket-id: 9, core-id: 1, thread-id: 0], CPU 38 [socket-id: 9, core-id: 2, thread-id: 0], CPU 39 [socket-id: 9, core-id: 3, thread-id: 0], CPU 40 [socket-id: 10, core-id: 0, thread-id: 0], CPU 41 [socket-id: 10, core-id: 1, thread-id: 0], CPU 42 [socket-id: 10, core-id: 2, thread-id: 0], CPU 43 [socket-id: 10, core-id: 3, thread-id: 0], CPU 44 [socket-id: 11, core-id: 0, thread-id: 0], CPU 45 [socket-id: 11, core-id: 1, thread-id: 0], CPU 46 [socket-id: 11, core-id: 2, thread-id: 0], CPU 47 [socket-id: 11, core-id: 3, thread-id: 0], CPU 48 [socket-id: 12, core-id: 0, thread-id: 0], CPU 49 [socket-id: 12, core-id: 1, thread-id: 0], CPU 50 [socket-id: 12, core-id: 2, thread-id: 0], CPU 51 [socket-id: 12, core-id: 3, thread-id: 0], CPU 52 [socket-id: 13, core-id: 0, thread-id: 0], CPU 53 [socket-id: 13, core-id: 1, thread-id: 0], CPU 54 [socket-id: 13, core-id: 2, thread-id: 0], CPU 55 [socket-id: 13, core-id: 3, thread-id: 0], CPU 56 [socket-id: 14, core-id: 0, thread-id: 0], CPU 57 [socket-id: 14, core-id: 1, thread-id: 0], CPU 58 [socket-id: 14, core-id: 2, thread-id: 0], CPU 59 [socket-id: 14, core-id: 3, thread-id: 0], CPU 60 [socket-id: 15, core-id: 0, thread-id: 0], CPU 61 [socket-id: 15, core-id: 1, thread-id: 0], CPU 62 [socket-id: 15, core-id: 2, thread-id: 0], CPU 63 [socket-id: 15, core-id: 3, thread-id: 0]
2020-04-07T12:30:29.976247Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
main_channel_link: add main channel client
main_channel_client_handle_pong: net test: latency 52.540000 ms, bitrate 6113597 bps (5.830380 Mbps) LOW BANDWIDTH
inputs_connect: inputs channel client create
red_qxl_set_cursor_peer:
inputs_channel_detach_tablet:
red_channel_client_disconnect: rcc=0x5626c717a9c0 (channel=0x5626c616a880 type=3 id=0)
red_channel_client_disconnect: rcc=0x5626c8351dd0 (channel=0x5626c7bd38d0 type=4 id=0)
red_channel_client_disconnect: rcc=0x5626c834c5b0 (channel=0x5626c616a230 type=2 id=0)
red_channel_client_disconnect: rcc=0x5626c719e9f0 (channel=0x5626c616a160 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x5626c719e9f0
red_client_destroy: destroy client 0x5626c6195570 with #channels=4
red_qxl_disconnect_cursor_peer:
red_qxl_disconnect_display_peer:
inputs_channel_detach_tablet:
inputs_channel_detach_tablet:
main_channel_link: add main channel client
main_channel_client_handle_pong: net test: latency 126.308000 ms, bitrate 9060022 bps (8.640310 Mbps) LOW BANDWIDTH
red_qxl_set_cursor_peer:
inputs_connect: inputs channel client create
red_channel_client_disconnect: rcc=0x5626c717a9f0 (channel=0x5626c616a160 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x5626c717a9f0
red_client_destroy: destroy client 0x5626c7ab6870 with #channels=4
red_channel_client_disconnect: rcc=0x5626c71529c0 (channel=0x5626c616a880 type=3 id=0)
red_qxl_disconnect_cursor_peer:
red_channel_client_disconnect: rcc=0x5626c7e59dd0 (channel=0x5626c7bd38d0 type=4 id=0)
red_qxl_disconnect_display_peer:
red_channel_client_disconnect: rcc=0x5626c7e5e5b0 (channel=0x5626c616a230 type=2 id=0)
[root@ovirt qemu]#


- /var/log/glusterfs/cli.log

[2020-04-07 13:25:39.305818] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:39.306091] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:39.306531] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:39.306758] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:39.307081] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:25:54.365843] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:25:54.448475] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:25:54.448600] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:25:54.449348] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:25:54.485288] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:25:54.566723] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:25:54.566792] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:25:54.567275] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:25:54.567305] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:25:54.603618] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:25:54.685718] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:25:54.685821] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:25:54.686537] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:54.686994] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:54.687289] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:54.687774] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:54.688061] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:25:54.688368] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:09.748782] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:09.831541] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:09.831610] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:09.832596] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:09.871269] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:09.955626] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:09.955726] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:09.956239] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:26:09.956284] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:09.993929] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:10.073634] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:10.073680] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:10.074454] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:10.074932] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:10.075241] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:10.075677] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:10.075920] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:10.076233] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:22.933923] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:23.016382] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:23.016486] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:23.020067] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:25.149347] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:25.229667] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:25.229716] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:25.230491] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:25.266853] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:25.348774] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:25.348815] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:25.349253] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:26:25.349284] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:25.385951] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:25.468521] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:25.468571] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:25.469286] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:25.469703] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:25.469965] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:25.470381] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:25.470595] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:25.470877] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:40.546233] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:40.629567] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:40.629667] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:40.630529] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:40.666769] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:40.746861] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:40.746934] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:40.747436] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:26:40.747474] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:40.789486] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:40.881750] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:40.881871] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:40.882615] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:40.883142] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:40.883532] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:40.884020] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:40.884283] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:40.884615] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:55.966643] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:56.049639] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:56.049735] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:56.050543] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:56.096892] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:56.182275] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:56.182335] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:56.182883] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:26:56.182918] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:26:56.221093] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:26:56.303302] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:26:56.303337] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:26:56.303996] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:56.304432] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:56.304706] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:56.305135] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:56.305359] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:26:56.305645] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:11.378699] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:11.457630] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:11.457697] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:11.458355] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:11.493643] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:11.571358] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:11.571360] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:11.572137] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:27:11.572167] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:11.608432] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:11.689353] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:11.689437] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:11.690117] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:11.690569] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:11.690910] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:11.691343] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:11.691577] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:11.691853] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:23.058796] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:23.141280] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:23.141326] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:23.144418] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:26.767415] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:26.846945] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:26.847030] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:26.847818] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:26.883554] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:26.962804] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:26.962905] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:26.963380] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:27:26.963415] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:27.000197] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:27.077981] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:27.078029] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:27.078729] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:27.079174] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:27.079448] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:27.079873] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:27.080105] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:27.080429] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:42.152150] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:42.233109] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:42.233149] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:42.233818] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:42.269022] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:42.347310] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:42.347347] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:42.347771] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:27:42.347798] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:42.383260] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:42.460650] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:42.460741] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:42.461348] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:42.461780] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:42.462085] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:42.462493] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:42.462711] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:42.462990] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:57.534379] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:57.614068] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:57.614115] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:57.614835] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:57.650436] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:57.729556] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:57.729652] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:57.730164] I [cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk] 0-cli: Received resp to uuid get
[2020-04-07 13:27:57.730204] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:27:57.768743] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:27:57.848062] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:27:57.848153] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:27:57.848830] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:57.849277] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:57.849554] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:57.849983] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:57.850217] I [cli-rpc-ops.c:867:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2020-04-07 13:27:57.850522] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:28:12.934596] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8
[2020-04-07 13:28:13.015126] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-04-07 13:28:13.015167] I [MSGID: 101190] [event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-04-07 13:28:13.015948] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2020-04-07 13:28:13.054894] I [cli.c:844:main] 0-cli: Started running /usr/sbin/gluster with version 6.8


- glusterd.log

[2020-04-07 13:25:22.884457] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vmstore
The message "I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 7 times between [2020-04-07 13:24:06.750320] and [2020-04-07 13:25:54.449004]
The message "I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 39 times between [2020-04-07 13:24:06.989378] and [2020-04-07 13:25:54.688025]
[2020-04-07 13:26:09.832058] I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2020-04-07 13:26:10.074287] I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-04-07 13:26:23.017801] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume big
[2020-04-07 13:26:23.018753] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data
[2020-04-07 13:26:23.019190] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2020-04-07 13:26:23.019628] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vmstore
[2020-04-07 13:27:23.143376] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data
[2020-04-07 13:27:23.143718] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2020-04-07 13:27:23.142636] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume big
[2020-04-07 13:27:23.144070] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vmstore
The message "I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 7 times between [2020-04-07 13:26:09.832058] and [2020-04-07 13:27:57.614518]
The message "I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 39 times between [2020-04-07 13:26:10.074287] and [2020-04-07 13:27:57.850194]
[2020-04-07 13:28:13.015558] I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2020-04-07 13:28:13.256713] I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-04-07 13:28:23.264138] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume big
[2020-04-07 13:28:23.264884] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data
[2020-04-07 13:28:23.265316] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2020-04-07 13:28:23.265644] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vmstore
[2020-04-07 13:29:23.389341] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data
[2020-04-07 13:29:23.389702] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2020-04-07 13:29:23.388761] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume big
[2020-04-07 13:29:23.390058] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vmstore
The message "I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2020-04-07 13:28:13.015558] and [2020-04-07 13:29:45.291552]
The message "I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 34 times between [2020-04-07 13:28:13.256713] and [2020-04-07 13:29:45.524677]
[2020-04-07 13:30:00.664512] I [MSGID: 106487] [glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2020-04-07 13:30:00.899102] I [MSGID: 106488] [glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-04-07 13:30:23.505881] I [MSGID: 106499] [glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status volume req for volume big

- events.log

[2020-04-07 15:21:30,978] INFO [utils - 66:log_event] - {'event': 'CLIENT_CONNECT', 'message': {'client_identifier': '10.50.50.11:49135', 'server_identifier': '10.50.50.11:49153', 'client_uid': 'CTX_ID:815a9519-d88e-402d-9473-19015cca1db8-GRAPH_ID:0-PID:131714-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0', 'brick_path': '/gluster_bricks/engine/engine', 'subdir_mount': '(null)'}, 'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586265690}
[2020-04-07 15:21:30,980] INFO [utils - 66:log_event] - {'event': 'CLIENT_DISCONNECT', 'message': {'client_identifier': '10.50.50.11:49135', 'server_identifier': '10.50.50.11:49153', 'client_uid': 'CTX_ID:815a9519-d88e-402d-9473-19015cca1db8-GRAPH_ID:0-PID:131714-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0', 'brick_path': '/gluster_bricks/engine/engine'}, 'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586265690}
[2020-04-07 15:36:33,598] INFO [utils - 66:log_event] - {'event': 'CLIENT_CONNECT', 'message': {'client_identifier': '10.50.50.11:49144', 'server_identifier': '10.50.50.11:49155', 'client_uid': 'CTX_ID:e1d09577-50d6-4679-b0e8-0fe554f9fd49-GRAPH_ID:0-PID:139160-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0', 'brick_path': '/gluster_bricks/big/big', 'subdir_mount': '(null)'}, 'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586266593}
[2020-04-07 15:36:33,599] INFO [utils - 66:log_event] - {'event': 'CLIENT_DISCONNECT', 'message': {'client_identifier': '10.50.50.11:49144', 'server_identifier': '10.50.50.11:49155', 'client_uid': 'CTX_ID:e1d09577-50d6-4679-b0e8-0fe554f9fd49-GRAPH_ID:0-PID:139160-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0', 'brick_path': '/gluster_bricks/big/big'}, 'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586266593}


- rhev-data-center-mnt-glusterSD-ovirtst.mydomain.storage:_vmstore.log

[2020-04-07 13:24:16.070237] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4197133: WRITE => -1 gfid=aa68cd63-f3d3-47bd-a177-c7f48bef3c59 fd=0x7f3c5019f348 (Invalid argument)
[2020-04-07 13:24:16.074521] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=aa68cd63-f3d3-47bd-a177-c7f48bef3c59 from backend
[2020-04-07 13:27:28.721262] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-vmstore-shard: Lookup on shard 523 failed. Base file gfid = d22530cf-2e50-4059-8924-0aafe38497b1 [No such file or directory]
[2020-04-07 13:27:28.721432] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4435189: WRITE => -1 gfid=d22530cf-2e50-4059-8924-0aafe38497b1 fd=0x7f3c4c07ab38 (No such file or directory)
[2020-04-07 13:29:16.232630] W [MSGID: 114031] [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0: remote operation failed [Invalid argument]
[2020-04-07 13:29:16.232675] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4496167: WRITE => -1 gfid=54bafbb5-edd6-42a0-aaba-a5295031be17 fd=0x7f3c48125838 (Invalid argument)
[2020-04-07 13:29:16.236352] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=54bafbb5-edd6-42a0-aaba-a5295031be17 from backend
[2020-04-07 13:34:16.378326] W [MSGID: 114031] [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0: remote operation failed [Invalid argument]
[2020-04-07 13:34:16.378461] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4698415: WRITE => -1 gfid=6beaa642-e73e-40c7-8adc-64275528b44f fd=0x7f3c48138cc8 (Invalid argument)
[2020-04-07 13:34:16.382824] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=6beaa642-e73e-40c7-8adc-64275528b44f from backend
[2020-04-07 13:39:16.546919] W [MSGID: 114031] [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0: remote operation failed [Invalid argument]
[2020-04-07 13:39:16.547022] W [fuse-bridge.c:2918:fuse_writev_cbk] 0-glusterfs-fuse: 4884227: WRITE => -1 gfid=f32843b7-4131-4536-abe2-f6969aa455ae fd=0x7f3c4813fb58 (Invalid argument)
[2020-04-07 13:39:16.550630] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of gfid=f32843b7-4131-4536-abe2-f6969aa455ae from backend


- gluster_bricks-vmstore-vmstore.log

[2020-04-07 13:21:28.128381] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-vmstore-server: disconnecting connection from CTX_ID:12a5f3
7d-94d3-4b22-aaf6-4ac5bb7567d6-GRAPH_ID:0-PID:131656-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0
[2020-04-07 13:21:28.128565] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-vmstore-server: Shutting down connection CTX_ID:12a5f37d-94
d3-4b22-aaf6-4ac5bb7567d6-GRAPH_ID:0-PID:131656-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0
[2020-04-07 13:24:16.069408] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed: offset 0, [Invalid arg
ument]
[2020-04-07 13:24:16.070018] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-vmstore-server: 3528796: WRITEV 18 (aa68cd63-
f3d3-47bd-a177-c7f48bef3c59), client: CTX_ID:20665547-59fd-4b4a-94fb-7fdfdca9f5a9-GRAPH_ID:0-PID:17102-HOST:ovirt.mydomain.local-PC_NAME:vmsto
re-client-0-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument]
[2020-04-07 13:27:28.719391] W [MSGID: 113020] [posix-helpers.c:1051:posix_gfid_set] 0-vmstore-posix: setting GFID on /gluster_bricks/vmstore
/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed  [File exists]
[2020-04-07 13:27:28.719978] E [MSGID: 113020] [posix-entry-ops.c:517:posix_mknod] 0-vmstore-posix: setting gfid on /gluster_bricks/vmstore/v
mstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File exists]
[2020-04-07 13:29:16.232263] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed: offset 0, [Invalid argument]
[2020-04-07 13:29:16.232578] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-vmstore-server: 3777485: WRITEV 12 (54bafbb5-edd6-42a0-aaba-a5295031be17), client: CTX_ID:20665547-59fd-4b4a-94fb-7fdfdca9f5a9-GRAPH_ID:0-PID:17102-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument]
[2020-04-07 13:34:16.377622] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed: offset 0, [Invalid argument]

- messages

Apr  7 15:25:26 ovirt gluster_bricks-big-big[39632]: [2020-04-07 13:25:26.789943] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev]
0-big-posix: write failed: offset 0, [Invalid argument]
Apr  7 15:25:26 ovirt gluster_bricks-big-big[39632]: [2020-04-07 13:25:26.790284] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev
_cbk] 0-big-server: 20535: WRITEV 1 (e965b2e9-500b-4520-a17e-d00b93a2e1bf), client: CTX_ID:2530ad70-1cec-43a2-bdf8-8c5a085fecb4-GRAPH_ID:0-PI
D:41137-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0, error-xlator: big-posix [Invalid argument]
Apr  7 15:26:16 ovirt gluster_bricks-engine-engine[10652]: [2020-04-07 13:26:16.183229] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_wr
itev] 0-engine-posix: write failed: offset 0, [Invalid argument]
Apr  7 15:26:16 ovirt gluster_bricks-engine-engine[10652]: [2020-04-07 13:26:16.183318] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_
writev_cbk] 0-engine-server: 359871: WRITEV 5 (cb0ea391-ed2f-4318-91d3-b8e18f6bdfb7), client: CTX_ID:08f296e3-beeb-4658-8e79-d9add812c3d7-GRA
PH_ID:0-PID:11400-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
Apr  7 15:27:28 ovirt gluster_bricks-vmstore-vmstore[10678]: [2020-04-07 13:27:28.719391] W [MSGID: 113020] [posix-helpers.c:1051:posix_gfid_set] 0-vmstore-posix: setting GFID on /gluster_bricks/vmstore/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed  [File exists]
Apr  7 15:27:28 ovirt gluster_bricks-vmstore-vmstore[10678]: [2020-04-07 13:27:28.719978] E [MSGID: 113020] [posix-entry-ops.c:517:posix_mknod] 0-vmstore-posix: setting gfid on /gluster_bricks/vmstore/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File exists]
Apr  7 15:29:16 ovirt gluster_bricks-data-data[10609]: [2020-04-07 13:29:16.232149] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-data-posix: write failed: offset 0, [Invalid argument]
 
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OXEDL5253WJBWISZCKMUBMVYFTVS6DBA/