[ovirt-users] vm paused unknown storage error one node out of 3 only

[David Gossage]

Out of a 3 node cluster running oVirt 3.6.6.2-1.el7.centos with a 3
replicate gluster 3.7.14 starting a VM i just copied in on one node of the
3 gets the following errors.  The other 2 the vm starts fine.  All ovirt
and gluster are centos 7 based. VM on start of the one node it tries to
default to on its own accord immediately puts into paused for unknown
reason.  Telling it to start on different node starts ok.  node with issue
already has 5 VMs running fine on it same gluster storage plus the hosted
engine on different volume.

gluster nodes logs did not have any errors for volume
nodes own gluster logs had this in log

dfb8777a-7e8c-40ff-8faa-252beabba5f8 couldnt find in .glusterfs .shard or
images/

7919f4a0-125c-4b11-b5c9-fb50cc195c43 is the gfid of the bootable drive of
the vm

[2016-08-11 04:31:39.982952] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-2: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.983683] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.984182] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.984221] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.985941] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
subvolume -1 found with event generation 3 for gfid
dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
[2016-08-11 04:31:39.986633] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.987644] E [MSGID: 109040]
[dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht:
(null): failed to lookup the file on GLUSTER1-dht [Stale file handle]
[2016-08-11 04:31:39.987751] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 15152930: READ => -1
gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43 fd=0x7f00a80bdb64 (Stale file
handle)
[2016-08-11 04:31:39.986567] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
operation failed [No such file or directory]
[2016-08-11 04:31:39.986567] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.210145] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
subvolume -1 found with event generation 3 for gfid
dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
[2016-08-11 04:35:21.210873] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.210888] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.210947] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.213270] E [MSGID: 109040]
[dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht:
(null): failed to lookup the file on GLUSTER1-dht [Stale file handle]
[2016-08-11 04:35:21.213345] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 15156910: READ => -1
gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43 fd=0x7f00a80bf6d0 (Stale file
handle)
[2016-08-11 04:35:21.211516] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
subvolume -1 found with event generation 3 for gfid
dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
[2016-08-11 04:35:21.212013] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.212081] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
operation failed [No such file or directory]
[2016-08-11 04:35:21.212121] W [MSGID: 114031]
[client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
operation failed [No such file or directory]

I attached vdsm.log starting from when I spun up vm on offending node
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160811/d3f6d7de/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.zip
Type: application/zip
Size: 2045746 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160811/d3f6d7de/attachment-0001.zip>