11:37 a.m.
I'm creating a test box I can more thoroughly mess with so I can submit to
bugzilla something. Since my errors all popped up while trying to get
ovirt and gluster functional again rather than thoroughly gather logs and
test my data is kinda sketchy.
*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
On Thu, Jul 21, 2016 at 8:18 AM, Scott <romracer(a)gmail.com> wrote:
I get similar problems with oVirt 4.0.1 and hosted engine. After
upgrading all my hosts to Gluster 3.7.13 (client and server), I get the
following:
$ sudo hosted-engine --set-maintenance --mode=none
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
line 73, in <module>
if not maintenance.set_mode(sys.argv[1]):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
line 61, in set_mode
value=m_global,
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 259, in set_maintenance_mode
str(value))
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 204, in set_global_md_flag
all_stats = broker.get_stats_from_storage(service)
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 232, in get_stats_from_storage
result = self._checked_communicate(request)
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 260, in _checked_communicate
.format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed
to read metadata: [Errno 1] Operation not permitted
If I only upgrade one host, then things will continue to work but my nodes
are constantly healing shards. My logs are also flooded with:
[2016-07-21 13:15:14.137734] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274714: READ => -1 gfid=4
41f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
permitted)
The message "W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
operation failed [Operation not permitted]" repeated 6 times between
[2016-07-21 13:13:24.134985] and [2016-07-21 13:15:04.132226]
The message "W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
operation failed [Operation not permitted]" repeated 8 times between
[2016-07-21 13:13:34.133116] and [2016-07-21 13:15:14.137178]
The message "W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
operation failed [Operation not permitted]" repeated 7 times between
[2016-07-21 13:13:24.135071] and [2016-07-21 13:15:14.137666]
[2016-07-21 13:15:24.134647] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
operation failed [Operation not permitted]
[2016-07-21 13:15:24.134764] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
operation failed [Operation not permitted]
[2016-07-21 13:15:24.134793] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274741: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
permitted)
[2016-07-21 13:15:34.135413] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274756: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
permitted)
[2016-07-21 13:15:44.141062] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274818: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
permitted)
[2016-07-21 13:15:54.133582] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
operation failed [Operation not permitted]
[2016-07-21 13:15:54.133629] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274853: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0036d8 (Operation not
permitted)
[2016-07-21 13:16:04.133666] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274879: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
permitted)
[2016-07-21 13:16:14.134954] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 274894: READ => -1
gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0036d8 (Operation not
permitted)
Scott
On Thu, Jul 21, 2016 at 6:57 AM Frank Rothenstein <
f.rothenstein(a)bodden-kliniken.de> wrote:
> Hey Devid,
>
> I have the very same problem on my test-cluster, despite on running ovirt
> 4.0.
> If you access your volumes via NFS all is fine, problem is FUSE. I stayed
> on 3.7.13, but have no solution yet, now I use NFS.
>
> Frank
>
> Am Donnerstag, den 21.07.2016, 04:28 -0500 schrieb David Gossage:
>
> Anyone running one of recent 3.6.x lines and gluster using 3.7.13? I am
> looking to upgrade gluster from 3.7.11->3.7.13 for some bug fixes, but have
> been told by users on gluster mail list due to some gluster changes I'd
> need to change the disk parameters to use writeback cache. Something to do
> with aio support being removed.
>
> I believe this could be done with custom parameters? But I believe
> strage tests are done using dd and would they fail with current settings
> then? Last upgrade to 3.7.13 I had to rollback to 3.7.11 due to stability
> isues where gluster storage would go into down state and always show N/A as
> space available/used. Even if hosts saw storage still and VM's were
> running on it on all 3 hosts.
>
> Saw a lot of messages like these that went away once gluster rollback
> finished
>
> [2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel
> 7.22
> [2016-07-09 15:27:49.555466] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote
> operation failed [Operation not permitted]
> [2016-07-09 15:27:49.556574] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote
> operation failed [Operation not permitted]
> [2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d
> fd=0x7f5224002f68 (Operation not permitted)
> [2016-07-09 15:27:59.612477] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote
> operation failed [Operation not permitted]
> [2016-07-09 15:27:59.613700] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote
> operation failed [Operation not permitted]
> [2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d
> fd=0x7f5224002f68 (Operation not permitted)
>
> *David Gossage*
> *Carousel Checks Inc. | System Administrator*
> *Office* 708.613.2284
>
> _______________________________________________
> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ------------------------------
>
>
>
>
> ______________________________________________________________________________
> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> Sandhufe 2
> 18311 Ribnitz-Damgarten
>
> Telefon: 03821-700-0
> Fax: 03821-700-240
>
> E-Mail: info(a)bodden-kliniken.de Internet: http://www.bodden-kliniken.de
>
>
> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 079/133/40188
>
> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski
>
>
> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt.
Wenn Sie nicht der vorge-
>
> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte,
dass jede Form der Veröf-
>
> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig
ist. Wir bitten Sie, sofort den
> Absender zu informieren und die E-Mail zu löschen.
>
>
> Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>