3:30 p.m.
--------------030708060003050100080602
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
I'm sure I must have just missed something...
I just setup a new ovirt cluster with gluster & nfs data domains.
VMs on the NFS domain startup with no issues.
VMs on the gluster domains complain of "Permission denied" on startup.
2016-05-17 14:14:51,959 ERROR [org.ovirt.engine.core.dal.dbbroker.audi
tloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] Correlation
ID: null, Call Stack: null, Custom Event ID: -1, Message: VM
billj7-2.j2noc.com is down with error. Exit message: internal error:
process exited while connecting to monitor: 2016-05-17T21:14:51.162932Z
qemu-kvm: -drive
file=/rhev/data-center/00000001-0001-0001-0001-0000000002c5/22df0943-c131-4ed8-ba9c-05923afcf8e3/images/2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25/a2b0a04d-041f-4342-9687-142cc641b35e,if=none,id=drive-virtio-disk0,format=raw,serial=2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25,cache=none,werror=stop,rerror=stop,aio=threads:
Could not open
'/rhev/data-center/00000001-0001-0001-0001-0000000002c5/22df0943-c131-4ed8-ba9c-05923afcf8e3/images/2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25/a2b0a04d-041f-4342-9687-142cc641b35e':
Permission denied
I did setup gluster permissions:
gluster volume set gv1 storage.owner-uid 36
gluster volume set gv1 storage.owner-gid 36
files look fine:
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# ls -lah
total 2.0G
drwxr-xr-x 2 vdsm kvm 4.0K May 17 09:39 .
drwxr-xr-x 11 vdsm kvm 4.0K May 17 10:40 ..
-rw-rw---- 1 vdsm kvm 20G May 17 10:33
a2b0a04d-041f-4342-9687-142cc641b35e
-rw-rw---- 1 vdsm kvm 1.0M May 17 09:38
a2b0a04d-041f-4342-9687-142cc641b35e.lease
-rw-r--r-- 1 vdsm kvm 259 May 17 09:39
a2b0a04d-041f-4342-9687-142cc641b35e.meta
I did check and vdsm user can read the file just fine.
*If I change mod disk to 666 VM starts up fine.*
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# grep 36
/etc/passwd /etc/group
/etc/passwd:vdsm:x:36:36:Node Virtualization Manager:/:/bin/bash
/etc/group:kvm:x:36:qemu,sanlock
ovirt-engine-3.6.4.1-1.el7.centos.noarch
glusterfs-3.7.11-1.el7.x86_64
I also set libvirt qemu user to root, for import-to-ovirt.pl script.
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# grep ^user
/etc/libvirt/qemu.conf
user = "root"
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# gluster volume
info gv1
Volume Name: gv1
Type: Replicate
Volume ID: 062aa1a5-91e8-420d-800e-b8bc4aff20d8
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-gl.j2noc.com:/ovirt-store/brick1/gv1
Brick2: ovirt2-gl.j2noc.com:/ovirt-store/brick1/gv1
Brick3: ovirt3-gl.j2noc.com:/ovirt-store/brick1/gv1
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
features.shard: on
features.shard-block-size: 64MB
storage.owner-uid: 36
storage.owner-gid: 36
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# gluster volume
status gv1
Status of volume: gv1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovirt1-gl.j2noc.com:/ovirt-store/bric
k1/gv1 49152 0 Y 2046
Brick ovirt2-gl.j2noc.com:/ovirt-store/bric
k1/gv1 49152 0 Y 22532
Brick ovirt3-gl.j2noc.com:/ovirt-store/bric
k1/gv1 49152 0 Y 59683
NFS Server on localhost 2049 0 Y 2200
Self-heal Daemon on localhost N/A N/A Y 2232
NFS Server on ovirt3-gl.j2noc.com 2049 0 Y 65363
Self-heal Daemon on ovirt3-gl.j2noc.com N/A N/A Y 65371
NFS Server on ovirt2-gl.j2noc.com 2049 0 Y 17621
Self-heal Daemon on ovirt2-gl.j2noc.com N/A N/A Y 17629
Task Status of Volume gv1
------------------------------------------------------------------------------
There are no active volume tasks
??
--------------030708060003050100080602
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I'm sure I must have just missed something...<br>
I just setup a new ovirt cluster with gluster & nfs data
domains.<br>
<br>
VMs on the NFS domain startup with no issues.<br>
VMs on the gluster domains complain of "Permission denied" on
startup.<br>
<br>
2016-05-17 14:14:51,959 ERROR
[org.ovirt.engine.core.dal.dbbroker.audi<br>
tloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) []
Correlation ID: null, Call Stack: null, Custom Event ID: -1,
Message: VM billj7-2.j2noc.com is down with error. Exit message:
internal error: process exited while connecting to monitor:
2016-05-17T21:14:51.162932Z qemu-kvm: -drive
file=/rhev/data-center/00000001-0001-0001-0001-0000000002c5/22df0943-c131-4ed8-ba9c-05923afcf8e3/images/2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25/a2b0a04d-041f-4342-9687-142cc641b35e,if=none,id=drive-virtio-disk0,format=raw,serial=2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25,cache=none,werror=stop,rerror=stop,aio=threads:
Could not open
'/rhev/data-center/00000001-0001-0001-0001-0000000002c5/22df0943-c131-4ed8-ba9c-05923afcf8e3/images/2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25/a2b0a04d-041f-4342-9687-142cc641b35e':
Permission denied<br>
<br>
<br>
I did setup gluster permissions:<br>
gluster volume set gv1 storage.owner-uid 36<br>
gluster volume set gv1 storage.owner-gid 36<br>
<br>
files look fine:<br>
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# ls -lah<br>
total 2.0G<br>
drwxr-xr-x 2 vdsm kvm 4.0K May 17 09:39 .<br>
drwxr-xr-x 11 vdsm kvm 4.0K May 17 10:40 ..<br>
-rw-rw---- 1 vdsm kvm 20G May 17 10:33
a2b0a04d-041f-4342-9687-142cc641b35e<br>
-rw-rw---- 1 vdsm kvm 1.0M May 17 09:38
a2b0a04d-041f-4342-9687-142cc641b35e.lease<br>
-rw-r--r-- 1 vdsm kvm 259 May 17 09:39
a2b0a04d-041f-4342-9687-142cc641b35e.meta<br>
<br>
I did check and vdsm user can read the file just fine.<br>
<b>If I change mod disk to 666 VM starts up fine.</b><br>
<br>
<br>
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# grep 36
/etc/passwd /etc/group<br>
/etc/passwd:vdsm:x:36:36:Node Virtualization Manager:/:/bin/bash<br>
/etc/group:kvm:x:36:qemu,sanlock<br>
<br>
<br>
ovirt-engine-3.6.4.1-1.el7.centos.noarch<br>
glusterfs-3.7.11-1.el7.x86_64<br>
<br>
<br>
I also set libvirt qemu user to root, for import-to-ovirt.pl script.<br>
<br>
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# grep ^user
/etc/libvirt/qemu.conf <br>
user = "root"<br>
<br>
<br>
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# gluster
volume info gv1<br>
<br>
Volume Name: gv1<br>
Type: Replicate<br>
Volume ID: 062aa1a5-91e8-420d-800e-b8bc4aff20d8<br>
Status: Started<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: ovirt1-gl.j2noc.com:/ovirt-store/brick1/gv1<br>
Brick2: ovirt2-gl.j2noc.com:/ovirt-store/brick1/gv1<br>
Brick3: ovirt3-gl.j2noc.com:/ovirt-store/brick1/gv1<br>
Options Reconfigured:<br>
performance.readdir-ahead: on<br>
performance.quick-read: off<br>
performance.read-ahead: off<br>
performance.io-cache: off<br>
performance.stat-prefetch: off<br>
cluster.eager-lock: enable<br>
network.remote-dio: enable<br>
cluster.quorum-type: auto<br>
cluster.server-quorum-type: server<br>
features.shard: on<br>
features.shard-block-size: 64MB<br>
storage.owner-uid: 36<br>
storage.owner-gid: 36<br>
<br>
[root@ovirt1 prod 2ddf0d0e-6a7e-4eb9-b1d5-6d7792da0d25]# gluster
volume status gv1<br>
Status of volume: gv1<br>
Gluster process TCP Port RDMA Port
Online Pid<br>
------------------------------------------------------------------------------<br>
Brick ovirt1-gl.j2noc.com:/ovirt-store/bric<br>
k1/gv1 49152 0
Y 2046 <br>
Brick ovirt2-gl.j2noc.com:/ovirt-store/bric<br>
k1/gv1 49152 0
Y 22532<br>
Brick ovirt3-gl.j2noc.com:/ovirt-store/bric<br>
k1/gv1 49152 0
Y 59683<br>
NFS Server on localhost 2049 0
Y 2200 <br>
Self-heal Daemon on localhost N/A N/A
Y 2232 <br>
NFS Server on ovirt3-gl.j2noc.com 2049 0
Y 65363<br>
Self-heal Daemon on ovirt3-gl.j2noc.com N/A N/A
Y 65371<br>
NFS Server on ovirt2-gl.j2noc.com 2049 0
Y 17621<br>
Self-heal Daemon on ovirt2-gl.j2noc.com N/A N/A
Y 17629<br>
<br>
Task Status of Volume gv1<br>
------------------------------------------------------------------------------<br>
There are no active volume tasks<br>
<br>
<br>
<br>
??<br>
<br>
<br>
</body>
</html>
--------------030708060003050100080602--