8:08 a.m.
Hi,
after an upgrade I get the following errors in the web gui:
VDSM ovirt-node01 command SpmStatusVDS failed: (13, 'Sanlock resource
read failure', 'Permission denied')
VDSM ovirt-node03 command HSMGetAllTasksStatusesVDS failed: Not SPM
These messages happen from all nodes.
I can stop vms and migrate them but I cannot start any vm again
How do I get bet to a sane state where one node is SPM.
Best,
Moritz
11:39 a.m.
Adding Nir, can you please have a look?
On Mon, May 29, 2017 at 2:15 PM, Moritz Baumann <moritz.baumann(a)inf.ethz.ch>
wrote:
Here is the vdsm.log from one node
Cheers,
Moritz
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
2:27 a.m.
On Mon, May 29, 2017 at 3:08 PM, Moritz Baumann <moritz.baumann(a)inf.ethz.ch>
wrote:
Hi,
after an upgrade I get the following errors in the web gui:
VDSM ovirt-node01 command SpmStatusVDS failed: (13, 'Sanlock resource read
failure', 'Permission denied')
What do you see on sanlock.log?
What kind of storage do you have, that you experience a permission denied
error? Is it a file-system based one, with actual permission issue?
Y.
VDSM ovirt-node03 command HSMGetAllTasksStatusesVDS failed: Not SPM
These messages happen from all nodes.
I can stop vms and migrate them but I cannot start any vm again
How do I get bet to a sane state where one node is SPM.
Best,
Moritz
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:58 a.m.
Hi,
VDSM ovirt-node01 command SpmStatusVDS failed: (13, 'Sanlock
resource read failure', 'Permission denied')
What do you see on sanlock.log?
I have attached the gzipped sanlock
What kind of storage do you have, that you experience a permission
denied error?
it is nfs3 based and I did an upgrade from RHEL 6.8 -> 6.9 and rebooted
the storage. I paused all running vms but forgot to put the storage
domains into maintenance.
At the moment all vms are stopped (clean) so I "could" remove the nodes
and reinstall if this helps.
Is it a file-system based one, with actual permission issue?
filesystem permissions are good, selinux is permissive at the moment.
Cheers,
Mo
1:06 p.m.
it is nfs3 based and I did an upgrade from RHEL 6.8 -> 6.9 and
rebooted
the storage. I paused all running vms but forgot to put the storage
domains into maintenance.
scratch-inf[0]:/export/scratch/ovirt# ls -lr *
iso:
total 0
-rwxr-xr-x. 1 36 kvm 0 29. Mai 18:02 __DIRECT_IO_TEST__
drwxr-xr-x. 4 36 kvm 32 21. Jul 2015 2851dcfe-3f64-408a-ad4a-c416790696eb
export:
total 0
-rwxr-xr-x. 1 36 kvm 0 29. Mai 16:28 __DIRECT_IO_TEST__
drwxr-xr-x. 5 36 kvm 45 21. Jul 2015 4cda1489-e241-4186-9500-0fd61640d895
data:
total 0
-rwxr-xr-x. 1 36 kvm 0 29. Mai 18:02 __DIRECT_IO_TEST__
drwxr-xr-x. 5 36 kvm 45 23. Jul 2015 c17d9d7f-e578-4626-a5d9-94ea555d7115
scratch-inf[0]:/export/scratch/ovirt# exportfs -v | grep ovirt
/export/scratch/ovirt
@ovirt-scratch(rw,async,wdelay,no_root_squash,no_subtree_check,fsid=200,sec=sys,rw,no_root_squash,no_all_squash)
I can write on nfs from all nodes as root (did not try with uid 36 but
don't see why it shouldn't work)
Cheers,
Mo
10:04 a.m.
I found some info on how to import an abandomed export domain
https://www.ovirt.org/documentation/how-to/storage/clear-the-storage-doma...
would the same (empty POOL_UUID and SHA_CKSUM in metadata) allow me to
import the data domain as a new one (and keep the existing vms) ?
10:35 a.m.
Just an idea, but could this be related to stale mounts from when you
rebooted the storage? Please try the following:
1. Place all nodes into maintenance mode
2. Disable the ovirt NFS exports
1. Comment out lines in /etc/exports
2. exportfs -r
3. Reboot your nodes
4. Re-enable the ovirt NFS exports
5. Activate your nodes
On Wed, May 31, 2017 at 10:04 AM, Moritz Baumann <moritz.baumann(a)inf.ethz.ch
wrote:
I found some info on how to import an abandomed export domain
https://www.ovirt.org/documentation/how-to/storage/clear-
the-storage-domain-pool-config-of-an-exported-nfs-domain/
would the same (empty POOL_UUID and SHA_CKSUM in metadata) allow me to
import the data domain as a new one (and keep the existing vms) ?
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Adam Litke
11:34 a.m.
Hi Adam,
Just an idea, but could this be related to stale mounts from when you
rebooted the storage? Please try the following:
1. Place all nodes into maintenance mode
2. Disable the ovirt NFS exports
1. Comment out lines in /etc/exports
2. exportfs -r
3. Reboot your nodes
4. Re-enable the ovirt NFS exports
5. Activate your nodes
all storage domains (data/iso) are down, so is the data center
(non-responsive) and no nfs mount is on any of the nodes.
I can however manually mount the data export and touch a file (as root).
So I think stale mounts is not the issue.
However I did the steps and the result is the same.
Best,
Mo
3:41 p.m.
I'm no NFS expert but for development domains I use the following options:
rw,sync,no_subtree_check,all_squash,anonuid=36,anongid=36
I wonder if something subtle changed on upgrade that interacts poorly with
your configuration?
On Wed, May 31, 2017 at 11:34 AM, Moritz Baumann <moritz.baumann(a)inf.ethz.ch
wrote:
Hi Adam,
Just an idea, but could this be related to stale mounts from when you
> rebooted the storage? Please try the following:
>
> 1. Place all nodes into maintenance mode
> 2. Disable the ovirt NFS exports
> 1. Comment out lines in /etc/exports
> 2. exportfs -r
> 3. Reboot your nodes
> 4. Re-enable the ovirt NFS exports
> 5. Activate your nodes
>
all storage domains (data/iso) are down, so is the data center
(non-responsive) and no nfs mount is on any of the nodes.
I can however manually mount the data export and touch a file (as root).
So I think stale mounts is not the issue.
However I did the steps and the result is the same.
Best,
Mo
--
Adam Litke
2766
days inactive
2768
days old
9 comments
4 participants
participants (4)
-
Adam Litke -
Moritz Baumann -
Sandro Bonazzola -
Yaniv Kaul