----- Original Message -----
From: "René Koch" <rkoch@linuxland.at> To: "Yedidyah Bar David" <didi@redhat.com>, "Martin Sivak" <msivak@redhat.com> Cc: users@ovirt.org Sent: Monday, March 3, 2014 4:10:51 PM Subject: Re: [Users] hosted engine issues
On 03/03/2014 02:13 PM, Yedidyah Bar David wrote:
Me neither. Is everything Read-Write? Read-Only FS might report no space left as well in some cases. Other than that, I do not know.
Perhaps some ipc resource? semaphores?
Please check:
ipcs
cat /proc/sys/kernel/sem
I know nothing about libvirt, that's just a wild guess.
# ipcs
------ Shared Memory Segments -------- key shmid owner perms bytes nattch status
0x00000000 0 root 644 80 2
0x00000000 32769 root 644 16384 2
0x00000000 65538 root 644 280 2
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 65537 root 600 1 0x000000a7 163842 root 600 1
This means you have 3 semaphore sets, of one semaphore each.
------ Message Queues -------- key msqid owner perms used-bytes messages
Also the rest is moderate usage.
# cat /proc/sys/kernel/sem 250 32000 32 128
So you are far from the maxima (250 per set, 32000 total, 128 sets).
Do you see anything in this output? I have no clue how to interpret this...
See e.g. http://man7.org/linux/man-pages/man5/proc.5.html Is the above on a node? engine? both nodes are similar? If so, that's not the reason for the "no space left on device". If this error is reproducible, you can try to find the process that this happens to (perhaps libvirtd, vdsmd, or the hosted-engine ha daemon) and do: strace -f -o /tmp/trace1 -tt -s 512 -p PID where PID is the pid of that process, then search /tmp/trace1 for 'no space left on device' and see the exact call that failed. -- Didi