Re: [Users] hosted engine issues

----- Original Message ----- > From: "René Koch" <rkoch(a)linuxland.at> To: "Yedidyah Bar David" <didi(a)redhat.com&gt;, "Martin Sivak" <msivak(a)redhat.com> Cc: users(a)ovirt.org > Sent: Monday, March 3, 2014 4:10:51 PM > Subject: Re: [Users] hosted engine issues > On 03/03/2014 02:13 PM, Yedidyah Bar David wrote: >>> Me neither. Is everything Read-Write? Read-Only FS might report no space >>> left >>> as well in some cases. Other than that, I do not know. >>> Perhaps some ipc resource? semaphores? >>> Please check: >>> ipcs >>> cat /proc/sys/kernel/sem >>> I know nothing about libvirt, that's just a wild guess. > # ipcs > ------ Shared Memory Segments -------- > key shmid owner perms bytes nattch status > 0x00000000 0 root 644 80 2 > 0x00000000 32769 root 644 16384 2 > 0x00000000 65538 root 644 280 2 > ------ Semaphore Arrays -------- > key semid owner perms nsems > 0x00000000 0 root 600 1 > 0x00000000 65537 root 600 1 > 0x000000a7 163842 root 600 1 This means you have 3 semaphore sets, of one semaphore each. > ------ Message Queues -------- > key msqid owner perms used-bytes messages Also the rest is moderate usage. > # cat /proc/sys/kernel/sem > 250 32000 32 128 So you are far from the maxima (250 per set, 32000 total, 128 sets). > Do you see anything in this output? > I have no clue how to interpret this... See e.g. http://man7.org/linux/man-pages/man5/proc.5.html Is the above on a node? engine? both nodes are similar? If so, that's not the reason for the "no space left on device".

If this error is reproducible, you can try to find the process that this happens to (perhaps libvirtd, vdsmd, or the hosted-engine ha daemon) and do: strace -f -o /tmp/trace1 -tt -s 512 -p PID where PID is the pid of that process, then search /tmp/trace1 for 'no space left on device' and see the exact call that failed.