<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="GENERATOR" content="GtkHTML/4.4.4">
</head>
<body>
Hi,<br>
<br>
this is a bit complex issue, so I´l try and be as clear as possible. We are running oVirt-3.1 in our production environment, based on minimal Fedora 17 installs. We have 4xHP 380's (Intel) running in one cluster, and 2xSun 7310's (AMD) in another cluster. They
have shared storage over NFS to a FreeBSD-based system that uses ZFS as a filesystem. The storage boots off of a mirrored ZFS pool made up of two USB's that only houses /, while /var, /usr, etc. lies on a separate ZFS pool made up of the rest of the HDD's
in the system. It looks like this:<br>
<br>
FS MOUNTPOINT<br>
pool1 (The mirrored USB's) none<br>
pool1/root / (mounted ro)<br>
pool2 (The regular HDD's) none<br>
pool2/root none<br>
pool2/root/usr /usr<br>
pool2/root/usr/home /usr/home<br>
pool2/root/usr/local /usr/local<br>
pool2/root/var /var<br>
tmpfs /tmp<br>
pool2/export /export<br>
pool2/export/ds1 /export/ds1<br>
pool2/export/ds1/data /export/ds1/data<br>
pool2/export/ds1/export /export/ds1/export<br>
pool2/export/ds1/iso /export/ds1/iso<br>
pool2/export/ds2 /export/ds2<br>
pool2/export/ds2/data /export/ds2/data<br>
<br>
/etc/exports:<br>
/export/ds1/data -alldirs -maproot=root 10.0.0.(all of the HV's)<br>
/export/ds1/export -alldirs -maproot=root 10.0.0.(all of the HV's)<br>
/export/ds1/iso -alldirs -maproot=root 10.0.0.(all of the HV's)<br>
/export/ds2/data -alldirs -maproot=root 10.0.0.(all of the HV's)<br>
<br>
To make those USB's last for as long as possible, / is usually mounted read-only. And when you need to change anything, you need to remount / to read-write, do the maintenance, and then remount back to read-only again. But when you issue a mount command, the
VM's in oVirt pause. At first we didn´t understand that was actually the cause and tried to correlate the seemingly spontaneous pausing to just about anything, Then I was logged in to both oVirt's webadmin, and the storage at the same and issued "mount -uw
/", and *boom*, random VM's started to pause:) Not all of them though, and not just every one in either cluster or something, it is completely random which VM's are paused every time.<br>
<br>
# time mount -ur /<br>
<br>
real 0m2.198s<br>
user 0m0.000s<br>
sys 0m0.002s<br>
<br>
And here´s what vdsm on one of the HV's thought about that:<br>
<a href="http://pastebin.com/MXjgpDfU">http://pastebin.com/MXjgpDfU</a><br>
<br>
It begins with all VM's being "Up", then me issuing the remount on the storage from read-write to read-only which took 2 secs to complete, vdsm freaking out when it shortly looses it´s connections and lastly me at 14:34 making them all run again from webadmin.<br>
<br>
Two things:<br>
1) Does anyone know of any improvements that could be made on the storage side, apart from the obvious "stop remounting", since patching must eventually be made, configurations changed, and so on. A smarter way of configuring something? Booting from another
ordinary HDD is sadly out of the question because there isn´t any room for any more, it´s full. And I would have really like it rather to boot from the HDD's that are already in there, but there are "other things" preventing that.<br>
2) Nothing in engine was logged about it, no "Events" were made and nothing in engine.log that could indicate something had gone wrong at all. If it wasn´t serious enough to issue a warning, why disrupt the service with pausing the machines? Or at least automatically
start them back up when connection to the storage almost immediately came back on it´s own. Saying nothing made it really hard to troubleshoot, since we didn´t initially knew at all what could be causing the pauses to happen, and when.<br>
<br>
Best Regards<br>
/Karli Sjöberg
</body>
</html>