<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Jan 12, 2016 at 9:32 AM, Markus Stockhausen <span dir="ltr"><<a href="mailto:stockhausen@collogia.de" target="_blank">stockhausen@collogia.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi there,<br>
<br>
we got a nasty situation yesterday in our OVirt 3.5.6 environment.<br>
We ran a LSM that failed during the cleanup operation. To be precise<br>
when the process deleted an image on the source NFS storage.<br></blockquote><div><br></div><div>Can you share with us your NFS server details? </div><div>Is the NFS connection healthy (can be seen with nfsstat)</div><div>Generally, delete on NFS should be a pretty quick operation. </div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Engine log gives:<br>
<br>
2016-01-11 20:49:45,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-14) [77277f0] START, DeleteImageGroupVDSCommand( storagePoolId = 94ed7a19-fade-4bd6-83f2-2cbb2f730b95, ignoreFailoverLimit = false, storageDomainId = 272ec473-6041-42ee-bd1a-732789dd18d4, imageGroupId = aed132ef-703a-44d0-b875-db8c0d2c1a92, postZeros = false, forceDelete = false), log id: b52d59c<br>
...<br>
2016-01-11 20:50:45,206 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-14) [77277f0] Failed in DeleteImageGroupVDS method<br>
<br>
VDSM (SPM) log gives:<br>
<br>
Thread-97::DEBUG::2016-01-11 20:49:45,737::fileSD::384::Storage.StorageDomain::(deleteImage) Removing file: /rhev/data-center/mnt/1.2.3.4:_var_nas2_OVirtIB/272ec473-6041-42ee-bd1a-732789dd18d4/images/_remojzBd1r/0d623afb-291e-4f4c-acba-caecb125c4ed<br>
...<br>
Thread-97::ERROR::2016-01-11 20:50:45,737::task::866::Storage.TaskManager.Task::(_setError) Task=`cd477878-47b4-44b1-85a3-b5da19543a5e`::Unexpected error<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/storage/task.py", line 873, in _run<br>
return fn(*args, **kargs)<br>
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper<br>
res = f(*args, **kwargs)<br>
File "/usr/share/vdsm/storage/hsm.py", line 1549, in deleteImage<br>
pool.deleteImage(dom, imgUUID, volsByImg)<br>
File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper<br>
return method(self, *args, **kwargs)<br>
File "/usr/share/vdsm/storage/sp.py", line 1884, in deleteImage<br>
domain.deleteImage(domain.sdUUID, imgUUID, volsByImg)<br>
File "/usr/share/vdsm/storage/fileSD.py", line 385, in deleteImage<br>
self.oop.os.remove(volPath)<br>
File "/usr/share/vdsm/storage/outOfProcess.py", line 245, in remove<br>
self._iop.unlink(path)<br>
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 455, in unlink<br>
return self._sendCommand("unlink", {"path": path}, self.timeout)<br>
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 385, in _sendCommand<br>
raise Timeout(os.strerror(errno.ETIMEDOUT))<br>
Timeout: Connection timed out<br>
<br>
Reading the docs I got the idea that vdsm default 60 second timeout<br>
for IO operations might be changed within /etc/vdsm/vdsm.conf<br>
<br>
[irs]<br>
process_pool_timeout = 180<br>
<br>
Can anyone confirm that this will solve the problem?<br>
<span class="HOEnZb"><font color="#888888"><br>
Markus<br>
<br>
<br>
<br>
<br>
<br>
</font></span><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>