This is a multi-part message in MIME format. --------------010305030902080508040003 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, due cleaning envir. I restarted all nodes and ovirt mgm server too, Now I can't get master domain ON. Tried to move to diff. storage with same result. All storages are OFF and DataCenter OFF too. All NFS mount points are connected and writeable via ovirt mountpoint. Any idea how to fix thi - quicky Rhel 7.1 / KVM 2.1.2 / libvirt 1.2.8-16 / vdsm 4.16.27 / gluster 3.7.5.1 / ovirt 3.5.2 VDSM log ======== Thread-22688::DEBUG::2016-02-24 01:32:34,852::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-22688::DEBUG::2016-02-24 01:32:34,852::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-22688::DEBUG::2016-02-24 01:32:34,852::task::993::Storage.TaskManager.Task::(_decref) Task=`492b6d3f-47b7-47fe-a713-6e47d5f2d3c1`::ref 0 aborting False Thread-22688::DEBUG::2016-02-24 01:32:34,855::stompReactor::162::yajsonrpc.StompServer::(send) Sending response Thread-22697::DEBUG::2016-02-24 01:32:34,967::stompReactor::162::yajsonrpc.StompServer::(send) Sending response Thread-200::DEBUG::2016-02-24 01:32:35,001::fileSD::153::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97 Thread-200::ERROR::2016-02-24 01:32:35,007::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 300e9ac8-3c2f-4703-9bb1-1df2130c7c97 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 160, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 89, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 351, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-130::DEBUG::2016-02-24 01:32:35,013::fileSD::153::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4 Thread-130::ERROR::2016-02-24 01:32:35,017::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 553d9b92-e4a0-4042-a579-4cabeb55ded4 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 160, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 89, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 351, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error thx Pavel --------------010305030902080508040003 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000066" bgcolor="#FFFFFF"> Hi, <br> due cleaning envir. I restarted all nodes and ovirt mgm server too, <br> Now I can't  get master domain ON. Tried to move to diff. storage with same result.<br> All storages are OFF and DataCenter OFF too.<br> All NFS mount points are connected and writeable via ovirt mountpoint.<br> Any idea how to fix thi - quicky<br> <br> Rhel 7.1 /  KVM 2.1.2 / libvirt 1.2.8-16 / vdsm 4.16.27 / gluster 3.7.5.1 / ovirt 3.5.2 <br> <br> VDSM log<br> ========<br> Thread-22688::DEBUG::2016-02-24 01:32:34,852::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<br> Thread-22688::DEBUG::2016-02-24 01:32:34,852::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br> Thread-22688::DEBUG::2016-02-24 01:32:34,852::task::993::Storage.TaskManager.Task::(_decref) Task=`492b6d3f-47b7-47fe-a713-6e47d5f2d3c1`::ref 0 aborting False<br> Thread-22688::DEBUG::2016-02-24 01:32:34,855::stompReactor::162::yajsonrpc.StompServer::(send) Sending response<br> Thread-22697::DEBUG::2016-02-24 01:32:34,967::stompReactor::162::yajsonrpc.StompServer::(send) Sending response<br> Thread-200::DEBUG::2016-02-24 01:32:35,001::fileSD::153::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97<br> Thread-200::ERROR::2016-02-24 01:32:35,007::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 300e9ac8-3c2f-4703-9bb1-1df2130c7c97 monitoring information<br> Traceback (most recent call last):<br>   File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain<br>     self.domain = sdCache.produce(self.sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 98, in produce<br>     domain.getRealDomain()<br>   File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<br>     return self._cache._realProduce(self._sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<br>     domain = self._findDomain(sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<br>     dom = findMethod(sdUUID)<br>   File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain<br>     return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<br>   File "/usr/share/vdsm/storage/fileSD.py", line 160, in __init__<br>     validateFileSystemFeatures(sdUUID, self.mountpoint)<br>   File "/usr/share/vdsm/storage/fileSD.py", line 89, in validateFileSystemFeatures<br>     oop.getProcessPool(sdUUID).directTouch(testFilePath)<br>   File "/usr/share/vdsm/storage/outOfProcess.py", line 351, in directTouch<br>     ioproc.touch(path, flags, mode)<br>   File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 507, in touch<br>     self.timeout)<br>   File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 391, in _sendCommand<br>     raise OSError(errcode, errstr)<br> OSError: [Errno 5] Input/output error<br> Thread-130::DEBUG::2016-02-24 01:32:35,013::fileSD::153::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4<br> Thread-130::ERROR::2016-02-24 01:32:35,017::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 553d9b92-e4a0-4042-a579-4cabeb55ded4 monitoring information<br> Traceback (most recent call last):<br>   File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain<br>     self.domain = sdCache.produce(self.sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 98, in produce<br>     domain.getRealDomain()<br>   File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<br>     return self._cache._realProduce(self._sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<br>     domain = self._findDomain(sdUUID)<br>   File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<br>     dom = findMethod(sdUUID)<br>   File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain<br>     return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<br>   File "/usr/share/vdsm/storage/fileSD.py", line 160, in __init__<br>     validateFileSystemFeatures(sdUUID, self.mountpoint)<br>   File "/usr/share/vdsm/storage/fileSD.py", line 89, in validateFileSystemFeatures<br>     oop.getProcessPool(sdUUID).directTouch(testFilePath)<br>   File "/usr/share/vdsm/storage/outOfProcess.py", line 351, in directTouch<br>     ioproc.touch(path, flags, mode)<br>   File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 507, in touch<br>     self.timeout)<br>   File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 391, in _sendCommand<br>     raise OSError(errcode, errstr)<br> OSError: [Errno 5] Input/output error<br> <br> thx Pavel<br> <br> </body> </html> --------------010305030902080508040003--