[ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

This is a multipart message in MIME format. ------=_NextPart_000_0B23_01CFBA53.164C8840 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello All, I am having an issue on my test setup, the master storage goes inactive. It is set up as to use Gluster and the volume is up and running without issues. Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the ERRORs): 2014-08-18 01:47:04,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method 2014-08-18 01:47:04,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:14,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method 2014-08-18 01:47:14,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method 2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:34,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method 2014-08-18 01:47:34,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:44,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method 2014-08-18 01:47:44,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:54,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method 2014-08-18 01:47:54,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:04,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method 2014-08-18 01:48:04,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:14,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method 2014-08-18 01:48:14,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:25,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method 2014-08-18 01:48:25,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:35,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method 2014-08-18 01:48:35,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:45,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method 2014-08-18 01:48:45,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:55,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method 2014-08-18 01:48:55,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:49:05,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method 2014-08-18 01:49:05,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 Here is the vdsm.log on one of the hosts: [root@u5 ~]# tail -f /var/log/vdsm/vdsm.log validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-1489::DEBUG::2014-08-17 19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] Thread-1489::DEBUG::2014-08-17 19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None) Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing request... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building response Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing response Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating json... Thread-15::DEBUG::2014-08-17 19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-1489::DEBUG::2014-08-17 19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is 147DEBUG|(3641) Building response Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing response Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response sized 86 Thread-1489::ERROR::2014-08-17 19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus return self._backend.getSpmStatus() File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus lVer, spmId = self.masterDomain.inquireClusterLock() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::885::Storage.TaskManager.Task::(_run) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: ada614f8-75c5-4728-9083-340f918a4417 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing (force False) Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True Thread-1489::INFO::2014-08-17 19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted: u'[Errno 5] Input/output error' - code 100 Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5] Input/output error Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False Thread-1489::DEBUG::2014-08-17 19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing -> state aborting Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting -> state failed Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::ERROR::2014-08-17 19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] Input/output error Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-1489::DEBUG::2014-08-17 19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [fedc88e] Thread-1489::DEBUG::2014-08-17 19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type': 'glusterfs', 'password': '******', 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None) Thread-1489::DEBUG::2014-08-17 19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/* Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|(3642) Building response Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing response Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Generating json... Thread-15::DEBUG::2014-08-17 19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',) Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: {2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain} Thread-1489::INFO::2014-08-17 19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-1489::DEBUG::2014-08-17 19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing -> state finished Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False Thread-1489::DEBUG::2014-08-17 19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}} Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}} Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing -> state finished Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False Thread-36::DEBUG::2014-08-17 19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-29::DEBUG::2014-08-17 19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds Thread-29::DEBUG::2014-08-17 19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) Thread-29::DEBUG::2014-08-17 19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last result Thread-29::DEBUG::2014-08-17 19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) Thread-29::DEBUG::2014-08-17 19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last result Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3643) Building response Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing response Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-29::DEBUG::2014-08-17 19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Queuing request... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3644) Building response Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing response Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Sending response sized 86 Thread-29::ERROR::2014-08-17 19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing response Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json... Thread-29::DEBUG::2014-08-17 19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing response Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json... Thread-29::ERROR::2014-08-17 19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Any help is getting this sorted is appreciated. Chris ------=_NextPart_000_0B23_01CFBA53.164C8840 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 14 = (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif";} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal>Hello = All,<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal>I am having an issue on my test setup, the master = storage goes inactive.  It is set up as to use Gluster and the = volume is up and running without issues.  Here is a snippet of = /var/log/ovirt-engine/engine.log (these are just the = ERRORs):<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal>2014-08-18 01:47:04,173 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:04,174 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-98) [40e8ae30] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:14,286 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:14,287 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-68) [357af14e] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:24,400 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:24,400 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-51) [11269c17] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:34,514 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:34,515 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-59) [40622d60] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:44,651 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:44,652 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = =3D u5.veerotech.net, HostId =3D 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, = storagePoolId =3D 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution = failed. Exception: VDSErrorException: VDSGenericException: = VDSErrorException: Failed to SpmStatusVDS, error =3D [Errno 5] = Input/output error, code =3D 100<o:p></o:p></p><p = class=3DMsoNormal>2014-08-18 01:47:54,765 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:54,766 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-73) [44b5af81] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:04,881 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:04,883 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:14,994 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:14,995 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-20) [4fed0eac] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:25,117 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:25,118 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-35) [48ff78af] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:35,231 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:35,232 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-16) [76fe9857] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:45,344 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:45,345 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-69) [61356e24] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:55,460 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:55,461 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-28) [38c2326b] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:49:05,574 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS = method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:49:05,575 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-87) [3c65466] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal>Here is the vdsm.log on one of the = hosts:<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal><span lang=3DFR-CA>[root@u5 ~]# tail -f = /var/log/vdsm/vdsm.log<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DFR-CA>    = </span>validateFileSystemFeatures(sdUUID, = self.mountpoint)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p></p><p = class=3DMsoNormal>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p></p><p class=3DMsoNormal>    = ioproc.touch(path, flags, mode)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p></p><p class=3DMsoNormal>    = self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p></p><p = class=3DMsoNormal>    raise OSError(errcode, = errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] = Input/output error<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client = [74.63.225.147]<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init = -> state preparing<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: = getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', = options=3DNone)<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding = callback 'glob'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got = request for method 'glob'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) = Building response<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding = callback 'touch'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got = request for method 'touch'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is = 147DEBUG|(3641) Building response<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response = sized 86<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::ERROR::2014-08-17 = 19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected = error<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent call = last):<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/task.py", line 873, in = _run<o:p></o:p></p><p class=3DMsoNormal>    return = fn(*args, **kargs)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/logUtils.py", line 45, in = wrapper<o:p></o:p></p><p class=3DMsoNormal>    res =3D = f(*args, **kwargs)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/hsm.py", line 611, in = getSpmStatus<o:p></o:p></p><p class=3DMsoNormal>    = status =3D self._getSpmStatusInfo(pool)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/hsm.py", line 605, in = _getSpmStatusInfo<o:p></o:p></p><p class=3DMsoNormal>    = (pool.spmRole,) + pool.getSpmStatus()))<o:p></o:p></p><p = class=3DMsoNormal>  File "/usr/share/vdsm/storage/sp.py", = line 126, in getSpmStatus<o:p></o:p></p><p = class=3DMsoNormal>    return = self._backend.getSpmStatus()<o:p></o:p></p><p class=3DMsoNormal>  = File "/usr/share/vdsm/storage/spbackends.py", line 416, in = getSpmStatus<o:p></o:p></p><p class=3DMsoNormal>    lVer, = spmId =3D self.masterDomain.inquireClusterLock()<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p></p><p class=3DMsoNormal>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p></p><p class=3DMsoNormal>    = domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom = =3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p></p><p class=3DMsoNormal>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p></p><p class=3DMsoNormal>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p></p><p = class=3DMsoNormal>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p></p><p class=3DMsoNormal>    = ioproc.touch(path, flags, mode)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p></p><p class=3DMsoNormal>    = self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p></p><p = class=3DMsoNormal>    raise OSError(errcode, = errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] = Input/output error<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,955::task::885::Storage.TaskManager.Task::(_run) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: = ada614f8-75c5-4728-9083-340f918a4417 = ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping = task<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state = preparing (force False)<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting = True<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is = aborted: u'[Errno 5] Input/output error' - code 100<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno = 5] Input/output error<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting = True<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force = False<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state = preparing -> state aborting<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy = none<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state = aborting -> state failed<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::ERROR::2014-08-17 = 19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] = Input/output error<o:p></o:p></p><p class=3DMsoNormal>Traceback (most = recent call last):<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/dispatcher.py", line 71, in = wrapper<o:p></o:p></p><p class=3DMsoNormal>    result =3D = ctask.prepare(func, *args, **kwargs)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/task.py", line 103, in = wrapper<o:p></o:p></p><p class=3DMsoNormal>    return = m(self, *a, **kw)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/task.py", line 1179, in = prepare<o:p></o:p></p><p class=3DMsoNormal>    raise = self.error<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] = Input/output error<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] = flowID [fedc88e]<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init = -> state preparing<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: = connectStorageServer(domType=3D7, = spUUID=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'port': '', = 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', = 'vfs_type': 'glusterfs', 'password': '******', 'id': = '08128803-0141-46aa-835a-2f9734ef0d3b'}], = options=3DNone)<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) = glusterDomPath: glusterSD/*<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|Receiving request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|Marshaling message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding = callback 'glob'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got = request for method 'glob'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|(3642) Building response<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Generating json...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD = uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: = {2ed32e34-2483-4262-a57b-72bf70f3e989: = storage.glusterSD.findDomain}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: = connectStorageServer, Return response: {'statuslist': [{'status': 0, = 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': = [{'status': 0, 'id': = '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state = preparing -> state finished<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting = False<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init = -> state preparing<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone)<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::INFO::2014-08-17 = 19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': = {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', = 'lastCheck': '6.3', 'valid': False}}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: = {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, = 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': = False}}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state = preparing -> state finished<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p></p><p = class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 = 19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting = False<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage)<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan)<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None)<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning = last result<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) = /usr/bin/sudo -n /sbin/multipath (cwd None)<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: = <err> =3D ''; <rc> =3D 0<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning = last result<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Marshaling message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding = callback 'glob'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got = request for method 'glob'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|(3643) Building response<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Queuing request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding = callback 'touch'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got = request for method 'touch'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|(3644) Building response<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Sending response sized 86<o:p></o:p></p><p = class=3DMsoNormal>Thread-29::ERROR::2014-08-17 = 19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring = information<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent = call last):<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain<o:p></o:p></p><p class=3DMsoNormal>    = self.domain.selftest()<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p></p><p class=3DMsoNormal>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p></p><p class=3DMsoNormal>    = domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom = =3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p></p><p class=3DMsoNormal>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p></p><p class=3DMsoNormal>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p></p><p = class=3DMsoNormal>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p></p><p class=3DMsoNormal>    = ioproc.touch(path, flags, mode)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p></p><p class=3DMsoNormal>    = self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p></p><p = class=3DMsoNormal>    raise OSError(errcode, = errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] = Input/output error<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 = 19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 = 19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding = callback 'glob'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got = request for method 'glob'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 = 19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding = callback 'touch'...<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got = request for method 'touch'<o:p></o:p></p><p = class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing = response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 = 19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p></p><p class=3DMsoNormal>Thread-29::ERROR::2014-08-17 = 19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring = information<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent = call last):<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain<o:p></o:p></p><p class=3DMsoNormal>    = self.domain.selftest()<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p></p><p class=3DMsoNormal>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p></p><p class=3DMsoNormal>    = domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom = =3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p></p><p class=3DMsoNormal>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p></p><p class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p></p><p class=3DMsoNormal>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p></p><p = class=3DMsoNormal>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p></p><p class=3DMsoNormal>    = ioproc.touch(path, flags, mode)<o:p></o:p></p><p = class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p></p><p class=3DMsoNormal>    = self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p></p><p = class=3DMsoNormal>    <span lang=3DFR-CA>raise = OSError(errcode, errstr)<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DFR-CA>OSError: [Errno 5] Input/output = error</span><o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Any help is = getting this sorted is appreciated.<o:p></o:p></p><p = class=3DMsoNormal>Chris<o:p></o:p></p></div></body></html> ------=_NextPart_000_0B23_01CFBA53.164C8840--