[ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

Sun Aug 17 19:40:22 EDT 2014

Hello All,

I am having an issue on my test setup, the master storage goes inactive.  It
is set up as to use Gluster and the volume is up and running without issues.
Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the
ERRORs):

2014-08-18 01:47:04,173 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method

2014-08-18 01:47:04,174 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:14,286 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method

2014-08-18 01:47:14,287 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method

2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:34,514 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method

2014-08-18 01:47:34,515 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:44,651 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method

2014-08-18 01:47:44,652 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName =
u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5,
storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed
to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:54,765 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method

2014-08-18 01:47:54,766 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:04,881 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method

2014-08-18 01:48:04,883 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:14,994 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method

2014-08-18 01:48:14,995 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:25,117 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method

2014-08-18 01:48:25,118 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:35,231 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method

2014-08-18 01:48:35,232 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:45,344 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method

2014-08-18 01:48:45,345 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:55,460 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method

2014-08-18 01:48:55,461 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:49:05,574 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method

2014-08-18 01:49:05,575 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

Here is the vdsm.log on one of the hosts:

[root at u5 ~]# tail -f /var/log/vdsm/vdsm.log

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-1489::DEBUG::2014-08-17
19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]

Thread-1489::DEBUG::2014-08-17
19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect:
getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing
request...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building
response

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating
json...

Thread-15::DEBUG::2014-08-17
19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-1489::DEBUG::2014-08-17
19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...

Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is
147DEBUG|(3641) Building response

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response
sized 86

Thread-1489::ERROR::2014-08-17
19:39:18,955::task::866::Storage.TaskManager.Task::(_setError)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/task.py", line 873, in _run

    return fn(*args, **kargs)

  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper

    res = f(*args, **kwargs)

  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus

    status = self._getSpmStatusInfo(pool)

  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo

    (pool.spmRole,) + pool.getSpmStatus()))

  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus

    return self._backend.getSpmStatus()

  File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus

    lVer, spmId = self.masterDomain.inquireClusterLock()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::885::Storage.TaskManager.Task::(_run)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run:
ada614f8-75c5-4728-9083-340f918a4417
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task

Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::1217::Storage.TaskManager.Task::(stop)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing
(force False)

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True

Thread-1489::INFO::2014-08-17
19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted:
u'[Errno 5] Input/output error' - code 100

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5]
Input/output error

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False

Thread-1489::DEBUG::2014-08-17
19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing ->
state aborting

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting ->
state failed

Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::ERROR::2014-08-17
19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5]
Input/output error

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper

    result = ctask.prepare(func, *args, **kwargs)

  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper

    return m(self, *a, **kw)

  File "/usr/share/vdsm/storage/task.py", line 1179, in prepare

    raise self.error

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-1489::DEBUG::2014-08-17
19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]
flowID [fedc88e]

Thread-1489::DEBUG::2014-08-17
19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect:
connectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '',
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type':
'glusterfs', 'password': '******', 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)

Thread-1489::DEBUG::2014-08-17
19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath:
glusterSD/*

Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|(3642) Building response

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Generating json...

Thread-15::DEBUG::2014-08-17
19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids:
(u'2ed32e34-2483-4262-a57b-72bf70f3e989',)

Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs:
{2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}

Thread-1489::INFO::2014-08-17
19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect:
connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}]}

Thread-1489::DEBUG::2014-08-17
19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist':
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing ->
state finished

Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::993::Storage.TaskManager.Task::(_decref)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False

Thread-1489::DEBUG::2014-08-17
19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect:
repoStats(options=None)

Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989':
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck':
'6.3', 'valid': False}}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing ->
state finished

Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:21,022::task::993::Storage.TaskManager.Task::(_decref)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False

Thread-36::DEBUG::2014-08-17
19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method

Thread-29::DEBUG::2014-08-17
19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this
will take up to 30 seconds

Thread-29::DEBUG::2014-08-17
19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)

Thread-29::DEBUG::2014-08-17
19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last
result

Thread-29::DEBUG::2014-08-17
19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n
/sbin/multipath (cwd None)

Thread-29::DEBUG::2014-08-17
19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> =
''; <rc> = 0

Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last
result

Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3643) Building response

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-29::DEBUG::2014-08-17
19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3644) Building response

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Sending response sized 86

Thread-29::ERROR::2014-08-17
19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain

    self.domain.selftest()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-29::DEBUG::2014-08-17
19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-29::ERROR::2014-08-17
19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain

    self.domain.selftest()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Any help is getting this sorted is appreciated.

Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140817/67db71c4/attachment-0001.html>