Re: [ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain - Users

20 Aug 2014

      This is a multipart message in MIME format.

------=_NextPart_000_0E08_01CFBC00.B56B8C60
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Still not able to correct this behavior.  Has anyone else seen this?

Chris

From: Chris @ VeeroTech.net [mailto:Chris@VeeroTech.net] 
Sent: Sunday, August 17, 2014 7:40 PM
To: users@ovirt.org
Subject: Master Storage goes Inactive - Failed to activate Storage Domain

Hello All,

I am having an issue on my test setup, the master storage goes inactive.  It
is set up as to use Gluster and the volume is up and running without issues.
Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the
ERRORs):

2014-08-18 01:47:04,173 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method

2014-08-18 01:47:04,174 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:14,286 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method

2014-08-18 01:47:14,287 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method

2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:34,514 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method

2014-08-18 01:47:34,515 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:44,651 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method

2014-08-18 01:47:44,652 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName =
u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5,
storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed
to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:47:54,765 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method

2014-08-18 01:47:54,766 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:04,881 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method

2014-08-18 01:48:04,883 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:14,994 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method

2014-08-18 01:48:14,995 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:25,117 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method

2014-08-18 01:48:25,118 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:35,231 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method

2014-08-18 01:48:35,232 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:45,344 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method

2014-08-18 01:48:45,345 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:48:55,460 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method

2014-08-18 01:48:55,461 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

2014-08-18 01:49:05,574 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method

2014-08-18 01:49:05,575 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100

Here is the vdsm.log on one of the hosts:

[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-1489::DEBUG::2014-08-17
19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]

Thread-1489::DEBUG::2014-08-17
19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect:
getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing
request...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building
response

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating
json...

Thread-15::DEBUG::2014-08-17
19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-1489::DEBUG::2014-08-17
19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...

Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is
147DEBUG|(3641) Building response

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response
sized 86

Thread-1489::ERROR::2014-08-17
19:39:18,955::task::866::Storage.TaskManager.Task::(_setError)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/task.py", line 873, in _run

    return fn(*args, **kargs)

  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper

    res = f(*args, **kwargs)

  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus

    status = self._getSpmStatusInfo(pool)

  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo

    (pool.spmRole,) + pool.getSpmStatus()))

  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus

    return self._backend.getSpmStatus()

  File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus

    lVer, spmId = self.masterDomain.inquireClusterLock()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::885::Storage.TaskManager.Task::(_run)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run:
ada614f8-75c5-4728-9083-340f918a4417
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task

Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::1217::Storage.TaskManager.Task::(stop)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing
(force False)

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True

Thread-1489::INFO::2014-08-17
19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted:
u'[Errno 5] Input/output error' - code 100

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5]
Input/output error

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True

Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False

Thread-1489::DEBUG::2014-08-17
19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing ->
state aborting

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none

Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting ->
state failed

Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::ERROR::2014-08-17
19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5]
Input/output error

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper

    result = ctask.prepare(func, *args, **kwargs)

  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper

    return m(self, *a, **kw)

  File "/usr/share/vdsm/storage/task.py", line 1179, in prepare

    raise self.error

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-1489::DEBUG::2014-08-17
19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]
flowID [fedc88e]

Thread-1489::DEBUG::2014-08-17
19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect:
connectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '',
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type':
'glusterfs', 'password': '******', 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)

Thread-1489::DEBUG::2014-08-17
19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath:
glusterSD/*

Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|(3642) Building response

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Generating json...

Thread-15::DEBUG::2014-08-17
19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids:
(u'2ed32e34-2483-4262-a57b-72bf70f3e989',)

Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs:
{2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}

Thread-1489::INFO::2014-08-17
19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect:
connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}]}

Thread-1489::DEBUG::2014-08-17
19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist':
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing ->
state finished

Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::993::Storage.TaskManager.Task::(_decref)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False

Thread-1489::DEBUG::2014-08-17
19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state
preparing

Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect:
repoStats(options=None)

Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989':
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck':
'6.3', 'valid': False}}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing ->
state finished

Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}

Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}

Thread-1489::DEBUG::2014-08-17
19:39:21,022::task::993::Storage.TaskManager.Task::(_decref)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False

Thread-36::DEBUG::2014-08-17
19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)

Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method

Thread-29::DEBUG::2014-08-17
19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this
will take up to 30 seconds

Thread-29::DEBUG::2014-08-17
19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)

Thread-29::DEBUG::2014-08-17
19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last
result

Thread-29::DEBUG::2014-08-17
19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n
/sbin/multipath (cwd None)

Thread-29::DEBUG::2014-08-17
19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> =
''; <rc> = 0

Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' got the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' released the operation mutex

Thread-29::DEBUG::2014-08-17
19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last
result

Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3643) Building response

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response
sized 168

Thread-29::DEBUG::2014-08-17
19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3644) Building response

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Sending response sized 86

Thread-29::ERROR::2014-08-17
19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain

    self.domain.selftest()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Thread-36::DEBUG::2014-08-17
19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-36::DEBUG::2014-08-17
19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)

Thread-36::DEBUG::2014-08-17
19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1

Thread-36::WARNING::2014-08-17
19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds

Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...

Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request
information...

Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding
callback 'glob'...

Thread-15::DEBUG::2014-08-17
19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request
for method 'glob'

Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-29::DEBUG::2014-08-17
19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message...

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request...

Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next
request...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding
callback 'touch'...

Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request
for method 'touch'

Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing
response

Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json...

Thread-29::ERROR::2014-08-17
19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain

    self.domain.selftest()

  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

    return getattr(self.getRealDomain(), attrName)

  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain

    return self._cache._realProduce(self._sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce

    domain = self._findDomain(sdUUID)

  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain

    dom = findMethod(sdUUID)

  File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain

    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))

  File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__

    validateFileSystemFeatures(sdUUID, self.mountpoint)

  File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures

    oop.getProcessPool(sdUUID).directTouch(testFilePath)

  File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch

    ioproc.touch(path, flags, mode)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch

    self.timeout)

  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand

    raise OSError(errcode, errstr)

OSError: [Errno 5] Input/output error

Any help is getting this sorted is appreciated.

Chris

------=_NextPart_000_0E08_01CFBC00.B56B8C60
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 14 =
(filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue =
vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span =
style=3D'color:#1F497D'>Still not able to correct this behavior. =
 Has anyone else seen this?<o:p></o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p> </o:p></span></p><div><p =
class=3DMsoNormal><span =
style=3D'color:#1F497D'>Chris<o:p></o:p></span></p></div><p =
class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p> </o:p></span></p><div><div =
style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
0in 0in'><p class=3DMsoNormal><b><span =
style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>=
</b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> =
Chris @ VeeroTech.net [mailto:Chris@VeeroTech.net] <br><b>Sent:</b> =
Sunday, August 17, 2014 7:40 PM<br><b>To:</b> =
users@ovirt.org<br><b>Subject:</b> Master Storage goes Inactive - Failed =
to activate Storage Domain<o:p></o:p></span></p></div></div><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Hello =
All,<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>I am having an issue on my test setup, the master =
storage goes inactive.  It is set up as to use Gluster and the =
volume is up and running without issues.  Here is a snippet of =
/var/log/ovirt-engine/engine.log (these are just the =
ERRORs):<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>2014-08-18 01:47:04,173 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:04,174 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:14,286 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:14,287 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-68) [357af14e] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:24,400 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:24,400 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-51) [11269c17] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:34,514 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:34,515 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-59) [40622d60] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:44,651 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:44,652 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName =
=3D u5.veerotech.net, HostId =3D 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, =
storagePoolId =3D 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution =
failed. Exception: VDSErrorException: VDSGenericException: =
VDSErrorException: Failed to SpmStatusVDS, error =3D [Errno 5] =
Input/output error, code =3D 100<o:p></o:p></p><p =
class=3DMsoNormal>2014-08-18 01:47:54,765 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:47:54,766 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-73) [44b5af81] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:04,881 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:04,883 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command =
SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:14,994 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:14,995 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:25,117 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:25,118 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-35) [48ff78af] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:35,231 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:35,232 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-16) [76fe9857] Command =
SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:45,344 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:45,345 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-69) [61356e24] Command =
SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:55,460 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:48:55,461 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-28) [38c2326b] Command =
SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:49:05,574 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS =
method<o:p></o:p></p><p class=3DMsoNormal>2014-08-18 01:49:05,575 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] =
(DefaultQuartzScheduler_Worker-87) [3c65466] Command =
SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: =
VDSErrorException: VDSGenericException: VDSErrorException: Failed to =
SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D =
100<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>Here is the vdsm.log on one of the =
hosts:<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal><span lang=3DFR-CA>[root@u5 ~]# tail -f =
/var/log/vdsm/vdsm.log<o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DFR-CA>    =
</span>validateFileSystemFeatures(sdUUID, =
self.mountpoint)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 86, in =
validateFileSystemFeatures<o:p></o:p></p><p =
class=3DMsoNormal>    =
oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/outOfProcess.py", line 320, in =
directTouch<o:p></o:p></p><p class=3DMsoNormal>    =
ioproc.touch(path, flags, mode)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
460, in touch<o:p></o:p></p><p class=3DMsoNormal>    =
self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
344, in _sendCommand<o:p></o:p></p><p =
class=3DMsoNormal>    raise OSError(errcode, =
errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] =
Input/output error<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client =
[74.63.225.147]<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init =
-> state preparing<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: =
getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', =
options=3DNone)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is =
118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling =
message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding =
callback 'glob'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got =
request for method 'glob'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) =
Building response<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response =
sized 168<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading =
domain in path =
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70=
f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is =
147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling =
message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding =
callback 'touch'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got =
request for method 'touch'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is =
147DEBUG|(3641) Building response<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response =
sized 86<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::ERROR::2014-08-17 =
19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected =
error<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent call =
last):<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/task.py", line 873, in =
_run<o:p></o:p></p><p class=3DMsoNormal>    return =
fn(*args, **kargs)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/logUtils.py", line 45, in =
wrapper<o:p></o:p></p><p class=3DMsoNormal>    res =3D =
f(*args, **kwargs)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/hsm.py", line 611, in =
getSpmStatus<o:p></o:p></p><p class=3DMsoNormal>    =
status =3D self._getSpmStatusInfo(pool)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/hsm.py", line 605, in =
_getSpmStatusInfo<o:p></o:p></p><p class=3DMsoNormal>    =
(pool.spmRole,) + pool.getSpmStatus()))<o:p></o:p></p><p =
class=3DMsoNormal>  File "/usr/share/vdsm/storage/sp.py", =
line 126, in getSpmStatus<o:p></o:p></p><p =
class=3DMsoNormal>    return =
self._backend.getSpmStatus()<o:p></o:p></p><p class=3DMsoNormal>  =
File "/usr/share/vdsm/storage/spbackends.py", line 416, in =
getSpmStatus<o:p></o:p></p><p class=3DMsoNormal>    lVer, =
spmId =3D self.masterDomain.inquireClusterLock()<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 49, in =
__getattr__<o:p></o:p></p><p class=3DMsoNormal>    return =
getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 52, in =
getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    =
return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 122, in =
_realProduce<o:p></o:p></p><p class=3DMsoNormal>    =
domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 141, in =
_findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom =
=3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/glusterSD.py", line 32, in =
findDomain<o:p></o:p></p><p class=3DMsoNormal>    return =
GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o=
:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 157, in =
__init__<o:p></o:p></p><p class=3DMsoNormal>    =
validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 86, in =
validateFileSystemFeatures<o:p></o:p></p><p =
class=3DMsoNormal>    =
oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/outOfProcess.py", line 320, in =
directTouch<o:p></o:p></p><p class=3DMsoNormal>    =
ioproc.touch(path, flags, mode)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
460, in touch<o:p></o:p></p><p class=3DMsoNormal>    =
self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
344, in _sendCommand<o:p></o:p></p><p =
class=3DMsoNormal>    raise OSError(errcode, =
errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] =
Input/output error<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,955::task::885::Storage.TaskManager.Task::(_run) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: =
ada614f8-75c5-4728-9083-340f918a4417 =
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping =
task<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state =
preparing (force False)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting =
True<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is =
aborted: u'[Errno 5] Input/output error' - code 100<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno =
5] Input/output error<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting =
True<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force =
False<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state =
preparing -> state aborting<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy =
none<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state =
aborting -> state failed<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::ERROR::2014-08-17 =
19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] =
Input/output error<o:p></o:p></p><p class=3DMsoNormal>Traceback (most =
recent call last):<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/dispatcher.py", line 71, in =
wrapper<o:p></o:p></p><p class=3DMsoNormal>    result =3D =
ctask.prepare(func, *args, **kwargs)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/task.py", line 103, in =
wrapper<o:p></o:p></p><p class=3DMsoNormal>    return =
m(self, *a, **kw)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/task.py", line 1179, in =
prepare<o:p></o:p></p><p class=3DMsoNormal>    raise =
self.error<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] =
Input/output error<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] =
flowID [fedc88e]<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init =
-> state preparing<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: =
connectStorageServer(domType=3D7, =
spUUID=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'port': '', =
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', =
'vfs_type': 'glusterfs', 'password': '******', 'id': =
'08128803-0141-46aa-835a-2f9734ef0d3b'}], =
options=3DNone)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) =
glusterDomPath: glusterSD/*<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUG|Receiving request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is =
118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUG|Marshaling message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding =
callback 'glob'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got =
request for method 'glob'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUG|(3642) Building response<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Generating json...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response =
sized 168<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD =
uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: =
{2ed32e34-2483-4262-a57b-72bf70f3e989: =
storage.glusterSD.findDomain}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: =
connectStorageServer, Return response: {'statuslist': [{'status': 0, =
'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) =
Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': =
[{'status': 0, 'id': =
'08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state =
preparing -> state finished<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting =
False<o:p></o:p></p><p class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init =
-> state preparing<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: =
repoStats(options=3DNone)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::INFO::2014-08-17 =
19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: =
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': =
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', =
'lastCheck': '6.3', 'valid': False}}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) =
Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: =
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, =
'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': =
False}}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state =
preparing -> state finished<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}<o:p></o:p></p><p =
class=3DMsoNormal>Thread-1489::DEBUG::2014-08-17 =
19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting =
False<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) =
/usr/bin/sudo -n /sbin/multipath (cwd None)<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: =
<err> =3D ''; <rc> =3D 0<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) =
Operation 'lvm invalidate operation' got the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) =
Operation 'lvm invalidate operation' released the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) =
Operation 'lvm invalidate operation' got the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) =
Operation 'lvm invalidate operation' released the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) =
Operation 'lvm invalidate operation' got the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) =
Operation 'lvm invalidate operation' released the operation =
mutex<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is =
118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Marshaling message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding =
callback 'glob'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got =
request for method 'glob'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|(3643) Building response<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response =
sized 168<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading =
domain in path =
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70=
f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is =
147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling =
message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Queuing request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding =
callback 'touch'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got =
request for method 'touch'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|(3644) Building response<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Sending response sized 86<o:p></o:p></p><p =
class=3DMsoNormal>Thread-29::ERROR::2014-08-17 =
19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD=
omain) Error while collecting domain =
2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring =
information<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent =
call last):<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/domainMonitor.py", line 215, in =
_monitorDomain<o:p></o:p></p><p class=3DMsoNormal>    =
self.domain.selftest()<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 49, in =
__getattr__<o:p></o:p></p><p class=3DMsoNormal>    return =
getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 52, in =
getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    =
return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 122, in =
_realProduce<o:p></o:p></p><p class=3DMsoNormal>    =
domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 141, in =
_findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom =
=3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/glusterSD.py", line 32, in =
findDomain<o:p></o:p></p><p class=3DMsoNormal>    return =
GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o=
:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 157, in =
__init__<o:p></o:p></p><p class=3DMsoNormal>    =
validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 86, in =
validateFileSystemFeatures<o:p></o:p></p><p =
class=3DMsoNormal>    =
oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/outOfProcess.py", line 320, in =
directTouch<o:p></o:p></p><p class=3DMsoNormal>    =
ioproc.touch(path, flags, mode)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
460, in touch<o:p></o:p></p><p class=3DMsoNormal>    =
self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
344, in _sendCommand<o:p></o:p></p><p =
class=3DMsoNormal>    raise OSError(errcode, =
errstr)<o:p></o:p></p><p class=3DMsoNormal>OSError: [Errno 5] =
Input/output error<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)<o:p></o:p></p><p class=3DMsoNormal>Thread-36::DEBUG::2014-08-17 =
19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> =3D =
1<o:p></o:p></p><p class=3DMsoNormal>Thread-36::WARNING::2014-08-17 =
19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing =
responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is =
118<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling =
message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting =
request information...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding =
callback 'glob'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got =
request for method 'glob'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-29::DEBUG::2014-08-17 =
19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading =
domain in path =
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70=
f3e989<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is =
147<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling =
message...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding =
callback 'touch'...<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got =
request for method 'touch'<o:p></o:p></p><p =
class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing =
response<o:p></o:p></p><p class=3DMsoNormal>Thread-15::DEBUG::2014-08-17 =
19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating =
json...<o:p></o:p></p><p class=3DMsoNormal>Thread-29::ERROR::2014-08-17 =
19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD=
omain) Error while collecting domain =
2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring =
information<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent =
call last):<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/domainMonitor.py", line 215, in =
_monitorDomain<o:p></o:p></p><p class=3DMsoNormal>    =
self.domain.selftest()<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 49, in =
__getattr__<o:p></o:p></p><p class=3DMsoNormal>    return =
getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 52, in =
getRealDomain<o:p></o:p></p><p class=3DMsoNormal>    =
return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 122, in =
_realProduce<o:p></o:p></p><p class=3DMsoNormal>    =
domain =3D self._findDomain(sdUUID)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/sdc.py", line 141, in =
_findDomain<o:p></o:p></p><p class=3DMsoNormal>    dom =
=3D findMethod(sdUUID)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/glusterSD.py", line 32, in =
findDomain<o:p></o:p></p><p class=3DMsoNormal>    return =
GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o=
:p></p><p class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 157, in =
__init__<o:p></o:p></p><p class=3DMsoNormal>    =
validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/fileSD.py", line 86, in =
validateFileSystemFeatures<o:p></o:p></p><p =
class=3DMsoNormal>    =
oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/share/vdsm/storage/outOfProcess.py", line 320, in =
directTouch<o:p></o:p></p><p class=3DMsoNormal>    =
ioproc.touch(path, flags, mode)<o:p></o:p></p><p =
class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
460, in touch<o:p></o:p></p><p class=3DMsoNormal>    =
self.timeout)<o:p></o:p></p><p class=3DMsoNormal>  File =
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line =
344, in _sendCommand<o:p></o:p></p><p =
class=3DMsoNormal>    <span lang=3DFR-CA>raise =
OSError(errcode, errstr)<o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DFR-CA>OSError: [Errno 5] Input/output =
error</span><o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Any help is =
getting this sorted is appreciated.<o:p></o:p></p><p =
class=3DMsoNormal>Chris<o:p></o:p></p></div></body></html>
------=_NextPart_000_0E08_01CFBC00.B56B8C60--

Re: [ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

Chris ＠ VeeroTech.net

tags

participants (1)