[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterTasksList
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 10:06:19,055::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:19,103::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000232199 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:19,103::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:21,105::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:21,152::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000259325 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:21,153::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:23,155::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:23,203::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000237248 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:23,203::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:24,829::task::595::Storage.TaskManager.Task::(_updateState) Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state init -> state preparing
Thread-13::INFO::2014-08-24 10:06:24,830::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-08-24 10:06:24,830::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24 10:06:24,830::task::1191::Storage.TaskManager.Task::(prepare) Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::finished: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24 10:06:24,830::task::595::Storage.TaskManager.Task::(_updateState) Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-08-24 10:06:24,831::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 10:06:24,831::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 10:06:24,831::task::993::Storage.TaskManager.Task::(_decref) Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24 10:06:25,205::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:25,253::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000253969 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:25,253::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-340::DEBUG::2014-08-24 10:06:26,992::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:26,993::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:26,993::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 10:06:26,994::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 10:06:27,042::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-340::ERROR::2014-08-24 10:06:27,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 10:06:27,255::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:27,302::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000244622 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:27,303::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:29,305::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:29,352::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000261808 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:29,352::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:31,355::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:31,402::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000251987 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:31,402::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:33,405::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:33,453::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000202061 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:33,454::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:35,456::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:35,503::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000280783 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:35,503::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-340::DEBUG::2014-08-24 10:06:37,058::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 10:06:37,059::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:37,060::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 10:06:37,060::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:37,060::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 10:06:37,061::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 10:06:37,114::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-340::ERROR::2014-08-24 10:06:37,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 10:06:37,506::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:37,556::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000253556 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:37,556::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:39,559::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:39,606::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000225893 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:39,606::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:40,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state init -> state preparing
Thread-13::INFO::2014-08-24 10:06:40,511::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-08-24 10:06:40,512::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24 10:06:40,512::task::1191::Storage.TaskManager.Task::(prepare) Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::finished: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24 10:06:40,512::task::595::Storage.TaskManager.Task::(_updateState) Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-08-24 10:06:40,512::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 10:06:40,512::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 10:06:40,512::task::993::Storage.TaskManager.Task::(_decref) Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24 10:06:41,609::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:41,655::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000266326 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:41,656::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:43,658::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:43,705::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000228889 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:43,706::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:45,155::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [5f151d27]
Thread-13::DEBUG::2014-08-24 10:06:45,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state init -> state preparing
Thread-13::INFO::2014-08-24 10:06:45,155::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)
Thread-13::DEBUG::2014-08-24 10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24 10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-08-24 10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24 10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-08-24 10:06:45,156::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24 10:06:45,157::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24 10:06:45,204::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-13::ERROR::2014-08-24 10:06:45,205::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based metadata
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
    status = self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus
    lVer, spmId = self.masterDomain.inquireClusterLock()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24 10:06:45,206::task::866::Storage.TaskManager.Task::(_setError) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
    raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: ("Can't read LV based metadata",)
Thread-13::DEBUG::2014-08-24 10:06:45,206::task::885::Storage.TaskManager.Task::(_run) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._run: 5e508d32-1d1f-4b8f-96be-7c8435476763 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task
Thread-13::DEBUG::2014-08-24 10:06:45,206::task::1217::Storage.TaskManager.Task::(stop) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-08-24 10:06:45,206::task::993::Storage.TaskManager.Task::(_decref) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 1 aborting True
Thread-13::INFO::2014-08-24 10:06:45,206::task::1171::Storage.TaskManager.Task::(prepare) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::aborting: Task is aborted: 'Error validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24 10:06:45,206::task::1176::Storage.TaskManager.Task::(prepare) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Prepare: aborted: Error validating master storage domain
Thread-13::DEBUG::2014-08-24 10:06:45,207::task::993::Storage.TaskManager.Task::(_decref) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24 10:06:45,207::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._doAbort: force False
Thread-13::DEBUG::2014-08-24 10:06:45,207::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-08-24 10:06:45,207::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::_aborting: recover policy none
Thread-13::DEBUG::2014-08-24 10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-08-24 10:06:45,208::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 10:06:45,208::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24 10:06:45,208::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Error validating master storage domain: ("Can\'t read LV based metadata",)', 'code': 355}}
Thread-349::DEBUG::2014-08-24 10:06:45,708::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:45,756::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000255964 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:45,756::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:45,883::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [852653f]
Thread-13::DEBUG::2014-08-24 10:06:45,883::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state init -> state preparing
Thread-13::INFO::2014-08-24 10:06:45,884::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type': 'glusterfs', 'password': '******', 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)
Thread-13::DEBUG::2014-08-24 10:06:45,889::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/*
Thread-15::DEBUG::2014-08-24 10:06:45,890::__init__::225::IOProcess::(_processLogs) Marshaling message...DEBUGDEBUGDEBUGDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Message size is 119
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling message...DEBUGDEBUGDEBUGDEBUGDEBUG|Queuing request...
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Waiting for next request...
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...
Thread-15::DEBUG::2014-08-24 10:06:45,891::__init__::225::IOProcess::(_processLogs) Extracting request information...
Thread-15::DEBUG::2014-08-24 10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Finding callback 'glob'...
Thread-15::DEBUG::2014-08-24 10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Got request for method 'glob'
Thread-15::DEBUG::2014-08-24 10:06:45,893::__init__::225::IOProcess::(_processLogs) Marshaling message...DEBUGDEBUGDEBUGDEBUGDEBUG|(89757) Building response
Thread-15::DEBUG::2014-08-24 10:06:45,893::__init__::225::IOProcess::(_processLogs) (89757) Queuing response
Thread-15::DEBUG::2014-08-24 10:06:45,893::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-15::DEBUG::2014-08-24 10:06:45,894::__init__::225::IOProcess::(_processLogs) Sending response sized 169
Thread-13::DEBUG::2014-08-24 10:06:45,895::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)
Thread-13::DEBUG::2014-08-24 10:06:45,895::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: {2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}
Thread-13::INFO::2014-08-24 10:06:45,895::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24 10:06:45,896::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::finished: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24 10:06:45,896::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-08-24 10:06:45,896::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 10:06:45,896::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 10:06:45,896::task::993::Storage.TaskManager.Task::(_decref) Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::ref 0 aborting False
Thread-13::DEBUG::2014-08-24 10:06:45,986::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [7b161bce]
Thread-13::DEBUG::2014-08-24 10:06:45,987::task::595::Storage.TaskManager.Task::(_updateState) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state init -> state preparing
Thread-13::INFO::2014-08-24 10:06:45,987::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)
Thread-13::DEBUG::2014-08-24 10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24 10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-08-24 10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24 10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-08-24 10:06:45,988::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24 10:06:45,989::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24 10:06:46,038::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-13::ERROR::2014-08-24 10:06:46,039::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based metadata
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
    status = self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus
    lVer, spmId = self.masterDomain.inquireClusterLock()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24 10:06:46,039::task::866::Storage.TaskManager.Task::(_setError) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
    raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: ("Can't read LV based metadata",)
Thread-13::DEBUG::2014-08-24 10:06:46,040::task::885::Storage.TaskManager.Task::(_run) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._run: 59772503-6431-42a9-b6b0-8e7f4b95a7f5 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task
Thread-13::DEBUG::2014-08-24 10:06:46,040::task::1217::Storage.TaskManager.Task::(stop) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-08-24 10:06:46,040::task::993::Storage.TaskManager.Task::(_decref) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 1 aborting True
Thread-13::INFO::2014-08-24 10:06:46,040::task::1171::Storage.TaskManager.Task::(prepare) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::aborting: Task is aborted: 'Error validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24 10:06:46,040::task::1176::Storage.TaskManager.Task::(prepare) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Prepare: aborted: Error validating master storage domain
Thread-13::DEBUG::2014-08-24 10:06:46,041::task::993::Storage.TaskManager.Task::(_decref) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24 10:06:46,041::task::928::Storage.TaskManager.Task::(_doAbort) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._doAbort: force False
Thread-13::DEBUG::2014-08-24 10:06:46,041::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-08-24 10:06:46,041::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::_aborting: recover policy none
Thread-13::DEBUG::2014-08-24 10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-08-24 10:06:46,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 10:06:46,042::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24 10:06:46,042::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Error validating master storage domain: ("Can\'t read LV based metadata",)', 'code': 355}}
Thread-340::DEBUG::2014-08-24 10:06:47,118::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 10:06:47,118::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:47,119::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 10:06:47,119::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-340::DEBUG::2014-08-24 10:06:47,119::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 10:06:47,119::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 10:06:47,168::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-340::ERROR::2014-08-24 10:06:47,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 10:06:47,759::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:47,807::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000229862 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:47,808::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:49,810::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:49,857::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000265654 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:49,857::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:51,859::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:51,907::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000233321 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:51,907::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 10:06:53,909::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24 10:06:53,957::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000235232 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24 10:06:53,958::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds