Re: [ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

This is a multipart message in MIME format. ------=_NextPart_000_0097_01CFBF84.28615750 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable See answers below. The only log requested that changed when trying to = activate is the vdsm.log, it is attached. -----Original Message----- From: Nir Soffer [mailto:nsoffer@redhat.com]=20 Sent: August 24, 2014 6:15 AM To: Chris @ VeeroTech.net Cc: users(a)ovirt.org Subject: Re: [ovirt-users] Master Storage goes Inactive - Failed to = activate Storage Domain ----- Original Message ----- activate Storage Domain Testing on another host only prove that your gluster service is = accessible from the other host. Please try to access gluster from the = host that vdsm is running on. I have tested and this functions as it should. Having only two hosts with gluster replication may be unreliable. Please = consult gluster folks how to setup gluster in a reliable way. I am only testing Ovirt with Gluster, this is not a production system. = I have other servers lined up should the testing prove successful. Please share with us these logs, for the timeframe of this issue: /var/log/messages /var/log/sanlock.log /var/log/vdsm/vdsm.log activate ------=_NextPart_000_0097_01CFBF84.28615750 Content-Type: text/plain; name="vdsm_log_u5.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="vdsm_log_u5.txt" [root@u5 ~]# tail -f /var/log/vdsm/vdsm.log return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterTasksList File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-349::DEBUG::2014-08-24 = 10:06:19,055::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:19,103::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000232199 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:19,103::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:21,105::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:21,152::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000259325 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:21,153::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:23,155::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:23,203::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000237248 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:23,203::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-13::DEBUG::2014-08-24 = 10:06:24,829::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state init -> = state preparing Thread-13::INFO::2014-08-24 = 10:06:24,830::logUtils::44::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone) Thread-13::INFO::2014-08-24 = 10:06:24,830::logUtils::47::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': = {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', = 'lastCheck': '7.8', 'valid': False}} Thread-13::DEBUG::2014-08-24 = 10:06:24,830::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::finished: = {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, = 'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}} Thread-13::DEBUG::2014-08-24 = 10:06:24,830::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state = preparing -> state finished Thread-13::DEBUG::2014-08-24 = 10:06:24,831::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {} Thread-13::DEBUG::2014-08-24 = 10:06:24,831::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::DEBUG::2014-08-24 = 10:06:24,831::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::ref 0 aborting False Thread-349::DEBUG::2014-08-24 = 10:06:25,205::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:25,253::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000253969 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:25,253::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-340::DEBUG::2014-08-24 = 10:06:26,992::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage) Thread-340::DEBUG::2014-08-24 = 10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:26,993::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan) Thread-340::DEBUG::2014-08-24 = 10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:26,993::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds Thread-340::DEBUG::2014-08-24 = 10:06:26,994::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None) Thread-340::DEBUG::2014-08-24 = 10:06:27,042::misc::751::Storage.SamplingMethod::(__call__) Returning = last result Thread-340::ERROR::2014-08-24 = 10:06:27,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce self.refreshStorage() File "/usr/share/vdsm/storage/misc.py", line 760, in helper return sm(*args, **kwargs) File "/usr/share/vdsm/storage/misc.py", line 745, in __call__ self.__lastResult =3D self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan supervdsm.getProxy().hbaRescan() File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in hbaRescan File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-349::DEBUG::2014-08-24 = 10:06:27,255::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:27,302::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000244622 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:27,303::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:29,305::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:29,352::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000261808 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:29,352::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:31,355::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:31,402::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000251987 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:31,402::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:33,405::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:33,453::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000202061 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:33,454::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:35,456::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:35,503::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000280783 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:35,503::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-340::DEBUG::2014-08-24 = 10:06:37,058::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage) Thread-340::DEBUG::2014-08-24 = 10:06:37,059::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:37,060::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan) Thread-340::DEBUG::2014-08-24 = 10:06:37,060::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:37,060::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds Thread-340::DEBUG::2014-08-24 = 10:06:37,061::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None) Thread-340::DEBUG::2014-08-24 = 10:06:37,114::misc::751::Storage.SamplingMethod::(__call__) Returning = last result Thread-340::ERROR::2014-08-24 = 10:06:37,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce self.refreshStorage() File "/usr/share/vdsm/storage/misc.py", line 760, in helper return sm(*args, **kwargs) File "/usr/share/vdsm/storage/misc.py", line 745, in __call__ self.__lastResult =3D self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan supervdsm.getProxy().hbaRescan() File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in hbaRescan File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-349::DEBUG::2014-08-24 = 10:06:37,506::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:37,556::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000253556 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:37,556::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:39,559::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:39,606::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000225893 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:39,606::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-13::DEBUG::2014-08-24 = 10:06:40,511::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state init -> = state preparing Thread-13::INFO::2014-08-24 = 10:06:40,511::logUtils::44::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone) Thread-13::INFO::2014-08-24 = 10:06:40,512::logUtils::47::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': = {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', = 'lastCheck': '3.4', 'valid': False}} Thread-13::DEBUG::2014-08-24 = 10:06:40,512::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::finished: = {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, = 'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}} Thread-13::DEBUG::2014-08-24 = 10:06:40,512::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state = preparing -> state finished Thread-13::DEBUG::2014-08-24 = 10:06:40,512::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {} Thread-13::DEBUG::2014-08-24 = 10:06:40,512::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::DEBUG::2014-08-24 = 10:06:40,512::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::ref 0 aborting False Thread-349::DEBUG::2014-08-24 = 10:06:41,609::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:41,655::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000266326 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:41,656::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:43,658::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:43,705::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000228889 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:43,706::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-13::DEBUG::2014-08-24 = 10:06:45,155::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] = flowID [5f151d27] Thread-13::DEBUG::2014-08-24 = 10:06:45,155::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state init -> = state preparing Thread-13::INFO::2014-08-24 = 10:06:45,155::logUtils::44::dispatcher::(wrapper) Run and protect: = getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', = options=3DNone) Thread-13::DEBUG::2014-08-24 = 10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage) Thread-13::DEBUG::2014-08-24 = 10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-13::DEBUG::2014-08-24 = 10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan) Thread-13::DEBUG::2014-08-24 = 10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-13::DEBUG::2014-08-24 = 10:06:45,156::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds Thread-13::DEBUG::2014-08-24 = 10:06:45,157::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None) Thread-13::DEBUG::2014-08-24 = 10:06:45,204::misc::751::Storage.SamplingMethod::(__call__) Returning = last result Thread-13::ERROR::2014-08-24 = 10:06:45,205::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based = metadata Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus status =3D self._getSpmStatusInfo(pool) File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus return self._backend.getSpmStatus() File "/usr/share/vdsm/storage/spbackends.py", line 416, in = getSpmStatus lVer, spmId =3D self.masterDomain.inquireClusterLock() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce self.refreshStorage() File "/usr/share/vdsm/storage/misc.py", line 760, in helper return sm(*args, **kwargs) File "/usr/share/vdsm/storage/misc.py", line 745, in __call__ self.__lastResult =3D self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan supervdsm.getProxy().hbaRescan() File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in hbaRescan File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-13::ERROR::2014-08-24 = 10:06:45,206::task::866::Storage.TaskManager.Task::(_setError) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res =3D f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus raise se.StorageDomainMasterError("Can't read LV based metadata") StorageDomainMasterError: Error validating master storage domain: = ("Can't read LV based metadata",) Thread-13::DEBUG::2014-08-24 = 10:06:45,206::task::885::Storage.TaskManager.Task::(_run) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._run: = 5e508d32-1d1f-4b8f-96be-7c8435476763 = ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task Thread-13::DEBUG::2014-08-24 = 10:06:45,206::task::1217::Storage.TaskManager.Task::(stop) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::stopping in state = preparing (force False) Thread-13::DEBUG::2014-08-24 = 10:06:45,206::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 1 aborting True Thread-13::INFO::2014-08-24 = 10:06:45,206::task::1171::Storage.TaskManager.Task::(prepare) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::aborting: Task is = aborted: 'Error validating master storage domain' - code 355 Thread-13::DEBUG::2014-08-24 = 10:06:45,206::task::1176::Storage.TaskManager.Task::(prepare) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Prepare: aborted: Error = validating master storage domain Thread-13::DEBUG::2014-08-24 = 10:06:45,207::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 0 aborting True Thread-13::DEBUG::2014-08-24 = 10:06:45,207::task::928::Storage.TaskManager.Task::(_doAbort) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._doAbort: force = False Thread-13::DEBUG::2014-08-24 = 10:06:45,207::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::DEBUG::2014-08-24 = 10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state = preparing -> state aborting Thread-13::DEBUG::2014-08-24 = 10:06:45,207::task::550::Storage.TaskManager.Task::(__state_aborting) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::_aborting: recover policy = none Thread-13::DEBUG::2014-08-24 = 10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state = aborting -> state failed Thread-13::DEBUG::2014-08-24 = 10:06:45,208::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {} Thread-13::DEBUG::2014-08-24 = 10:06:45,208::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::ERROR::2014-08-24 = 10:06:45,208::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': = {'message': 'Error validating master storage domain: ("Can\'t read LV = based metadata",)', 'code': 355}} Thread-349::DEBUG::2014-08-24 = 10:06:45,708::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:45,756::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000255964 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:45,756::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-13::DEBUG::2014-08-24 = 10:06:45,883::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] = flowID [852653f] Thread-13::DEBUG::2014-08-24 = 10:06:45,883::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state init -> = state preparing Thread-13::INFO::2014-08-24 = 10:06:45,884::logUtils::44::dispatcher::(wrapper) Run and protect: = connectStorageServer(domType=3D7, = spUUID=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'port': '', = 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', = 'vfs_type': 'glusterfs', 'password': '******', 'id': = '08128803-0141-46aa-835a-2f9734ef0d3b'}], options=3DNone) Thread-13::DEBUG::2014-08-24 = 10:06:45,889::hsm::2378::Storage.HSM::(__prefetchDomains) = glusterDomPath: glusterSD/* Thread-15::DEBUG::2014-08-24 = 10:06:45,890::__init__::225::IOProcess::(_processLogs) Marshaling = message...DEBUGDEBUGDEBUGDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Message size is = 119 Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling = message... Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling = message...DEBUGDEBUGDEBUGDEBUGDEBUG|Queuing request... Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Waiting for next = request... Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool... Thread-15::DEBUG::2014-08-24 = 10:06:45,891::__init__::225::IOProcess::(_processLogs) Extracting = request information... Thread-15::DEBUG::2014-08-24 = 10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Finding = callback 'glob'... Thread-15::DEBUG::2014-08-24 = 10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Got = request for method 'glob' Thread-15::DEBUG::2014-08-24 = 10:06:45,893::__init__::225::IOProcess::(_processLogs) Marshaling = message...DEBUGDEBUGDEBUGDEBUGDEBUG|(89757) Building response Thread-15::DEBUG::2014-08-24 = 10:06:45,893::__init__::225::IOProcess::(_processLogs) (89757) Queuing = response Thread-15::DEBUG::2014-08-24 = 10:06:45,893::__init__::225::IOProcess::(_processLogs) Generating = json... Thread-15::DEBUG::2014-08-24 = 10:06:45,894::__init__::225::IOProcess::(_processLogs) Sending response = sized 169 Thread-13::DEBUG::2014-08-24 = 10:06:45,895::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD = uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',) Thread-13::DEBUG::2014-08-24 = 10:06:45,895::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: = {2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain} Thread-13::INFO::2014-08-24 = 10:06:45,895::logUtils::47::dispatcher::(wrapper) Run and protect: = connectStorageServer, Return response: {'statuslist': [{'status': 0, = 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-13::DEBUG::2014-08-24 = 10:06:45,896::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::finished: {'statuslist': = [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-13::DEBUG::2014-08-24 = 10:06:45,896::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state = preparing -> state finished Thread-13::DEBUG::2014-08-24 = 10:06:45,896::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {} Thread-13::DEBUG::2014-08-24 = 10:06:45,896::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::DEBUG::2014-08-24 = 10:06:45,896::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::ref 0 aborting False Thread-13::DEBUG::2014-08-24 = 10:06:45,986::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] = flowID [7b161bce] Thread-13::DEBUG::2014-08-24 = 10:06:45,987::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state init -> = state preparing Thread-13::INFO::2014-08-24 = 10:06:45,987::logUtils::44::dispatcher::(wrapper) Run and protect: = getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', = options=3DNone) Thread-13::DEBUG::2014-08-24 = 10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage) Thread-13::DEBUG::2014-08-24 = 10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-13::DEBUG::2014-08-24 = 10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan) Thread-13::DEBUG::2014-08-24 = 10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-13::DEBUG::2014-08-24 = 10:06:45,988::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds Thread-13::DEBUG::2014-08-24 = 10:06:45,989::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None) Thread-13::DEBUG::2014-08-24 = 10:06:46,038::misc::751::Storage.SamplingMethod::(__call__) Returning = last result Thread-13::ERROR::2014-08-24 = 10:06:46,039::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based = metadata Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus status =3D self._getSpmStatusInfo(pool) File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus return self._backend.getSpmStatus() File "/usr/share/vdsm/storage/spbackends.py", line 416, in = getSpmStatus lVer, spmId =3D self.masterDomain.inquireClusterLock() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce self.refreshStorage() File "/usr/share/vdsm/storage/misc.py", line 760, in helper return sm(*args, **kwargs) File "/usr/share/vdsm/storage/misc.py", line 745, in __call__ self.__lastResult =3D self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan supervdsm.getProxy().hbaRescan() File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in hbaRescan File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-13::ERROR::2014-08-24 = 10:06:46,039::task::866::Storage.TaskManager.Task::(_setError) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res =3D f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus raise se.StorageDomainMasterError("Can't read LV based metadata") StorageDomainMasterError: Error validating master storage domain: = ("Can't read LV based metadata",) Thread-13::DEBUG::2014-08-24 = 10:06:46,040::task::885::Storage.TaskManager.Task::(_run) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._run: = 59772503-6431-42a9-b6b0-8e7f4b95a7f5 = ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task Thread-13::DEBUG::2014-08-24 = 10:06:46,040::task::1217::Storage.TaskManager.Task::(stop) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::stopping in state = preparing (force False) Thread-13::DEBUG::2014-08-24 = 10:06:46,040::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 1 aborting True Thread-13::INFO::2014-08-24 = 10:06:46,040::task::1171::Storage.TaskManager.Task::(prepare) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::aborting: Task is = aborted: 'Error validating master storage domain' - code 355 Thread-13::DEBUG::2014-08-24 = 10:06:46,040::task::1176::Storage.TaskManager.Task::(prepare) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Prepare: aborted: Error = validating master storage domain Thread-13::DEBUG::2014-08-24 = 10:06:46,041::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 0 aborting True Thread-13::DEBUG::2014-08-24 = 10:06:46,041::task::928::Storage.TaskManager.Task::(_doAbort) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._doAbort: force = False Thread-13::DEBUG::2014-08-24 = 10:06:46,041::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::DEBUG::2014-08-24 = 10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state = preparing -> state aborting Thread-13::DEBUG::2014-08-24 = 10:06:46,041::task::550::Storage.TaskManager.Task::(__state_aborting) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::_aborting: recover policy = none Thread-13::DEBUG::2014-08-24 = 10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state = aborting -> state failed Thread-13::DEBUG::2014-08-24 = 10:06:46,042::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {} Thread-13::DEBUG::2014-08-24 = 10:06:46,042::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {} Thread-13::ERROR::2014-08-24 = 10:06:46,042::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': = {'message': 'Error validating master storage domain: ("Can\'t read LV = based metadata",)', 'code': 355}} Thread-340::DEBUG::2014-08-24 = 10:06:47,118::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage) Thread-340::DEBUG::2014-08-24 = 10:06:47,118::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:47,119::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan) Thread-340::DEBUG::2014-08-24 = 10:06:47,119::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method Thread-340::DEBUG::2014-08-24 = 10:06:47,119::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds Thread-340::DEBUG::2014-08-24 = 10:06:47,119::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None) Thread-340::DEBUG::2014-08-24 = 10:06:47,168::misc::751::Storage.SamplingMethod::(__call__) Returning = last result Thread-340::ERROR::2014-08-24 = 10:06:47,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce self.refreshStorage() File "/usr/share/vdsm/storage/misc.py", line 760, in helper return sm(*args, **kwargs) File "/usr/share/vdsm/storage/misc.py", line 745, in __call__ self.__lastResult =3D self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan supervdsm.getProxy().hbaRescan() File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in hbaRescan File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in = _callmethod conn.send((self._id, methodname, args, kwds)) IOError: [Errno 32] Broken pipe Thread-349::DEBUG::2014-08-24 = 10:06:47,759::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:47,807::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000229862 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:47,808::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:49,810::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:49,857::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000265654 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:49,857::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:51,859::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:51,907::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000233321 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:51,907::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds Thread-349::DEBUG::2014-08-24 = 10:06:53,909::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None) Thread-349::DEBUG::2014-08-24 = 10:06:53,957::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000235232 s, 0.0 kB/s\n"; <rc> =3D 1 Thread-349::WARNING::2014-08-24 = 10:06:53,958::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds ------=_NextPart_000_0097_01CFBF84.28615750--