Re: [ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

24 Aug 2014

      This is a multipart message in MIME format.

------=_NextPart_000_0097_01CFBF84.28615750
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

See answers below.  The only log requested that changed when trying to =
activate is the vdsm.log, it is attached.

-----Original Message-----
From: Nir Soffer [mailto:nsoffer@redhat.com]=20
Sent: August 24, 2014 6:15 AM
To: Chris @ VeeroTech.net
Cc: users@ovirt.org
Subject: Re: [ovirt-users] Master Storage goes Inactive - Failed to =
activate Storage Domain

----- Original Message -----
...
From: "Chris @ VeeroTech.net" <Chris@VeeroTech.net>
To: "Nir Soffer" <nsoffer@redhat.com>
Cc: users@ovirt.org
Sent: Friday, August 22, 2014 4:22:46 AM
Subject: RE: [ovirt-users] Master Storage goes Inactive - Failed to =
activate	Storage Domain
=20
I have verified that the storage service (gluster) is functioning=20
correctly by mounting it on another server and writing/reading.
Testing on another host only prove that your gluster service is =
accessible from the other host. Please try to access gluster from the =
host that vdsm is running on.

I have tested and this functions as it should.
...
The setup is basic,
two identical servers set up as hosts, four drives on each server. =20
First drive is currently for the OS, second and third drive are set up =
...
to use Gluster distributed replicate.  Fourth is not used.
Having only two hosts with gluster replication may be unreliable. Please =
consult gluster folks how to setup gluster in a reliable way.

I am only testing Ovirt with Gluster, this is not a production system.  =
I have other servers lined up should the testing prove successful.
...
Please let me know what
other information you require, and thank you for the help.
Please share with us these logs, for the timeframe of this issue:

/var/log/messages
/var/log/sanlock.log
/var/log/vdsm/vdsm.log
...
=20
Chris
=20
=20
-----Original Message-----
From: Nir Soffer [mailto:nsoffer@redhat.com]
Sent: Thursday, August 21, 2014 6:54 PM
To: Chris @ VeeroTech.net
Cc: users@ovirt.org
Subject: Re: [ovirt-users] Master Storage goes Inactive - Failed to=20
activate Storage Domain
=20
----- Original Message -----
...
From: "Chris @ VeeroTech.net" <Chris@VeeroTech.net>
To: users@ovirt.org
Sent: Monday, August 18, 2014 2:40:22 AM
Subject: [ovirt-users] Master Storage goes Inactive - Failed to =
activate
  Storage Domain
=20
=20
=20
Hello All,
=20
=20
=20
I am having an issue on my test setup, the master storage goes=20
inactive. It is set up as to use Gluster and the volume is up and=20
running without issues.
Here is a snippet of /var/log/ovirt-engine/engine.log (these are=20
just the
ERRORs):
=20
...
=20
Here is the vdsm.log on one of the hosts:
=20
=20
=20
[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
=20
...
=20
Thread-36::WARNING::2014-08-17
19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::
(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox,=20
will not accept requests until init succeeds
=20
Your host cannot write to the storage.
=20
We need more information about your setup.
=20
Can you check that the storage service is accessible?
=20
=20
------=_NextPart_000_0097_01CFBF84.28615750
Content-Type: text/plain;
	name="vdsm_log_u5.txt"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="vdsm_log_u5.txt"

[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterTasksList
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 =
10:06:19,055::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:19,103::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000232199 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:19,103::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:21,105::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:21,152::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000259325 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:21,153::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:23,155::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:23,203::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000237248 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:23,203::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 =
10:06:24,829::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state init -> =
state preparing
Thread-13::INFO::2014-08-24 =
10:06:24,830::logUtils::44::dispatcher::(wrapper) Run and protect: =
repoStats(options=3DNone)
Thread-13::INFO::2014-08-24 =
10:06:24,830::logUtils::47::dispatcher::(wrapper) Run and protect: =
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': =
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', =
'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24 =
10:06:24,830::task::1191::Storage.TaskManager.Task::(prepare) =
Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::finished: =
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, =
'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24 =
10:06:24,830::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state =
preparing -> state finished
Thread-13::DEBUG::2014-08-24 =
10:06:24,831::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 =
10:06:24,831::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 =
10:06:24,831::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`3cc3ea32-1a3f-42e0-9482-189437782cfd`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24 =
10:06:25,205::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:25,253::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000253969 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:25,253::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-340::DEBUG::2014-08-24 =
10:06:26,992::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 =
10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:26,993::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 =
10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:26,993::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 =
10:06:26,994::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 =
10:06:27,042::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result
Thread-340::ERROR::2014-08-24 =
10:06:27,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD=
omain) Error while collecting domain =
2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in =
_monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult =3D self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 =
10:06:27,255::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:27,302::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000244622 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:27,303::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:29,305::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:29,352::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000261808 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:29,352::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:31,355::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:31,402::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000251987 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:31,402::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:33,405::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:33,453::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000202061 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:33,454::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:35,456::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:35,503::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000280783 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:35,503::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-340::DEBUG::2014-08-24 =
10:06:37,058::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 =
10:06:37,059::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:37,060::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 =
10:06:37,060::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:37,060::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 =
10:06:37,061::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 =
10:06:37,114::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result
Thread-340::ERROR::2014-08-24 =
10:06:37,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD=
omain) Error while collecting domain =
2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in =
_monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult =3D self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 =
10:06:37,506::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:37,556::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000253556 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:37,556::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:39,559::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:39,606::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000225893 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:39,606::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 =
10:06:40,511::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state init -> =
state preparing
Thread-13::INFO::2014-08-24 =
10:06:40,511::logUtils::44::dispatcher::(wrapper) Run and protect: =
repoStats(options=3DNone)
Thread-13::INFO::2014-08-24 =
10:06:40,512::logUtils::47::dispatcher::(wrapper) Run and protect: =
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': =
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', =
'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24 =
10:06:40,512::task::1191::Storage.TaskManager.Task::(prepare) =
Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::finished: =
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, =
'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24 =
10:06:40,512::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state =
preparing -> state finished
Thread-13::DEBUG::2014-08-24 =
10:06:40,512::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 =
10:06:40,512::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 =
10:06:40,512::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`48a7069a-a873-4931-be41-36b0ddfbfbca`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24 =
10:06:41,609::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:41,655::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000266326 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:41,656::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:43,658::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:43,705::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000228889 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:43,706::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 =
10:06:45,155::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] =
flowID [5f151d27]
Thread-13::DEBUG::2014-08-24 =
10:06:45,155::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state init -> =
state preparing
Thread-13::INFO::2014-08-24 =
10:06:45,155::logUtils::44::dispatcher::(wrapper) Run and protect: =
getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', =
options=3DNone)
Thread-13::DEBUG::2014-08-24 =
10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24 =
10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-13::DEBUG::2014-08-24 =
10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24 =
10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-13::DEBUG::2014-08-24 =
10:06:45,156::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24 =
10:06:45,157::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24 =
10:06:45,204::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result
Thread-13::ERROR::2014-08-24 =
10:06:45,205::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based =
metadata
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
    status =3D self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 416, in =
getSpmStatus
    lVer, spmId =3D self.masterDomain.inquireClusterLock()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult =3D self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24 =
10:06:45,206::task::866::Storage.TaskManager.Task::(_setError) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res =3D f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
    raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: =
("Can't read LV based metadata",)
Thread-13::DEBUG::2014-08-24 =
10:06:45,206::task::885::Storage.TaskManager.Task::(_run) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._run: =
5e508d32-1d1f-4b8f-96be-7c8435476763 =
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task
Thread-13::DEBUG::2014-08-24 =
10:06:45,206::task::1217::Storage.TaskManager.Task::(stop) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::stopping in state =
preparing (force False)
Thread-13::DEBUG::2014-08-24 =
10:06:45,206::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 1 aborting True
Thread-13::INFO::2014-08-24 =
10:06:45,206::task::1171::Storage.TaskManager.Task::(prepare) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::aborting: Task is =
aborted: 'Error validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24 =
10:06:45,206::task::1176::Storage.TaskManager.Task::(prepare) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Prepare: aborted: Error =
validating master storage domain
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::task::928::Storage.TaskManager.Task::(_doAbort) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._doAbort: force =
False
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state =
preparing -> state aborting
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::task::550::Storage.TaskManager.Task::(__state_aborting) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::_aborting: recover policy =
none
Thread-13::DEBUG::2014-08-24 =
10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state =
aborting -> state failed
Thread-13::DEBUG::2014-08-24 =
10:06:45,208::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 =
10:06:45,208::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24 =
10:06:45,208::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': =
{'message': 'Error validating master storage domain: ("Can\'t read LV =
based metadata",)', 'code': 355}}
Thread-349::DEBUG::2014-08-24 =
10:06:45,708::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:45,756::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000255964 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:45,756::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-13::DEBUG::2014-08-24 =
10:06:45,883::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] =
flowID [852653f]
Thread-13::DEBUG::2014-08-24 =
10:06:45,883::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state init -> =
state preparing
Thread-13::INFO::2014-08-24 =
10:06:45,884::logUtils::44::dispatcher::(wrapper) Run and protect: =
connectStorageServer(domType=3D7, =
spUUID=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'port': '', =
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', =
'vfs_type': 'glusterfs', 'password': '******', 'id': =
'08128803-0141-46aa-835a-2f9734ef0d3b'}], options=3DNone)
Thread-13::DEBUG::2014-08-24 =
10:06:45,889::hsm::2378::Storage.HSM::(__prefetchDomains) =
glusterDomPath: glusterSD/*
Thread-15::DEBUG::2014-08-24 =
10:06:45,890::__init__::225::IOProcess::(_processLogs) Marshaling =
message...DEBUGDEBUGDEBUGDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Message size is =
119
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling =
message...
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling =
message...DEBUGDEBUGDEBUGDEBUGDEBUG|Queuing request...
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Waiting for next =
request...
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Queuing request =
in the thread pool...
Thread-15::DEBUG::2014-08-24 =
10:06:45,891::__init__::225::IOProcess::(_processLogs) Extracting =
request information...
Thread-15::DEBUG::2014-08-24 =
10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Finding =
callback 'glob'...
Thread-15::DEBUG::2014-08-24 =
10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Got =
request for method 'glob'
Thread-15::DEBUG::2014-08-24 =
10:06:45,893::__init__::225::IOProcess::(_processLogs) Marshaling =
message...DEBUGDEBUGDEBUGDEBUGDEBUG|(89757) Building response
Thread-15::DEBUG::2014-08-24 =
10:06:45,893::__init__::225::IOProcess::(_processLogs) (89757) Queuing =
response
Thread-15::DEBUG::2014-08-24 =
10:06:45,893::__init__::225::IOProcess::(_processLogs) Generating =
json...
Thread-15::DEBUG::2014-08-24 =
10:06:45,894::__init__::225::IOProcess::(_processLogs) Sending response =
sized 169
Thread-13::DEBUG::2014-08-24 =
10:06:45,895::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD =
uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)
Thread-13::DEBUG::2014-08-24 =
10:06:45,895::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: =
{2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}
Thread-13::INFO::2014-08-24 =
10:06:45,895::logUtils::47::dispatcher::(wrapper) Run and protect: =
connectStorageServer, Return response: {'statuslist': [{'status': 0, =
'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24 =
10:06:45,896::task::1191::Storage.TaskManager.Task::(prepare) =
Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::finished: {'statuslist': =
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24 =
10:06:45,896::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state =
preparing -> state finished
Thread-13::DEBUG::2014-08-24 =
10:06:45,896::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 =
10:06:45,896::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 =
10:06:45,896::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::ref 0 aborting False
Thread-13::DEBUG::2014-08-24 =
10:06:45,986::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] =
flowID [7b161bce]
Thread-13::DEBUG::2014-08-24 =
10:06:45,987::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state init -> =
state preparing
Thread-13::INFO::2014-08-24 =
10:06:45,987::logUtils::44::dispatcher::(wrapper) Run and protect: =
getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', =
options=3DNone)
Thread-13::DEBUG::2014-08-24 =
10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24 =
10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-13::DEBUG::2014-08-24 =
10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24 =
10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-13::DEBUG::2014-08-24 =
10:06:45,988::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24 =
10:06:45,989::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24 =
10:06:46,038::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result
Thread-13::ERROR::2014-08-24 =
10:06:46,039::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based =
metadata
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
    status =3D self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 416, in =
getSpmStatus
    lVer, spmId =3D self.masterDomain.inquireClusterLock()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult =3D self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24 =
10:06:46,039::task::866::Storage.TaskManager.Task::(_setError) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res =3D f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
    raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: =
("Can't read LV based metadata",)
Thread-13::DEBUG::2014-08-24 =
10:06:46,040::task::885::Storage.TaskManager.Task::(_run) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._run: =
59772503-6431-42a9-b6b0-8e7f4b95a7f5 =
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task
Thread-13::DEBUG::2014-08-24 =
10:06:46,040::task::1217::Storage.TaskManager.Task::(stop) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::stopping in state =
preparing (force False)
Thread-13::DEBUG::2014-08-24 =
10:06:46,040::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 1 aborting True
Thread-13::INFO::2014-08-24 =
10:06:46,040::task::1171::Storage.TaskManager.Task::(prepare) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::aborting: Task is =
aborted: 'Error validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24 =
10:06:46,040::task::1176::Storage.TaskManager.Task::(prepare) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Prepare: aborted: Error =
validating master storage domain
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::task::993::Storage.TaskManager.Task::(_decref) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::task::928::Storage.TaskManager.Task::(_doAbort) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._doAbort: force =
False
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state =
preparing -> state aborting
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::task::550::Storage.TaskManager.Task::(__state_aborting) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::_aborting: recover policy =
none
Thread-13::DEBUG::2014-08-24 =
10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState) =
Task=3D`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state =
aborting -> state failed
Thread-13::DEBUG::2014-08-24 =
10:06:46,042::resourceManager::940::Storage.ResourceManager.Owner::(relea=
seAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24 =
10:06:46,042::resourceManager::977::Storage.ResourceManager.Owner::(cance=
lAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24 =
10:06:46,042::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': =
{'message': 'Error validating master storage domain: ("Can\'t read LV =
based metadata",)', 'code': 355}}
Thread-340::DEBUG::2014-08-24 =
10:06:47,118::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24 =
10:06:47,118::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:47,119::misc::741::Storage.SamplingMethod::(__call__) Trying to =
enter sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24 =
10:06:47,119::misc::743::Storage.SamplingMethod::(__call__) Got in to =
sampling method
Thread-340::DEBUG::2014-08-24 =
10:06:47,119::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, =
this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24 =
10:06:47,119::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo =
-n /sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24 =
10:06:47,168::misc::751::Storage.SamplingMethod::(__call__) Returning =
last result
Thread-340::ERROR::2014-08-24 =
10:06:47,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD=
omain) Error while collecting domain =
2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in =
_monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
    self.refreshStorage()
  File "/usr/share/vdsm/storage/misc.py", line 760, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
    self.__lastResult =3D self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
    supervdsm.getProxy().hbaRescan()
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in hbaRescan
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in =
_callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24 =
10:06:47,759::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:47,807::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000229862 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:47,808::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:49,810::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:49,857::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000265654 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:49,857::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:51,859::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:51,907::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000233321 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:51,907::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds
Thread-349::DEBUG::2014-08-24 =
10:06:53,909::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
/bin/dd =
if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_=
md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd =
None)
Thread-349::DEBUG::2014-08-24 =
10:06:53,957::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) =
FAILED: <err> =3D "/bin/dd: reading =
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o=
utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 =
B) copied, 0.000235232 s, 0.0 kB/s\n"; <rc> =3D 1
Thread-349::WARNING::2014-08-24 =
10:06:53,958::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini=
tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not =
accept requests until init succeeds

------=_NextPart_000_0097_01CFBF84.28615750--

Re: [ovirt-users] Master Storage goes Inactive - Failed to activate Storage Domain

Chris ＠ Veerotech