<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>Still not able to correct this behavior. Has anyone else seen this?<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='color:#1F497D'>Chris<o:p></o:p></span></p></div><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Chris @ VeeroTech.net [mailto:Chris@VeeroTech.net] <br><b>Sent:</b> Sunday, August 17, 2014 7:40 PM<br><b>To:</b> users@ovirt.org<br><b>Subject:</b> Master Storage goes Inactive - Failed to activate Storage Domain<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hello All,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am having an issue on my test setup, the master storage goes inactive. It is set up as to use Gluster and the volume is up and running without issues. Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the ERRORs):<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2014-08-18 01:47:04,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:04,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:14,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:14,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:34,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:34,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:44,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:44,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:54,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:47:54,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:04,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:04,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:14,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:14,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:25,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:25,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:35,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:35,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:45,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:45,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:55,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:48:55,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:49:05,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method<o:p></o:p></p><p class=MsoNormal>2014-08-18 01:49:05,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Here is the vdsm.log on one of the hosts:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=FR-CA>[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log<o:p></o:p></span></p><p class=MsoNormal><span lang=FR-CA> </span>validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures<o:p></o:p></p><p class=MsoNormal> oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch<o:p></o:p></p><p class=MsoNormal> ioproc.touch(path, flags, mode)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch<o:p></o:p></p><p class=MsoNormal> self.timeout)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand<o:p></o:p></p><p class=MsoNormal> raise OSError(errcode, errstr)<o:p></o:p></p><p class=MsoNormal>OSError: [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state preparing<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding callback 'glob'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request for method 'glob'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response sized 168<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e989<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding callback 'touch'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request for method 'touch'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is 147DEBUG|(3641) Building response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response sized 86<o:p></o:p></p><p class=MsoNormal>Thread-1489::ERROR::2014-08-17 19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error<o:p></o:p></p><p class=MsoNormal>Traceback (most recent call last):<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/task.py", line 873, in _run<o:p></o:p></p><p class=MsoNormal> return fn(*args, **kargs)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/logUtils.py", line 45, in wrapper<o:p></o:p></p><p class=MsoNormal> res = f(*args, **kwargs)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus<o:p></o:p></p><p class=MsoNormal> status = self._getSpmStatusInfo(pool)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo<o:p></o:p></p><p class=MsoNormal> (pool.spmRole,) + pool.getSpmStatus()))<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus<o:p></o:p></p><p class=MsoNormal> return self._backend.getSpmStatus()<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus<o:p></o:p></p><p class=MsoNormal> lVer, spmId = self.masterDomain.inquireClusterLock()<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__<o:p></o:p></p><p class=MsoNormal> return getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<o:p></o:p></p><p class=MsoNormal> return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<o:p></o:p></p><p class=MsoNormal> domain = self._findDomain(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<o:p></o:p></p><p class=MsoNormal> dom = findMethod(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain<o:p></o:p></p><p class=MsoNormal> return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__<o:p></o:p></p><p class=MsoNormal> validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures<o:p></o:p></p><p class=MsoNormal> oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch<o:p></o:p></p><p class=MsoNormal> ioproc.touch(path, flags, mode)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch<o:p></o:p></p><p class=MsoNormal> self.timeout)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand<o:p></o:p></p><p class=MsoNormal> raise OSError(errcode, errstr)<o:p></o:p></p><p class=MsoNormal>OSError: [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::885::Storage.TaskManager.Task::(_run) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: ada614f8-75c5-4728-9083-340f918a4417 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing (force False)<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted: u'[Errno 5] Input/output error' - code 100<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing -> state aborting<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting -> state failed<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::ERROR::2014-08-17 19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Traceback (most recent call last):<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper<o:p></o:p></p><p class=MsoNormal> result = ctask.prepare(func, *args, **kwargs)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/task.py", line 103, in wrapper<o:p></o:p></p><p class=MsoNormal> return m(self, *a, **kw)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/task.py", line 1179, in prepare<o:p></o:p></p><p class=MsoNormal> raise self.error<o:p></o:p></p><p class=MsoNormal>OSError: [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [fedc88e]<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state preparing<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type': 'glusterfs', 'password': '******', 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/*<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding callback 'glob'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request for method 'glob'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|(3642) Building response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response sized 168<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: {2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing -> state finished<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state preparing<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)<o:p></o:p></p><p class=MsoNormal>Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing -> state finished<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<o:p></o:p></p><p class=MsoNormal>Thread-1489::DEBUG::2014-08-17 19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last result<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last result<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding callback 'glob'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request for method 'glob'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3643) Building response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response sized 168<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e989<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding callback 'touch'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request for method 'touch'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3644) Building response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Sending response sized 86<o:p></o:p></p><p class=MsoNormal>Thread-29::ERROR::2014-08-17 19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information<o:p></o:p></p><p class=MsoNormal>Traceback (most recent call last):<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain<o:p></o:p></p><p class=MsoNormal> self.domain.selftest()<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__<o:p></o:p></p><p class=MsoNormal> return getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<o:p></o:p></p><p class=MsoNormal> return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<o:p></o:p></p><p class=MsoNormal> domain = self._findDomain(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<o:p></o:p></p><p class=MsoNormal> dom = findMethod(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain<o:p></o:p></p><p class=MsoNormal> return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__<o:p></o:p></p><p class=MsoNormal> validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures<o:p></o:p></p><p class=MsoNormal> oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch<o:p></o:p></p><p class=MsoNormal> ioproc.touch(path, flags, mode)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch<o:p></o:p></p><p class=MsoNormal> self.timeout)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand<o:p></o:p></p><p class=MsoNormal> raise OSError(errcode, errstr)<o:p></o:p></p><p class=MsoNormal>OSError: [Errno 5] Input/output error<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)<o:p></o:p></p><p class=MsoNormal>Thread-36::DEBUG::2014-08-17 19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1<o:p></o:p></p><p class=MsoNormal>Thread-36::WARNING::2014-08-17 19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request information...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding callback 'glob'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request for method 'glob'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-29::DEBUG::2014-08-17 19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e989<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next request...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding callback 'touch'...<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request for method 'touch'<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing response<o:p></o:p></p><p class=MsoNormal>Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json...<o:p></o:p></p><p class=MsoNormal>Thread-29::ERROR::2014-08-17 19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information<o:p></o:p></p><p class=MsoNormal>Traceback (most recent call last):<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain<o:p></o:p></p><p class=MsoNormal> self.domain.selftest()<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__<o:p></o:p></p><p class=MsoNormal> return getattr(self.getRealDomain(), attrName)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<o:p></o:p></p><p class=MsoNormal> return self._cache._realProduce(self._sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<o:p></o:p></p><p class=MsoNormal> domain = self._findDomain(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<o:p></o:p></p><p class=MsoNormal> dom = findMethod(sdUUID)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain<o:p></o:p></p><p class=MsoNormal> return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__<o:p></o:p></p><p class=MsoNormal> validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures<o:p></o:p></p><p class=MsoNormal> oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p></p><p class=MsoNormal> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch<o:p></o:p></p><p class=MsoNormal> ioproc.touch(path, flags, mode)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch<o:p></o:p></p><p class=MsoNormal> self.timeout)<o:p></o:p></p><p class=MsoNormal> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand<o:p></o:p></p><p class=MsoNormal> <span lang=FR-CA>raise OSError(errcode, errstr)<o:p></o:p></span></p><p class=MsoNormal><span lang=FR-CA>OSError: [Errno 5] Input/output error</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any help is getting this sorted is appreciated.<o:p></o:p></p><p class=MsoNormal>Chris<o:p></o:p></p></div></body></html>