- Users - oVirt List Archives

Moving a disk from a ex-pool machine?
by Ernest Beinrohr 25 Aug '14

25 Aug '14

This is a multi-part message in MIME format. --------------070409080601010206050401 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I have ovirt 3.4 and I cannot move one disk to another storage. The machine was part of a pool and is detached already and offline. I still cannot move the disk to another storage. Is there any way of moving that disk? PS: except adding another disk and 'dd'-ing of course. PS2: i need to do this for 30+ vms :( -- Ernest Beinrohr, AXON PRO Ing <http://www.beinrohr.sk/ing.php>, RHCE <http://www.beinrohr.sk/rhce.php>, RHCVA <http://www.beinrohr.sk/rhce.php>, LPIC <http://www.beinrohr.sk/lpic.php>, VCA <http://www.beinrohr.sk/vca.php>, +421-2-62410360 +421-903-482603 --------------070409080601010206050401 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi, I have ovirt 3.4 and I cannot move one disk to another storage. The machine was part of a pool and is detached already and offline. I still cannot move the disk to another storage. Is there any way of moving that disk? PS: except adding another disk and 'dd'-ing of course. PS2: i need to do this for 30+ vms :( <div class="moz-signature">-- <div id="oernii_footer" style="color: gray;"> Ernest Beinrohr, AXON PRO <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/ing.php">Ing</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCE</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCVA</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/lpic.php">LPIC</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/vca.php">VCA</a>, +421-2-62410360 +421-903-482603 </div> <img src="http://nojsstats.appspot.com/UA-44497096-1/email.beinrohr.sk" moz-do-not-send="true" border="0" height="1" width="1"> </div> </body> </html> --------------070409080601010206050401--

2 1

Call for Papers Deadline Friday: Latinoware 2014
by Brian Proffitt 25 Aug '14

25 Aug '14

The CfP for the eleventh annual Latin American Free Software Conference closes on Aug. 29. http://2014.latinoware.org/?page_id=3033 (The page appears to be down right now, but hopefully will be up soon.) Peace, Brian -- Brian Proffitt oVirt Community Manager Project Atomic Community Lead Open Source and Standards, Red Hat - http://community.redhat.com Phone: +1 574 383 9BKP IRC: bkp @ OFTC

1 0

rhev
by mhd 25 Aug '14

25 Aug '14

i install rhev in machine and rhevm another machine , and i use vdsm hooks and create script file and putted in /usr/libexec/vdsm/hooks/before_vm_start , but when i restart rhev the script file is fly not only this file every file which i can create is fly .......... how i can fixed script file for vdsm hooks............???????

3 2

ovirt35 - deep dive - Prevent host fencing while kdumping - integrating with kexec tools
by Barak Azulay 25 Aug '14

25 Aug '14

------=_Part_37888330_1059625418.1408974845452 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit The following is a new meeting request: Subject: ovirt35 - deep dive - Prevent host fencing while kdumping - integrating with kexec tools Organizer: "Barak Azulay" <bazulay(a)redhat.com> Time: Tuesday, August 26, 2014, 4:30:00 PM - 5:30:00 PM GMT +02:00 Jerusalem Invitees: mperina(a)redhat.com; ovedo(a)redhat.com; users(a)ovirt.org; devel(a)ovirt.org *~*~*~*~*~*~*~*~*~* This session describes how kdump mechanism is integrated into oVirt 3.5 non-responsive host treatment in order to prevent host being fenced while gathering its crash dump. google hangout link: https://plus.google.com/events/c2np5fedno8e8qifas6el8a3ih4 wiki page link:http://www.ovirt.org/Fence_kdump ------=_Part_37888330_1059625418.1408974845452 Content-Type: text/calendar; charset=utf-8; method=REQUEST; name=meeting.ics Content-Transfer-Encoding: 7bit BEGIN:VCALENDAR PRODID:Zimbra-Calendar-Provider VERSION:2.0 METHOD:REQUEST BEGIN:VTIMEZONE TZID:Asia/Jerusalem BEGIN:STANDARD DTSTART:16010101T020000 TZOFFSETTO:+0200 TZOFFSETFROM:+0300 RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=10;BYDAY=-1SU TZNAME:IST END:STANDARD BEGIN:DAYLIGHT DTSTART:16010101T020000 TZOFFSETTO:+0300 TZOFFSETFROM:+0200 RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=3;BYDAY=-1FR TZNAME:IDT END:DAYLIGHT END:VTIMEZONE BEGIN:VEVENT UID:cf2e371b-ae13-4500-967b-2acd264a5b65 SUMMARY:ovirt35 - deep dive - Prevent host fencing while kdumping - integrat ing with kexec tools ATTENDEE;CN=Martin Perina;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TR UE:mailto:mperina@redhat.com ATTENDEE;CN=Oved Ourfalli;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TR UE:mailto:ovedo@redhat.com ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE:mailto:users@o virt.org ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE:mailto:devel@o virt.org ORGANIZER;CN=Barak Azulay:mailto:bazulay@redhat.com DTSTART;TZID="Asia/Jerusalem":20140826T163000 DTEND;TZID="Asia/Jerusalem":20140826T173000 STATUS:CONFIRMED CLASS:PUBLIC X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY TRANSP:OPAQUE LAST-MODIFIED:20140825T135405Z DTSTAMP:20140825T135405Z SEQUENCE:0 DESCRIPTION:The following is a new meeting request:\n\nSubject: ovirt35 - de ep dive - Prevent host fencing while kdumping - integrating with kexec tools \nOrganizer: "Barak Azulay" <bazulay(a)redhat.com> \n\nTime: Tuesday\, August 26\, 2014\, 4:30:00 PM - 5:30:00 PM GMT +02:00 Jerusalem\n \nInvitees: mper ina(a)redhat.com\; ovedo(a)redhat.com\; users(a)ovirt.org\; devel(a)ovirt.org \n\n\n *~*~*~*~*~*~*~*~*~*\n\nThis session describes how kdump mechanism is integra ted into oVirt 3.5 non-responsive host treatment \nin order to prevent host being fenced while gathering its crash dump. \n\ngoogle hangout link: https: //plus.google.com/events/c2np5fedno8e8qifas6el8a3ih4 \nwiki page link:http:/ /www.ovirt.org/Fence_kdump BEGIN:VALARM ACTION:DISPLAY TRIGGER;RELATED=START:-PT5M DESCRIPTION:Reminder END:VALARM END:VEVENT END:VCALENDAR ------=_Part_37888330_1059625418.1408974845452--

1 0

ON_QA > Verified
by Jorick Astrego 25 Aug '14

25 Aug '14

Hi, After the rc1 refresh a couple of my bugs have been fixed. I see other bugs switched from ON_QA to Verified. The only option I have is CLOSED. What is the recommended action? Should I set the bug to CLOSED or just wait until someone sets it to Verified? Kind regards, Jorick Astrego Netbulae

2 1

oVirt Manager Portal is hanging and very slow
by Anish Mohan1 25 Aug '14

25 Aug '14

Hello , We have oVirt Engine version 3.4.0-1 el6 in our development Environment . We are facing below issues, Can anyone help us. Now portal is working very slow and it is intermediately hanging If I click on any tab it will take 5 to 10 minutes to load the page , some times it will take more than 30 minutes and we need to bounce the server for getting the console. Also we are getting continuous admin@internal login / log out message and ETL service Error . Thanks & Regards Anish Mohan =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

2 1

Missing tab in ovirt gui
by Leif Larsson 25 Aug '14

25 Aug '14

Hi, Im missing the "Images" tab that should be displayed after clicking on "ovirt-image-repository", under "External providers". Its a default ovirt 3.4 install. The port 9292 was initially firewalled but is now open. The ovirt-engine server has been restarted. Everything else is working as expected. Any ideas ? Thanks! /Leif

2 1

Master Storage goes Inactive - Failed to activate Storage Domain
by Chris ＠ VeeroTech.net 24 Aug '14

24 Aug '14

This is a multipart message in MIME format. ------=_NextPart_000_0B23_01CFBA53.164C8840 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello All, I am having an issue on my test setup, the master storage goes inactive. It is set up as to use Gluster and the volume is up and running without issues. Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the ERRORs): 2014-08-18 01:47:04,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method 2014-08-18 01:47:04,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [40e8ae30] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:14,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method 2014-08-18 01:47:14,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [357af14e] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method 2014-08-18 01:47:24,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [11269c17] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:34,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method 2014-08-18 01:47:34,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [40622d60] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:44,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method 2014-08-18 01:47:44,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:47:54,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method 2014-08-18 01:47:54,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [44b5af81] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:04,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method 2014-08-18 01:48:04,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:14,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method 2014-08-18 01:48:14,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4fed0eac] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:25,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method 2014-08-18 01:48:25,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [48ff78af] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:35,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method 2014-08-18 01:48:35,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [76fe9857] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:45,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method 2014-08-18 01:48:45,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [61356e24] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:48:55,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method 2014-08-18 01:48:55,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [38c2326b] Command SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 2014-08-18 01:49:05,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method 2014-08-18 01:49:05,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3c65466] Command SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100 Here is the vdsm.log on one of the hosts: [root@u5 ~]# tail -f /var/log/vdsm/vdsm.log validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-1489::DEBUG::2014-08-17 19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] Thread-1489::DEBUG::2014-08-17 19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None) Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing request... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building response Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing response Thread-15::DEBUG::2014-08-17 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating json... Thread-15::DEBUG::2014-08-17 19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-1489::DEBUG::2014-08-17 19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is 147DEBUG|(3641) Building response Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing response Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response sized 86 Thread-1489::ERROR::2014-08-17 19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus return self._backend.getSpmStatus() File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus lVer, spmId = self.masterDomain.inquireClusterLock() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::885::Storage.TaskManager.Task::(_run) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: ada614f8-75c5-4728-9083-340f918a4417 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task Thread-1489::DEBUG::2014-08-17 19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing (force False) Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True Thread-1489::INFO::2014-08-17 19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted: u'[Errno 5] Input/output error' - code 100 Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5] Input/output error Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True Thread-1489::DEBUG::2014-08-17 19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False Thread-1489::DEBUG::2014-08-17 19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing -> state aborting Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none Thread-1489::DEBUG::2014-08-17 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting -> state failed Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::ERROR::2014-08-17 19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] Input/output error Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-1489::DEBUG::2014-08-17 19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] flowID [fedc88e] Thread-1489::DEBUG::2014-08-17 19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type': 'glusterfs', 'password': '******', 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None) Thread-1489::DEBUG::2014-08-17 19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/* Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUG|(3642) Building response Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing response Thread-15::DEBUG::2014-08-17 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Generating json... Thread-15::DEBUG::2014-08-17 19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',) Thread-1489::DEBUG::2014-08-17 19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: {2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain} Thread-1489::INFO::2014-08-17 19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-1489::DEBUG::2014-08-17 19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': [{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]} Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing -> state finished Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False Thread-1489::DEBUG::2014-08-17 19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state preparing Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-1489::INFO::2014-08-17 19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}} Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}} Thread-1489::DEBUG::2014-08-17 19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing -> state finished Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseA ll) Owner.releaseAll requests {} resources {} Thread-1489::DEBUG::2014-08-17 19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl l) Owner.cancelAll requests {} Thread-1489::DEBUG::2014-08-17 19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False Thread-36::DEBUG::2014-08-17 19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-29::DEBUG::2014-08-17 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-29::DEBUG::2014-08-17 19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds Thread-29::DEBUG::2014-08-17 19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) Thread-29::DEBUG::2014-08-17 19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last result Thread-29::DEBUG::2014-08-17 19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) Thread-29::DEBUG::2014-08-17 19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-29::DEBUG::2014-08-17 19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last result Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3643) Building response Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing response Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response sized 168 Thread-29::DEBUG::2014-08-17 19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Queuing request... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|(3644) Building response Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing response Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json... Thread-15::DEBUG::2014-08-17 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Sending response sized 86 Thread-29::ERROR::2014-08-17 19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Thread-36::DEBUG::2014-08-17 19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-36::DEBUG::2014-08-17 19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None) Thread-36::DEBUG::2014-08-17 19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/bin/dd: reading `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1 Thread-36::WARNING::2014-08-17 19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests until init succeeds Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing responseDEBUGDEBUG|Receiving request... Thread-15::DEBUG::2014-08-17 19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118 Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request information... Thread-15::DEBUG::2014-08-17 19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding callback 'glob'... Thread-15::DEBUG::2014-08-17 19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request for method 'glob' Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing response Thread-15::DEBUG::2014-08-17 19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json... Thread-29::DEBUG::2014-08-17 19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e 989 Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147 Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message... Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request... Thread-15::DEBUG::2014-08-17 19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next request... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in the thread pool... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding callback 'touch'... Thread-15::DEBUG::2014-08-17 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request for method 'touch' Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing response Thread-15::DEBUG::2014-08-17 19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json... Thread-29::ERROR::2014-08-17 19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 86, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 5] Input/output error Any help is getting this sorted is appreciated. Chris ------=_NextPart_000_0B23_01CFBA53.164C8840 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 14 = (filtered medium)"><style></style></head><body lang=3DEN-US link=3Dblue = vlink=3Dpurple><div class=3DWordSection1>Hello = All,<o:p></o:p><o:p> </o:p>I am having an issue on my test setup, the master = storage goes inactive.  It is set up as to use Gluster and the = volume is up and running without issues.  Here is a snippet of = /var/log/ovirt-engine/engine.log (these are just the = ERRORs):<o:p></o:p><o:p> </o:p>2014-08-18 01:47:04,173 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:04,174 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-98) [40e8ae30] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:47:14,286 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:14,287 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-68) [357af14e] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:47:24,400 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:24,400 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-51) [11269c17] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:47:34,514 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:34,515 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-59) [40622d60] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:47:44,651 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:44,652 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = =3D u5.veerotech.net, HostId =3D 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, = storagePoolId =3D 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution = failed. Exception: VDSErrorException: VDSGenericException: = VDSErrorException: Failed to SpmStatusVDS, error =3D [Errno 5] = Input/output error, code =3D 100<o:p></o:p>2014-08-18 01:47:54,765 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:47:54,766 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-73) [44b5af81] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:04,881 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:04,883 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:14,994 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:14,995 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-20) [4fed0eac] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:25,117 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:25,118 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-35) [48ff78af] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:35,231 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:35,232 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-16) [76fe9857] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:45,344 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:45,345 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-69) [61356e24] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:48:55,460 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:48:55,461 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-28) [38c2326b] Command = SpmStatusVDSCommand(HostName =3D u5.veerotech.net, HostId =3D = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p>2014-08-18 01:49:05,574 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS = method<o:p></o:p>2014-08-18 01:49:05,575 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler_Worker-87) [3c65466] Command = SpmStatusVDSCommand(HostName =3D u6.veerotech.net, HostId =3D = 4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =3D = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception: = VDSErrorException: VDSGenericException: VDSErrorException: Failed to = SpmStatusVDS, error =3D [Errno 5] Input/output error, code =3D = 100<o:p></o:p><o:p> </o:p>Here is the vdsm.log on one of the = hosts:<o:p></o:p><o:p> </o:p>[root@u5 ~]# tail -f = /var/log/vdsm/vdsm.log<o:p></o:p>    = validateFileSystemFeatures(sdUUID, = self.mountpoint)<o:p></o:p>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p>    = ioproc.touch(path, flags, mode)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p>    = self.timeout)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p>    raise OSError(errcode, = errstr)<o:p></o:p>OSError: [Errno 5] = Input/output error<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000230303 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000259145 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client = [74.63.225.147]<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init = -> state preparing<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect: = getSpmStatus(spUUID=3D'83ff2f2d-3c8e-4865-8656-c01d867cff06', = options=3DNone)<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding = callback 'glob'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got = request for method 'glob'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) = Building response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating = json...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding = callback 'touch'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got = request for method 'touch'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is = 147DEBUG|(3641) Building response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response = sized 86<o:p></o:p>Thread-1489::ERROR::2014-08-17 = 19:39:18,955::task::866::Storage.TaskManager.Task::(_setError) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected = error<o:p></o:p>Traceback (most recent call = last):<o:p></o:p>  File = "/usr/share/vdsm/storage/task.py", line 873, in = _run<o:p></o:p>    return = fn(*args, **kargs)<o:p></o:p>  File = "/usr/share/vdsm/logUtils.py", line 45, in = wrapper<o:p></o:p>    res =3D = f(*args, **kwargs)<o:p></o:p>  File = "/usr/share/vdsm/storage/hsm.py", line 611, in = getSpmStatus<o:p></o:p>    = status =3D self._getSpmStatusInfo(pool)<o:p></o:p>  File = "/usr/share/vdsm/storage/hsm.py", line 605, in = _getSpmStatusInfo<o:p></o:p>    = (pool.spmRole,) + pool.getSpmStatus()))<o:p></o:p>  File "/usr/share/vdsm/storage/sp.py", = line 126, in getSpmStatus<o:p></o:p>    return = self._backend.getSpmStatus()<o:p></o:p>  = File "/usr/share/vdsm/storage/spbackends.py", line 416, in = getSpmStatus<o:p></o:p>    lVer, = spmId =3D self.masterDomain.inquireClusterLock()<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p>    = domain =3D self._findDomain(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p>    dom = =3D findMethod(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p>    = ioproc.touch(path, flags, mode)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p>    = self.timeout)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p>    raise OSError(errcode, = errstr)<o:p></o:p>OSError: [Errno 5] = Input/output error<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,955::task::885::Storage.TaskManager.Task::(_run) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._run: = ada614f8-75c5-4728-9083-340f918a4417 = ('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping = task<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,955::task::1217::Storage.TaskManager.Task::(stop) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state = preparing (force False)<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting = True<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is = aborted: u'[Errno 5] Input/output error' - code 100<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno = 5] Input/output error<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting = True<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force = False<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state = preparing -> state aborting<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy = none<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`ada614f8-75c5-4728-9083-340f918a4417`::moving from state = aborting -> state failed<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p>Thread-1489::ERROR::2014-08-17 = 19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5] = Input/output error<o:p></o:p>Traceback (most = recent call last):<o:p></o:p>  File = "/usr/share/vdsm/storage/dispatcher.py", line 71, in = wrapper<o:p></o:p>    result =3D = ctask.prepare(func, *args, **kwargs)<o:p></o:p>  File = "/usr/share/vdsm/storage/task.py", line 103, in = wrapper<o:p></o:p>    return = m(self, *a, **kw)<o:p></o:p>  File = "/usr/share/vdsm/storage/task.py", line 1179, in = prepare<o:p></o:p>    raise = self.error<o:p></o:p>OSError: [Errno 5] = Input/output error<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000245377 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147] = flowID [fedc88e]<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init = -> state preparing<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect: = connectStorageServer(domType=3D7, = spUUID=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'port': '', = 'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', = 'vfs_type': 'glusterfs', 'password': '******', 'id': = '08128803-0141-46aa-835a-2f9734ef0d3b'}], = options=3DNone)<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) = glusterDomPath: glusterSD/*<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|Receiving request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|Marshaling message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding = callback 'glob'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got = request for method 'glob'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUG|(3642) Building response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Generating json...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD = uuids: (u'2ed32e34-2483-4262-a57b-72bf70f3e989',)<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs: = {2ed32e34-2483-4262-a57b-72bf70f3e989: = storage.glusterSD.findDomain}<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect: = connectStorageServer, Return response: {'statuslist': [{'status': 0, = 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist': = [{'status': 0, 'id': = '08128803-0141-46aa-835a-2f9734ef0d3b'}]}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state = preparing -> state finished<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:20,601::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting = False<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init = -> state preparing<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone)<o:p></o:p>Thread-1489::INFO::2014-08-17 = 19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989': = {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', = 'lastCheck': '6.3', 'valid': False}}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished: = {'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1, = 'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': = False}}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state = preparing -> state finished<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(relea= seAll) Owner.releaseAll requests {} resources {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cance= lAll) Owner.cancelAll requests {}<o:p></o:p>Thread-1489::DEBUG::2014-08-17 = 19:39:21,022::task::993::Storage.TaskManager.Task::(_decref) = Task=3D`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting = False<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000253763 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000237548 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.sdc.refreshStorage)<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to = enter sampling method (storage.iscsi.rescan)<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to = sampling method<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, = this will take up to 30 seconds<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo = -n /sbin/iscsiadm -m session -R (cwd None)<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning = last result<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) = /usr/bin/sudo -n /sbin/multipath (cwd None)<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: = <err> =3D ''; <rc> =3D 0<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) = Operation 'lvm invalidate operation' got the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) = Operation 'lvm invalidate operation' released the operation = mutex<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning = last result<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Marshaling message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding = callback 'glob'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got = request for method 'glob'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|(3643) Building response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response = sized 168<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Queuing request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding = callback 'touch'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got = request for method 'touch'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|(3644) Building response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Sending response sized 86<o:p></o:p>Thread-29::ERROR::2014-08-17 = 19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring = information<o:p></o:p>Traceback (most recent = call last):<o:p></o:p>  File = "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain<o:p></o:p>    = self.domain.selftest()<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p>    = domain =3D self._findDomain(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p>    dom = =3D findMethod(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p>    = ioproc.touch(path, flags, mode)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p>    = self.timeout)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p>    raise OSError(errcode, = errstr)<o:p></o:p>OSError: [Errno 5] = Input/output error<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000228935 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000187527 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000238901 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000230531 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = /bin/dd = if=3D/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_= md/outbox iflag=3Ddirect,fullblock bs=3D512 count=3D8 skip=3D8 (cwd = None)<o:p></o:p>Thread-36::DEBUG::2014-08-17 = 19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) = FAILED: <err> =3D "/bin/dd: reading = `/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/o= utbox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 = B) copied, 0.000245347 s, 0.0 kB/s\n"; <rc> =3D = 1<o:p></o:p>Thread-36::WARNING::2014-08-17 = 19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_ini= tMailbox) HSM_MailboxMonitor - Could not initialize mailbox, will not = accept requests until init succeeds<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing = responseDEBUGDEBUG|Receiving request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is = 118<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting = request information...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding = callback 'glob'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got = request for method 'glob'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p>Thread-29::DEBUG::2014-08-17 = 19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading = domain in path = /rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70= f3e989<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is = 147<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling = message...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next = request...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request = in the thread pool...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding = callback 'touch'...<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got = request for method 'touch'<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing = response<o:p></o:p>Thread-15::DEBUG::2014-08-17 = 19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating = json...<o:p></o:p>Thread-29::ERROR::2014-08-17 = 19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain = 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring = information<o:p></o:p>Traceback (most recent = call last):<o:p></o:p>  File = "/usr/share/vdsm/storage/domainMonitor.py", line 215, in = _monitorDomain<o:p></o:p>    = self.domain.selftest()<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 49, in = __getattr__<o:p></o:p>    return = getattr(self.getRealDomain(), attrName)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 52, in = getRealDomain<o:p></o:p>    = return self._cache._realProduce(self._sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 122, in = _realProduce<o:p></o:p>    = domain =3D self._findDomain(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/sdc.py", line 141, in = _findDomain<o:p></o:p>    dom = =3D findMethod(sdUUID)<o:p></o:p>  File = "/usr/share/vdsm/storage/glusterSD.py", line 32, in = findDomain<o:p></o:p>    return = GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))<o:p></o= :p>  File = "/usr/share/vdsm/storage/fileSD.py", line 157, in = __init__<o:p></o:p>    = validateFileSystemFeatures(sdUUID, self.mountpoint)<o:p></o:p>  File = "/usr/share/vdsm/storage/fileSD.py", line 86, in = validateFileSystemFeatures<o:p></o:p>    = oop.getProcessPool(sdUUID).directTouch(testFilePath)<o:p></o:p>  File = "/usr/share/vdsm/storage/outOfProcess.py", line 320, in = directTouch<o:p></o:p>    = ioproc.touch(path, flags, mode)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 460, in touch<o:p></o:p>    = self.timeout)<o:p></o:p>  File = "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line = 344, in _sendCommand<o:p></o:p>    raise = OSError(errcode, errstr)<o:p></o:p>OSError: [Errno 5] Input/output = error<o:p></o:p><o:p> </o:p><o:p> </o:p>Any help is = getting this sorted is appreciated.<o:p></o:p>Chris<o:p></o:p></div></body></html> ------=_NextPart_000_0B23_01CFBA53.164C8840--

3 5

I wrote a howto manual of ovirt/glusterfs in Chinese
by lofyer 24 Aug '14

24 Aug '14

It's about skills of deploying and using oVirt/Glusterfs. https://inthecloud.readthedocs.org And I'm writing some aritcles about machine learning. It will be great if you put it to the documentation list of oVirt.

2 3

What's the Audit Mode's Function in Quota？
by xiec.fnst＠cn.fujitsu.com 24 Aug '14

24 Aug '14

--_000_EE4D679B9474414187D2E27D8B6890F692B663G08CNEXMBPEKD03g0_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 SGksYWxsDQoNCiAgICAgICAgIEZyb20gdGhlIHRyZWUgb2YgcmVzb3VyY2UoanVzdCBsaWtlIHRo ZSBvcGVyYXRpb24gb2YgYXNzaWduaW5nIFF1b3RhIHRvIGEgdXNlcix3ZSBjYW4gc2VlIHRoZSCh sFF1b3RhobEgVGFiIGluIHRoZSBuYXZpZ2F0aW9uIHBhbmUpLCBXZSBjYW4gc3RpbGwgQWRkL0Vk aXQvQ29weS9SZW1vdmUgdGhlIFF1b3RhIHdoZW4gRGlzYWJsZSBNb2RlIGlzIG9uLiAgU28gd2hh dKGvcyB0aGUgZnVuY3Rpb24gb2YgQXVkaXQgTW9kZShBbnl0aGluZyB3ZSBjYW4gZG8gaW4gQXVk aXQgTW9kZSBjYW4gYmUgZG9uZSBpbiB0aGUgRGlzYWJsZSBNb2RlLCBleGNlcHQgZm9yIGhpZGRl biB0aGUgobBRdW90YaGxIHRhYiBpbiBEZXRhaWxlZCBQYW5lKT8NCg== --_000_EE4D679B9474414187D2E27D8B6890F692B663G08CNEXMBPEKD03g0_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dgb2312"> <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang=3D"ZH-CN" link=3D"blue" vlink=3D"purple" style=3D"text-justify-t= rim:punctuation"> <div class=3D"WordSection1"> Hi,all<o:p></o:p> <o:p> </o:p>      &= nbsp;   From the tree of resource(just like the operation of assi= gning Quota to a user,we can see the =A1=B0Quota=A1=B1 Tab in the navigatio= n pane), We can still Add/Edit/Copy/Remove the Quota when Disable Mode is o= n.  So what=A1=AFs the function of Audit Mode(Anything we can do in Audit Mode can be done in= the Disable Mode, except for hidden the =A1=B0Quota=A1=B1 tab in Detailed = Pane)?<o:p></o:p> </div> </body> </html> --_000_EE4D679B9474414187D2E27D8B6890F692B663G08CNEXMBPEKD03g0_--

2 1